JP7485893B2 - Information processing device, control program, and control method - Google Patents

Description

The present invention relates to an information processing device, a control program, and a control method.

There is a known technology for creating albums in a digital photo frame that uses face recognition software to perform facial recognition on images of people read from an external memory, displays a list of thumbnail images of specific people that are found, and when a person for whom an album is to be created is selected, the image of that person is displayed as an album (see, for example, Patent Document 1).

There are also known cameras that can register facial images in the camera and perform person searches within captured images.

In addition, because children's faces are prone to change as they grow, there is known technology that sends a message encouraging the user to update the registered face image after a certain period of time depending on the age of the person registered in the camera (see Patent Document 2).

JP 2011-129033 A JP 2014-182485 A

However, in the case of the technology disclosed in Patent Document 2, children grow quickly and their faces change as they grow, so it is necessary to register facial images periodically, which is a cumbersome operation.

The present invention provides a technology that makes it easy to manage images of each subject without having to register the images of the subject in advance.

An information processing device according to one aspect of the present invention is characterized by comprising an acquisition unit that acquires subject information for each of one or more subjects, a classification unit that classifies the facial images of people captured in each of one or more recorded images by person, and an association processing unit that associates an image in which the person is captured with subject information corresponding to the person based on the frequency of occurrence of each of the classified people.

According to one aspect of the present invention, it is possible to easily manage images of each subject without having to register the images of the subject in advance.

FIG. 1 illustrates an example of an information processing device according to an embodiment of the present invention. 1 is a diagram illustrating an example of a network configuration of an image collection and management system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of functional blocks of the image collection management system according to the present embodiment. 4 is a diagram showing an example of a data structure of a database managed by a server 15 in the present embodiment. FIG. 4 is a diagram illustrating an example of a face tag in the present embodiment. FIG. FIG. 4 is a diagram showing an example of a collection screen in the present embodiment. FIG. 13 is a diagram showing an example of a detailed screen for each face tag in the embodiment. FIG. 13 is a diagram showing an example of a correction target image selection screen in the present embodiment. FIG. 13 is a diagram showing an example of a correction image selection screen in the present embodiment. FIG. 2 is a sequence diagram of the entire image collection management system according to the present embodiment. 11 is a flowchart of an image analysis process (S3) in the present embodiment. 11 is a flowchart (part 1) of a clustering process (S4) and a tagging process (S5) in this embodiment. 13 is a flowchart (part 2) of the clustering process (S4) and the tagging process (S5) in this embodiment. 11 is a flowchart of a collection screen display process (S7) and a face tag detail screen display process (S10) in the present embodiment. FIG. 2 is a block diagram showing an example of a configuration of a hardware environment of a computer that executes a program according to the present embodiment.

FIG. 1 is a diagram showing an example of an information processing device in this embodiment. The information processing device 1 includes an acquisition unit 2, a classification unit 3, and an associated processing unit 4. An example of the information processing device 1 is a server 15, which will be described later.

The acquisition unit 2 acquires subject information for one or more subjects. An example of the acquisition unit 2 is the subject information registration unit 33.

The classification unit 3 classifies the facial images of people captured in each of the one or more recorded images by person. An example of the classification unit 3 is the clustering unit 36.

The related processing unit 4 associates images of people with subject information corresponding to the people based on the frequency of appearance of each classified person. An example of the related processing unit 4 is the tagging unit 37.

This configuration makes it easy to manage images for each subject without having to register the image of the subject in advance.

Here, the subject information may include age information of the subject. In this case, the related processing unit 4 can estimate the age of the person from the classified face image of the person, and identify which of the subjects the person is based on the estimated age of the person and the age information.

By configuring it in this way, if a child appears in a photo among adults, the child can be inferred from the photo and the subject can be identified by comparing it with pre-registered age information.

When multiple subjects corresponding to the acquired multiple subject information are captured in an image, the related processing unit 4 may associate the image in which the multiple subjects are captured with group information corresponding to the multiple subjects.

By configuring it in this way, images that contain two or more children as subjects can be tagged with, for example, "children." As a result, classification can be easily separated, making it easier to manage images that only contain children.

When an image contains a subject group including a first subject (referred to as a "registered subject") corresponding to the acquired subject information and a second subject (referred to as an "unregistered subject") that has a frequency equal to or higher than a predetermined level and is other than the first subject, the related processing unit 4 may associate the image in which the subject group is captured with group information corresponding to the subject group.

By configuring it in this way, images that show a family, including children and adults, can be tagged with, for example, "everyone." As a result, it is easy to categorize, making it easier to manage images that show multiple people, including adults.

The classification unit 3 may form a set of one or more recorded images for each predetermined period, and determine which set to use to classify the facial image based on the date and time the image was captured. By configuring in this way, facial image recognition can be performed using a set within a predetermined period from the date and time of capture.

The classification unit 3 may change the frequency with which groups are formed based on the subject information. By configuring in this way, when groups are formed, for example, children can be formed more frequently than adults.

When the image is a moving image, the classification unit 3 may identify still images that make up the moving image. The identified still images are, for example, some of the still images that make up the moving image, and these may be still images from the first half of the moving image. An example of the classification unit 3 is not only the clustering unit 36 but also the image analysis unit 35. In this case, the related processing unit 4 may associate the moving image with subject information corresponding to the person (registered subject) captured in the still image based on the frequency of each person classified using the still images.

By configuring it in this way, videos can be classified by subject without having to pre-register facial images of the subjects, making it easier to manage images by subject.

Figure 2 is a diagram showing an example of the network configuration of the image collection management system in this embodiment. The image collection management system 11 includes one or more user terminals 12, a server 15, and a communication network 16. The one or more user terminals 12 and the server 15 are connected to each other via the communication network 16 so that they can communicate with each other.

The user terminal 12 is a portable electronic terminal device with a communication function that can be connected to the communication network 16, such as a smartphone, tablet terminal, or mobile phone carried by the user. The user terminal 12 has an operation display unit 13, a microphone for inputting sound, a speaker for outputting sound, a camera, etc. The operation display unit 13 is a device that allows input operations to be made to the screen 14 and displays images on the screen 14, and is, for example, a touch panel display.

The user terminal 12 also has an imaging device for capturing still and video images of a subject. The imaging device may be built into the user terminal 12 or may be an external type. The user terminal 12 may be a so-called personal computer or a camera that can be connected to a communication network such as the Internet. The photos and videos captured by the imaging device may be stored in a storage device within the user terminal 12 or may be stored in a removable storage medium, etc.

The user terminal 12 can download and install a specific application program from an application program download site on the Internet. This allows the user to operate the application program by performing input operations on the operation display unit 13 of the user terminal 12. In this embodiment, as an example, it is assumed that dedicated application software (hereinafter referred to as "dedicated application software") that allows photos and videos of children and family to be shared and organized on a cloud server or the like is installed on the user terminal 12.

The user can cause the user terminal 12 to access the server 15 by starting up this dedicated application software on the user terminal 12. When the user terminal 12 accesses the server 15, the user may be required to input, for example, a login user ID that has been registered in advance.

When the dedicated application software installed in the user terminal 12 is started, a graphical user interface (GUI) screen, which will be described later, is displayed on the screen 14 of the user terminal 12. The GUI screen displays a screen for registering photos and videos of children and family members to the server 15 and for viewing registered photos and videos.

The user can start up the dedicated application software installed in the user terminal 12 and operate the GUI screen to access the server 15. The user can then operate the GUI screen to upload photos and videos stored in the user terminal 12 to the server 15.

The server 15 is an information processing device that manages photos and videos using a dedicated application program installed on the user terminal 12 as an application program interface (API). In response to requests from the dedicated application program, the server 15 collects uploaded photos and videos, and enables family members and users within a certain range to view the collected photos and videos. Hereinafter, photos (still images) and videos are collectively referred to as images.

Server 15 may be a server system consisting of one or more physical information processing devices, or it may be a virtual server system consisting of one or more virtual server devices.

Figure 3 is a diagram showing an example of functional blocks of the image collection management system 11 in this embodiment. The following describes the functions of the user terminal 12 and the server 15.

The user terminal 12 includes an operation display unit 13, a communication unit 21, a control unit 22, an imaging unit 23, and a storage unit 24. The communication unit 21 is an interface that enables communication with devices connected to the communication network 16. Here, the communication unit 21 communicates with the server 15 via the communication network 16.

The operation display unit 13 is a touch panel display that allows input by a user's input operation and also allows display. Note that in this embodiment, a touch panel display is used as an example, but this is not limited thereto, and the input unit and the display unit may be independent of each other.

The control unit 22 controls the overall operation of the user terminal 12. The control unit 22 controls the display state of the screen 14 based on user interface (UI) data 26 of a dedicated application program 25 stored in the memory unit 24. The imaging unit 23 is an imaging device such as a CCD (Charge Coupled Device).

The storage unit 24 stores installed dedicated application programs 25, UI data 26, images (photo data, video data) 27, various data, etc. Here, data includes not only status information and text data, but also graphical data. The images 27 include metadata such as the date and time of capture and location information from the Global Positioning System (GPS).

The server 15 includes a communication unit 31, a control unit 32, and a memory unit 41. The communication unit 31 is an interface that enables communication with devices connected to the communication network 16. The communication unit 31 communicates with the user terminal 12 via the communication network 16.

The control unit 32 controls the operation of the entire server 15. At the same time, the control unit 32 executes processes in response to requests from the user terminal 12, and performs control such as returning the execution results to the user terminal 12. The control unit 32 executes the application program according to this embodiment, thereby functioning as a subject information registration unit 33, an image acquisition unit 34, an image analysis unit 35, a clustering unit 36, a tag assignment unit 37, and an output unit 38.

The subject information registration unit 33 accepts information (subject information) about children among the user's family members (children, couples, grandparents, etc.) in response to a registration request from the user terminal 12. The subject information registration unit 33 registers the subject information accepted from the user terminal 12 in the storage unit 41. Here, a subject whose subject information is registered in the storage unit 41 (specifically, the subject information management DB 43 described later) is referred to as a "registered subject." Also, as described later, among the people captured in the image, a person whose subject information is not registered in the subject information management DB 43 (a subject other than a registered subject) is referred to as an "unregistered subject." Also, when simply referring to a "subject," it may refer to a subject before registration, may include registered subjects and unregistered subjects, or may simply refer to a person in the image.

The image acquisition unit 34 acquires images (including photo data and videos) related to the user's family sent from the user terminal 12 in response to an upload request from the user terminal 12.

The image analysis unit 35 analyzes the acquired image, detects the face of the subject in the image, estimates the posture of the detected subject, estimates the age of the detected subject, detects the number of subjects detected in the image, calculates feature vectors from the face image, etc. After the analysis process, the image analysis unit 35 stores the analysis results in the storage unit 41. In addition, the image analysis unit 35 stores the image after analysis in the storage unit 41.

The clustering unit 36 performs cluster analysis (clustering) on the images (face images) stored in the storage unit 41 using the clustering conditions and the image analysis results. In clustering, a set of data is divided into subsets (clusters) so that the feature vectors of the face images included in each subset are common or similar. Here, in this embodiment, images in which a single person appears are used as the images to be clustered, but this is not limited to this, and images in which multiple people appear may also be clustered. In this case, the face image of each person appearing in the image may be extracted, and clustering may be performed on each of the extracted face images.

The tagging unit 37 can associate images containing face images captured in large numbers (frequently occurring) as a result of clustering with subject information. Alternatively, the tagging unit 37 may associate images containing face images classified by clustering with subject information based on the age estimation result.

Then, the tagging unit 37 assigns a "child tag" as a face tag to the image associated with the subject information.

In addition, for an image containing multiple people, if all of those people are registered subjects (i.e., if all the people in the photo are children), the tagging unit 37 assigns a "brother tag (or sister tag)" as a face tag to the image.

In addition, if an image containing multiple people includes a person other than a registered subject (an unregistered subject) (i.e., if the people in the photo include an adult), the tagging unit 37 assigns a "family tag" as a face tag to the image.

The output unit 38 displays the images with face tags on the screen 14 of the user terminal 12 for each subject, brother/sister, or family member according to the face tag in response to the screen operation of the user terminal 12. Note that the output unit 38 may output the images with tags to a printer, display device, other storage medium, etc., in response to the screen operation of the user terminal 12, or may transmit them to an external device via the communication network 16.

The memory unit 51 has an image storage area 42. The image storage area 42 stores images updated by the user, images for which analysis by the image analysis unit 35 has been completed, or thumbnail images generated from these images, etc.

The storage unit 51 also stores a subject information management database 43, an image analysis result management database 44, a clustering condition management database 45, a collection screen management database 46, a face tag master information management database 47, a subject-specific face tag management database 48, and a face tag-specific detail screen management database 49. In the following, the databases are referred to as "DBs."

The subject information management DB 43 is a database that manages information about the target subject (i.e., the "registered subject"; in this example, the child in the user's family) in order to assign a face tag to each subject. The image analysis result management DB 44 is a database that stores the results of the analysis by the image analysis unit 35. The clustering condition management DB 45 is a database that stores the conditions (clustering conditions) when the clustering unit 36 performs clustering. The face tag master information management DB 46 is a database that manages the master information of the face tag. The collection screen management DB 47 is a database that manages the contents of the collection screen displayed on the screen 14 of the user terminal 12. The subject-specific face tag management DB 48 is a database that manages subjects (including not only registered subjects but also multiple subjects including unregistered subjects) and the corresponding face tags. The face tag-specific detail screen management DB 49 is a database that manages a detail screen (face tag-specific detail screen) that displays an image to which a selected face tag has been assigned.

Figure 4 is a diagram showing an example of the data structure of a database managed by the server 15 in this embodiment. In the following, each record in the subject information management DB 43 may be referred to as subject information. Each record in the image analysis result management DB 44 may be referred to as image analysis result information. Each record in the clustering condition management DB 45 may be referred to as clustering condition information. Each record in the face tag master information management DB 46 may be referred to as face tag master information. Each record in the collection screen management DB 47 may be referred to as collection screen information. Each record in the subject-specific face tag management DB 48 may be referred to as subject-specific face tag information. Each record in the face tag-specific detail screen management DB 49 may be referred to as face tag-specific detail screen information.

The subject information management DB43 includes the data items "Family ID", "Subject ID", "Subject Name", and "Date of Birth". The "Family ID" item stores information (family ID) that identifies the user's family. The "Subject ID" item stores identification information (subject ID) that identifies the subject person. The "Subject Name" item stores the name of the person corresponding to the subject ID (subject name). The "Date of Birth" item stores the subject's date of birth.

The image analysis result management DB44 includes the data items "Image ID", "Image capture date and time", "Image capture location", "Face detection result", "Number of subjects", "Posture estimation result", "Age estimation result", and "Feature vector". The "Image ID" item stores information that identifies the image (image ID). The "Image capture date and time" item stores the date and time when the image was captured. The "Image capture location" item stores the location (position) where the image was captured. The "Face detection result" item stores information about faces detected by face detection processing. The "Number of subjects" item stores the number of faces detected. The "Posture estimation result" item stores information about the posture of the person whose face is detected in the image. The "Age estimation result" item stores the age estimated from the face detected in the image. The "Feature vector" item stores a feature vector (feature amount) that represents the features of the face.

The clustering condition management DB45 includes the data items "Family ID" and "Period." The "Family ID" item stores information (family ID) that identifies the user's family. The "Period" item stores the cycle and period for which clustering is performed (from year/month/day to year/month/day), deadline, date, time, etc.

The face tag master information management DB46 includes the data items "face tag ID" and "face tag type." The "face tag ID" item stores information (face tag ID) that identifies the type of tag (face tag) that allows images to be viewed by child. The "face tag type" item stores the type of face tag. In this example, the face tag types stored are "child," "family," and "brothers/sisters."

Collection screen management DB47 includes data items of "Family ID" and "Face tag ID" (or "Face tag ID + subnumber"). The item "Family ID" stores information (family ID) that identifies a family. The item "Face tag ID" stores face tag IDs of people classified by clustering and associated with subject information. Note that if the same type of face tag ID exists for the same family ID, the subject ID of the associated subject information is assigned as a subnumber to that face tag ID and stored.

The subject-specific face tag management DB 48 includes data items of "family ID", "face tag ID", "subject ID number", "face tag notation", and "profile image ID". The item "family ID" stores information (family ID) that identifies a family. The item "face tag ID" stores the face tag ID of a person classified by clustering and associated with subject information. In the case where the same type of face tag ID exists for the same family ID, the subject ID of the associated subject information is stored as the sub number of the face tag ID in the "subject ID number". In the "face tag notation", the subject name corresponding to the subject ID of the associated subject information is stored as the notation content of the face to be specifically displayed on the screen 14. In the "profile image ID", identification information (profile image ID) that identifies a thumbnail image of an image selected from the clustered images as a profile image to be displayed on the screen 14 is stored.

The face tag detail screen management DB49 includes the data items "family ID", "face tag ID", "branch number", "year and month", "age", "growth memo", "weight", "height", and "image ID". The item "family ID" stores information (family ID) that identifies a family member. The item "face tag ID" stores the face tag ID of a person classified by clustering and associated with the subject information. In the "branch number", if the same type of face tag ID exists for the same family ID, the subject ID of the associated subject information is stored as the branch number of the face tag ID. The item "year and month" stores the year and month when the image was captured. The item "age" stores the age (monthly age) obtained by subtracting the date of birth corresponding to the subject ID registered in the subject information management DB43 from the year and month when the image was captured. The item "growth memo" stores comments arbitrarily entered by the user. The item "weight" stores the weight of a subject (which may be not only a registered subject but also multiple subjects including an unregistered subject) arbitrarily entered by the user. The "Height" item stores the height of the subject (which may be multiple subjects including not only registered subjects but also unregistered subjects) that is arbitrarily input by the user. The "Image ID" item stores the image ID of the image captured in that year and month that corresponds to that face tag ID.

Figure 5 is a diagram illustrating an example of a face tag in this embodiment. A face tag is a tag that enables photos and videos to be viewed by child, and is attached to images that include children or subject information related to the images. Note that images can be viewed not only by children, but also by family members or brothers and sisters.

Examples of types of face tags include "children," "family," and "brothers/sisters." When a child tag is assigned to an image that shows a person alone, the name of that person (registered subject) is displayed on the screen 14 of the user terminal 12. For example, in the case of an image that shows a child named "Aki," a child tag is assigned and "Aki" is displayed on the screen 14. Child tags are assigned to the number of registered subjects (children) registered in the subject information management DB 43.

In addition, images that contain two or more children are given a brothers/sisters tag. Images with the brothers/sisters tag are displayed as "children" on the screen 14. For example, an image that contains a child named "Aki" and a child named "Haru" is given the brothers/sisters tag and displayed as "children" on the screen 14. Note that if there are no brothers or sisters, this tag is not given. When multiple subject information items are registered with the same family ID in the subject information management DB 43, the registered subjects are determined to be brothers or sisters.

In addition, a family tag is added to an image that shows one or more children and one or more adults together. An image with a family tag added is displayed as "everyone" on the screen 14. For example, an image that shows a child named "Aki" and his mother is added with a family tag and displayed as "everyone" on the screen 14. Note that in this example, images of adults only are not displayed on the screen 14.

Figure 6 is a diagram showing an example of a collection screen in this embodiment. When a user starts the dedicated application program 25 on the user terminal 12, a collection screen 51 is displayed. The collection screen 51 allows the user to centrally view images that have been uploaded by the user and collected and managed on the server 15.

The collection screen 51 includes a video display section 52, an album section for each person 53, and a favorites section 55. The video display section 52 displays videos uploaded to the server 15, and tapping on a video causes it to be played.

In the album column 53 for each person, the person's profile image 53a and the corresponding face tag notation (see FIG. 5) are displayed for each face tag. The number of profile images displayed is the same as the number of registered subjects (children) registered in the subject information management DB 43. The profile image is a thumbnail image generated from one of the images to which a face tag has been added. Note that by tapping the face tag notation portion 53b, the corresponding profile image 53a can be changed.

In the case of an image in which a child named "Aki" appears, a child tag is added and the image is displayed as "Aki" on the collection screen 51. In the case of an image in which a child named "Haru" appears, a child tag is added and the image is displayed as "Haru" on the collection screen 51.
In the case of an image that includes a child named "Haru" and an adult (e.g., dad and mom), a family tag is added, and in the album column 53 for each person on the collection screen 51, "everyone" is displayed.
In the case of an image showing a child named "Aki" and a child named "Haru", the brothers/sisters tag is added and the image is displayed as "children" in the album column 53 for each person on the collection screen 51.

The edit button 54 is a button for manually correcting the combination of the profile image 53a and the face tag notation in the album column 53 for each person if it is incorrect. Pressing the edit button 54 transitions to the screen in Figure 8.

The favorites section 55 allows the user to register their favorite photos and videos. In order to view content that does not fit within the screen 14, the collection screen 51 can be scrolled up and down or left and right to display (display) the hidden parts.

Here, when the profile image 53a of any person is tapped in the album column 53 for each person, a transition to a details screen (details screen by face tag) in which images with that face tag can be viewed is displayed. For example, when the profile image 53a of "Aki" or the child tag "Aki" 53b is tapped, a transition to the screen in FIG. 7 is displayed.

Figure 7 is a diagram showing an example of a face tag detail screen in this embodiment. The face tag detail screen 61 is a detail screen for viewing and managing images of a person selected in the album column 53 for each person on the collection screen 51.

The face tag detail screen 61 includes a profile image 62, a description field 63 corresponding to the face tag, an age in months (days after birth) display field 64, a timeline display field 65, a timeline corresponding image display field 66, a growth memo input field 67, and a weight/height input field 68.

The profile image 62 and the description field 63 corresponding to the face tag are the same as those displayed in the album field 53 for each person on the collection screen 51. The age (days after birth) display field 64 displays the current age (days after birth) of the registered subject. The chronological display field 65 displays the age in months and year and month for each month. The chronological corresponding image display field 66 displays images for the period corresponding to the chronological display field 65. The growth memo input field 67 allows comments about the child's growth at that time to be entered and recorded. The weight/height input field 68 allows the child's weight and height at that time to be entered and recorded.

When the face tag detail screen 61 is scrolled, a time-series display field 65, a time-series corresponding image display field 66, a growth memo input field 67, and a weight/height input field 68 are displayed in order of age and month.

Next, a method for correcting a wrong combination of a person in the profile image 53a and the name in the face tag notation portion 53b in the person-by-person album column 53 will be described with reference to FIGS. 8 and 9. FIG.
8 is a diagram showing an example of a correction target image selection screen in this embodiment. The correction target image selection screen 71 is displayed when the edit button 54 on the collection screen 51 is pressed. The face tag correction screen 71 includes a correction image selection field 72. The correction image selection field 72 displays profile images of registered subjects to which child tags have been added. That is, the number of profile images corresponding to each child is displayed as many as the number of children. Here, an image in which the combination of the person in the profile image 53a and the name in the face tag notation portion 53b is incorrect (i.e., the image to be corrected) can be selected.

In the image to be edited selection field 72, tap the profile image with the character "Haru" written on it as the image to be edited. Then, the screen in Figure 9 will be displayed.

FIG. 9 is a diagram showing an example of a correction image selection screen in this embodiment. The correction image selection screen 81 includes a correction image selection field 82. The correction image selection field 82 displays profile images of registered subjects to which a child tag has been assigned. The correction image selection field 82 displays profile images of registered subjects to which the same and different child tags have been assigned. Different child tags refer to registered subjects of different children, such as "Aki" and "Haru." The same child tag refers to child tags for the same person.

On the correction image selection screen 81, tap the correct profile image that corresponds to the face tag notation. This allows the face tag notation to be associated with the corresponding correct profile image. It is also possible to associate the person identified by the face tag with subject information.

Note that multiple profile images (thumbnail images) may be generated for the same child. In this case, for example, the correction image selection screen 81 also displays profile images with the same child tag, so if there are multiple profile images for the same child, the user may select from among them the profile image that best suits their preferences.

10 is a sequence diagram of the entire image collection management system in this embodiment. The image collection management system 11 in this embodiment has three main phases: a subject information registration phase, an image collection phase, and an image viewing phase.
In the subject information registration phase, the user starts the dedicated application program 25 on the user terminal 12, displays a specific screen, and inputs the name (subject name) and date of birth of the subject (e.g., the user's own child). At this time, if the user has multiple children, the input is repeated for each child. When the input is complete, the user presses the registration button.

Then, the user terminal 12 transmits the subject information (including the family ID, subject name, and date of birth) to the server 15 (S1). The server 15 assigns a subject ID to each piece of subject information and registers it in the subject information management DB 43.

Next, in the image collection phase, the user uploads the images stored in the user terminal 12 to the server 15 via a specific screen of the dedicated application program 25 (S2).

The server 15 analyzes the images uploaded from the user terminal 12 (S3) and performs clustering processing to classify the images by specific subjects, multiple subjects, etc. (S4). The server 15 associates the classified images with subject information based on the frequency of photography and age estimated from the face image, and assigns face tags (S5).

Finally, in the image viewing phase, the user launches the dedicated application program 25 and attempts to display the collection screen. The user terminal 12 then transmits a collection screen display request (including the family ID) to the server 15 (S6).

When the server 15 acquires the collection screen display request, it performs processing (collection screen display processing) for displaying the collection screen 51 (S7). Here, the server 15 acquires collection screen information corresponding to the family ID from the collection screen management DB 47 as content information for the collection screen 51, and acquires subject-specific face tag information corresponding to the family ID from the subject-specific face tag management DB. The server 15 transmits the collection screen information and subject-specific face tag information to the user terminal 12 (S8).

When the user terminal 12 acquires the collection screen information and subject-specific face tag information from the server 15, it displays the collection screen 51 using the acquired collection screen information and subject-specific face tag information.

Then, the user selects and taps the profile image 53a of any person (subject) from the person-by-person album column 53 on the collection screen 51. The user terminal 12 then transmits to the server 15 a face tag detail screen viewing request (family ID, face tag ID, subject ID) to view the face tag detail screen 61 corresponding to the tapped profile image 53a (S9).

When the server 15 acquires a request to view a face tag detail screen, it performs a process (face tag detail screen display process) to display the face tag detail screen 61 (S10). Here, the server 15 acquires face tag detail screen information corresponding to the family ID, face tag ID, and subject ID from the face tag detail screen management DB 49 as content information for the face tag detail screen. The server 15 transmits the acquired face tag detail screen information to the user terminal 12 (S11).

When the user terminal 12 acquires the face tag detail screen information, it displays the face tag detail screen 61 using the acquired face tag detail screen information.

Figure 11 is a flowchart of the image analysis process (S3) in this embodiment. The control unit 32 functions as an image acquisition unit 34 and an image analysis unit 35 by reading the program according to this embodiment.

The image acquisition unit 34 acquires an image uploaded from the user terminal 12 (S21). The image analysis unit 35 uses a face detection algorithm to detect a face from the acquired image, and stores the face detection result in the image analysis result management DB 44 (S22).

The image analysis unit 35 counts the number of people in the image from the number of faces detected in the acquired image, and stores this number in the image analysis result management DB 44 (S23). The image analysis unit 35 uses a posture estimation algorithm to estimate the posture of the person relative to the detected face from the acquired image, and stores the posture estimation result in the image analysis result management DB 44 (S24). When estimating the posture of the person, for example, it estimates which way the face is facing.

The image analysis unit 35 uses an age estimation algorithm to estimate the age of the person corresponding to the detected face from the acquired image, and stores the age estimation result in the image analysis result management DB 44 (S25).

The image analysis unit 35 calculates the face detection result, the posture estimation result, and a feature vector of the face image from the posture estimation result, and stores the calculated feature vector in the image analysis result management DB 44 (S26). The image analysis unit 35 stores the analyzed image in the image storage area 42 (S27).

FIGS. 12A and 12B are flowcharts of the clustering process (S4) and the tagging process (S5) in this embodiment. The control unit 32 functions as a clustering unit 36 and a tagging unit 37 by reading the program according to this embodiment.

The clustering unit 36 determines the family and period to be clustered based on the clustering conditions (S31). Here, the clustering unit 36 obtains clustering condition information corresponding to the family ID from the clustering condition management DB 45, and determines the period (the period to be clustered).

The clustering unit 36 reads out images that fall within the determined period from the image storage area 42 based on the image capture date and time information attached to the images (S32). The clustering unit 36 references the number of subjects in the image analysis result management DB 44 and extracts images in which the subject appears alone (face images) from the read out images (S33).

The clustering unit 36 obtains a feature vector corresponding to the image ID of the read image from the image analysis result management DB 44, and performs clustering on the face images based on the obtained feature vector, classifying them by person (S34).

The tagging unit 37 associates the face image with the subject information based on the frequency of appearance of the person (or age information), and assigns a child tag to all images in the cluster (S35). Here, the tagging unit 37 obtains the subject information corresponding to the family ID from the subject information management DB 43.

For example, when only one piece of subject information is obtained from the subject information management DB43 (i.e., when there is one child), the tag assignment unit 37 identifies the person in the cluster with the largest number of images captured among the clusters classified by person as the subject information, associates the family ID, image ID, subject ID of the subject information, and face tag ID (ID indicating "child tag"), and registers it in the subject-specific face tag management DB48.

For example, when there are multiple subject information retrieved from the subject information management DB 43 (i.e. when there are brothers and sisters), the tag assignment unit 37 estimates the age of the person in the cluster classified by person, and based on the age and the date of birth of the subject information, associates the image ID of the estimated face image with the subject ID of the subject information, the face tag ID (ID indicating the "child tag"), and the family ID, and registers them in the subject-specific face tag management DB 48.

The tag assignment unit 37 generates a profile image (thumbnail image) of the person using one of the images of the person to which the child tag has been assigned, and sets a notation tag (S36). Here, the tag assignment unit 37 generates a profile image and a profile image ID that identifies the profile image from one of the face images associated with the subject information, and stores them in the subject-specific face tag management DB. The tag assignment unit 37 also obtains the family ID and the subject name corresponding to the subject ID from the subject information management DB 43.

The tag assignment unit 37 registers the acquired subject name in the "face tag notation" item of the subject-specific face tag information corresponding to the family ID, face tag ID, and sub-number (subject ID) in the subject-specific face tag management DB 48. The tag assignment unit 37 repeats S35 and S36 the number of times corresponding to the number of clusters classified by person.

The tagging unit 37 references the "Number of subjects" item in the image analysis result management DB 44 and obtains one unprocessed image that contains multiple people from the read images (S37). The tagging unit 37 uses the feature vectors in the image analysis result management DB 44 to determine whether all the people in the image are registered subjects (S38).

If all the people in the image are registered subjects (YES in S38), the tagging unit 37 assigns a brother/sister tag to the image (S39). Here, the tagging unit 37 associates the family ID, image ID, and face tag ID (ID that identifies the "brother/sister tag"), and registers them in the subject-specific face tag management DB 48.

If none of the people in the image are registered subjects, i.e., if there is one or more unregistered subjects in the image (NO in S38), the tagging unit 37 assigns a family tag to the image (S40). Here, the tagging unit 37 associates the family ID, image ID, and face tag ID (ID that identifies the "family tag") and registers them in the subject-specific face tag management DB 48. Note that if the frequency of an unregistered subject (the number of images of that person) is above a certain level, a family tag may be assigned to the image.

The tagging unit 37 repeats the process of S37 to S40 for the number of images that contain multiple people.

Then, the tag assignment unit 37 generates a profile image (thumbnail image) for the brothers/sisters tag using one of the images to which the brothers/sisters tag has been assigned. The tag assignment unit 37 sets a notation tag (S41). Here, the tag assignment unit 37 generates a profile image and a profile image ID that identifies the profile image from one of the images to which the brothers/sisters tag has been assigned, and stores them in the subject-specific face tag management DB. Furthermore, the tag assignment unit 37 stores "children" in the item "face tag notation" as the brothers/sisters tag.

The tag assignment unit 37 generates a family tag profile image (thumbnail image) using one of the images to which a family tag has been assigned. The tag assignment unit 37 sets a notation tag (S42). Here, the tag assignment unit 37 generates a profile image and a profile image ID that identifies the profile image from one of the images to which a family tag has been assigned, and stores them in the subject-specific face tag management DB. Furthermore, the tag assignment unit 37 stores "everyone" in the item "face tag notation" as a family tag.

The profile image (thumbnail image) generated in S36, S41, and S42 is not limited to one image, and multiple candidate profile images (thumbnail images) may be generated. In this case, for example, among the multiple candidate profile images (thumbnail images), the one with the most recent shooting date and time may be set as the profile image by default. Then, when the user wants to change the profile image according to his/her preference, the face tag notation portion 53b may be tapped to change the corresponding profile image 53a, as described in FIG. 6.

Figure 13 is a flowchart of the collection screen display process (S7) and the face tag detail screen display process (S10) in this embodiment. The control unit 32 functions as the output unit 38 by reading the program according to this embodiment.

The output unit 38 receives a collection screen display request (including a family ID) from the user terminal 12 (S51). The output unit 38 then receives collection screen information corresponding to the family ID from the collection screen management DB 47 (S52).

The output unit 38 acquires all face tag IDs and face tag ID + branch numbers included in the acquired collection screen information (S53). The output unit 38 acquires the family ID and the "face tag notation" and "profile image ID" corresponding to the face tag ID or the face tag ID + branch number from the subject-specific face tag management DB 48 (S54).

The output unit 38 acquires a profile image corresponding to the acquired profile image ID from the image storage area 42 (S55). The output unit 38 transmits the collection screen information, the face tag notation, and the profile image to the user terminal 12 (S56).

When the user terminal 12 acquires the collection screen information, the face tag notation, and the profile image, the face tag notation and the profile image are displayed in pairs in the album column 53 for each person on the collection screen 51. The user then selects one of the profile images from the album column 53 for each person on the collection screen 51. The user terminal 12 then transmits a request to view a detailed screen by face tag to the server 15.

The output unit 38 is in a waiting state until it receives a request to view a detailed screen by face tag (NO in S57). When a request to view a detailed screen by face tag is received (YES in S57), the output unit 38 acquires a request to view a detailed screen by face tag (including a family ID, a face tag ID, and a sub-number (subject ID)) (S58).

The output unit 38 acquires face tag detail screen information corresponding to the family ID, face tag ID, and sub-number (subject ID) from the face tag detail screen management DB 49 (S59). The output unit 38 acquires an image corresponding to the image ID included in the face tag detail screen information (S60).

The output unit 38 sorts the acquired images in chronological order by the image capture date and time (S61). The output unit 38 transmits the acquired face tag-specific detail screen information and the sorted images to the user terminal 12 (S62).

When the user terminal 12 acquires the face tag detail screen information and the sorted images, the profile image 62 on the face tag detail screen 61 and the description field 63 corresponding to the face tag display the profile image and face tag description tapped in the person-specific album field 53 on the collection screen 51 before the transition.

In addition, based on the detailed screen information by face tag, an age in months (days after birth) display field 64, a time series display field 65, a time series corresponding image display field 66, a growth memo input field 67, and a weight/height input field 68 are displayed. Images are also displayed in chronological order according to the time series corresponding image display field 66.

Figure 14 is an example of a configuration block diagram of a hardware environment of a computer that executes a program in this embodiment. A computer 101 functions as a user terminal 12 and a server 15. The computer 101 is composed of a CPU 102, a ROM 103, a RAM 104, a storage device 105, an input I/F 106, an output I/F 107, a communication I/F 108, a reading device 109, and a bus 110.

Here, CPU refers to a central processing unit. ROM refers to a read-only memory. RAM refers to a random access memory. I/F refers to an interface. The bus 110 is connected to the CPU 102, ROM 103, RAM 104, storage device 105, input I/F 106, output I/F 107, communication I/F 108, and, if necessary, a reading device 109.

When the computer 101 is the server 15, the CPU 102 reads out the program according to this embodiment from the storage device 105, and functions as a subject information registration unit 33, an image acquisition unit 34, an image analysis unit 35, a clustering unit 36, a tag assignment unit 37, and an output unit 38. The ROM 103 indicates a read-only memory. The RAM 104 is a temporary storage memory.

The storage device 105 is a device that stores large amounts of information. Various types of storage devices can be used as the storage device 105, such as a hard disk, a solid-state drive (SSD), or a flash memory card. The storage device 105 stores a program according to an embodiment of the present invention and various data stored in the storage unit 40.

The input I/F 106 can be connected to input devices such as a keyboard, mouse, electronic camera, web camera, microphone, scanner, sensor, tablet, touch panel display, information reading device, etc. The output I/F 107 can be connected to output devices such as a display, touch panel, projector, printer, speaker, etc.

The communication I/F 108 is an interface such as a port for connecting to a communication network and communicating with other devices. The communication network may be the Internet, a local area network (LAN), a wide area network (WAN), a dedicated line, a wired network, a wireless network, or the like. The reading device 109 is a device for reading portable recording media.

The program that realizes the processing described in the above embodiment may be stored, for example, in the storage device 105 from the program provider via the communication network and the communication I/F 108. The program that realizes the processing described in the above embodiment may also be stored in a portable storage medium that is commercially available and in circulation. In this case, the portable storage medium may be set in the reading device 105, and the program may be read and executed by the CPU 102. Various types of storage media can be used as the portable storage medium, such as a CD-ROM, a flexible disk, an optical disk, a magneto-optical disk, an IC card, a USB memory device, or a semiconductor memory card. The program stored in such a storage medium is read by the reading device 109.

The program may be installed on a stand-alone computer, or may be installed by a cloud computer and provide only the functions to the user. In this embodiment, an application program capable of communicating with the server 15 is installed on the user terminal 12 to control the image collection management system 11 in this embodiment, but this is not limited to this. For example, a web browser installed on the user terminal 12 may be used to access the server 15 and use the system according to this embodiment operated on the server 15 side.

According to this embodiment, it is possible to easily manage images for each subject without having to register the image of the subject in advance. Note that in this embodiment, information about children is registered as the subject information to be registered, but this is not limited thereto, and for example, information about adults may also be registered.

The frequency of clustering may also be varied based on family information (children, adults). For example, clustering may be performed more frequently for children than for adults. Specifically, clustering may be performed at regular intervals for photos containing people determined to be children (people under a specified age) through age estimation processing by the image analysis unit 35, and clustering may also be performed at longer intervals for photos containing people determined to be adults (people over a specified age).

The clustering unit 36 can also form a set of images to be clustered for each specified period. The clustering unit 36 can then determine which set to use to cluster the face images by specifying the date and time the images were taken.

In the above embodiment, the image is mainly a photograph (still image), but it may be a video. For example, when the image is a video, the image analysis unit 35 can identify one of the still images that make up the video. In this case, the image analysis unit 35 can calculate the face detection result, the posture estimation result, and the posture estimation result for the identified still image, and calculate the feature vector of the face image from the face detection result, the posture estimation result, and the posture estimation result. After this, as in the case of still images, the tagging unit 37 associates the video with the subject information corresponding to the person captured in the still image based on the frequency of each person classified using the still image. By configuring in this way, it is possible to easily manage the face image of each subject in the video as well, without having to register the face image of the subject in advance.

Although this aspect has been described above based on the embodiment and modified examples, the embodiment of the above-mentioned aspect is intended to facilitate understanding of this aspect and does not limit this aspect. This aspect may be modified or improved without departing from the spirit and scope of the claims, and equivalents are included in this aspect. Furthermore, if a technical feature is not described as essential in this specification, it may be deleted as appropriate.

In addition to the above embodiment, the following is further noted.
(Appendix 1)
An acquisition unit that acquires subject information of one or more subjects;
a classification unit that classifies face images of people captured in each of the one or more recorded images by person;
an association processing unit that associates an image in which the person is captured with subject information corresponding to the person based on the frequency of appearance of each of the classified persons;
An information processing device comprising:
By configuring in this way, it is possible to easily manage images of each subject without having to register the image of the subject in advance.
(Appendix 2)
The subject information includes age information of the subject,
The information processing device described in Appendix 1, characterized in that the related processing unit is further equipped with: an age estimation unit that estimates an age of the person from the classified face image of the person; and a determination unit that identifies the person as one of the subjects based on the estimated age of the person and the age information.
With this configuration, if a child appears in a photograph among adults, the child can be inferred from the photograph and identified by comparing the child's age with pre-registered age information.
(Appendix 3)
The information processing device according to claim 1 or 2, characterized in that, when multiple subjects corresponding to the acquired multiple pieces of subject information are captured in the image, the association processing unit associates the image in which the multiple subjects are captured with group information corresponding to the multiple registered subjects.
With this configuration, images that contain two or more children as subjects can be tagged with, for example, "children." As a result, classification can be easily achieved, making it easier to manage images that contain only children.
(Appendix 4)
The information processing device described in any one of Appendices 1 to 3, characterized in that, when a subject group including a first subject corresponding to the acquired subject information and a second subject other than the first subject and having a frequency equal to or higher than a predetermined frequency is captured in the image, the association processing unit associates the image in which the subject group is captured with group information corresponding to the subject group.
With this configuration, images of family members, including children and adults, can be tagged with, for example, "everyone." As a result, classification can be easily achieved, making it easier to manage images that include multiple people, including adults.
(Appendix 5)
The information processing device according to any one of appendices 1 to 4, wherein the classification unit forms a set of one or more of the recorded images for each predetermined period of time, and determines which of the sets to use to classify the face images based on the shooting date and time of the images.
With this configuration, face image recognition can be performed using a set within a predetermined period from the photographing date and time.
(Appendix 6)
The information processing device according to claim 5, wherein the classification unit changes a frequency of forming the groups based on the subject information.
With this configuration, when forming a group, for example, children can be more frequently involved than adults.
(Appendix 7)
When the image is a moving image, the classification unit identifies still images constituting the moving image;
The information processing device described in any one of appendices 1 to 6, characterized in that the association processing unit associates the moving image with subject information corresponding to the person captured in the still image based on the frequency of appearance of each of the people classified using the still image.
By configuring in this way, it is possible to easily manage the facial image of each subject even in a moving image, without having to register the facial image of the subject in advance.
(Appendix 8)
On the computer,
An acquisition process for acquiring object information of one or more objects;
A classification process for classifying face images of people captured in each of the one or more recorded images by person;
an association process for associating an image in which the person is captured with subject information corresponding to the person based on the frequency of appearance of each of the classified persons;
A control program that executes the above.
By configuring in this way, it is possible to easily manage images of each subject without having to register the image of the subject in advance.
(Appendix 9)
The computer
Obtaining subject information for each of the one or more subjects;
Classifying the facial images of people captured in each of the one or more recorded images by person;
associate an image in which the person is captured with subject information corresponding to the person based on the frequency of appearance of each of the classified persons;
A control method for executing a task.
By configuring in this way, it is possible to easily manage images of each subject without having to register the image of the subject in advance.

REFERENCE SIGNS LIST 1 Information processing device 2 Acquisition unit 3 Classification unit 4 Related processing unit 11 Image collection management system 12 User terminal 15 Server 16 Communication network 31 Communication unit 32 Control unit 33 Subject information registration unit 34 Image acquisition unit 35 Image analysis unit 36 Clustering unit 37 Tagging unit 38 Output unit 41 Memory unit 42 Image storage area 43 Subject information management DB
44 Image analysis result management DB
45 Clustering condition management DB
46 Collection screen management DB
47 Face tag master information management DB
48 Subject-specific face tag management DB

Claims (4)

an acquisition unit that acquires subject information related to a child of a family associated with identification information that identifies the family;
a classification unit that classifies facial images of individuals appearing alone from a plurality of recorded images into groups of individuals;
an association processing unit that associates an image in which the person is captured with the subject information corresponding to the person based on the frequency of appearance of each of the classified persons, and adds first tag information indicating that the person is a child to the image in which the person is captured;
Equipped with
The information processing device is characterized in that the associated processing unit acquires an image containing multiple subjects from the multiple images, and if the multiple subjects appearing in the acquired image are subjects corresponding to subject information associated with an image to which the first tag information has been assigned, assigns second tag information to the acquired image indicating that the subjects are brothers or sisters. On the computer,
an acquisition process for acquiring subject information related to a child of the family associated with identification information identifying the family;
A classification process for classifying facial images of individuals that are captured alone from the multiple recorded images;
an association process of associating an image in which the person is captured with the subject information corresponding to the person based on the frequency of appearance of each of the classified persons, and adding first tag information indicating that the person is a child to the image in which the person is captured;
Run the command,
The associated processing is a control program that acquires an image containing multiple subjects from the multiple images, and if the multiple subjects in the acquired image are subjects corresponding to subject information associated with an image to which the first tag information has been assigned, assigns second tag information to the acquired image indicating that the subjects are brothers or sisters. The computer
obtaining subject information relating to children of the family associated with identification information identifying the family;
A process of classifying facial images of individuals that are captured alone from the multiple recorded images;
an association process of associating an image in which the person is captured with the subject information corresponding to the person based on the frequency of appearance of each of the classified persons, and adding first tag information indicating that the person is a child to the image in which the person is captured;
Run
The associated processing is a control method in which an image containing multiple subjects is acquired from the multiple images, and if the multiple subjects appearing in the acquired image are subjects corresponding to subject information associated with an image to which the first tag information is assigned, second tag information indicating that the subjects are brothers or sisters is assigned to the acquired image. A server and a user terminal are provided,
The server,
an acquisition unit that acquires subject information related to a child of a family associated with identification information that identifies the family;
a classification unit that classifies facial images of individuals appearing alone from a plurality of recorded images into groups of individuals;
an association processing unit that associates an image in which the person is captured with the subject information corresponding to the person based on the frequency of appearance of each of the classified persons, and adds first tag information indicating that the person is a child to the image in which the person is captured;
Equipped with
the association processing unit acquires an image including a plurality of subjects from the plurality of images, and when the plurality of subjects appearing in the acquired image correspond to subject information associated with an image to which the first tag information is assigned, assigns second tag information to the acquired image, the second tag information indicating that the subjects are brothers or sisters;
The information processing system is characterized in that the server causes the user terminal to display an image to which at least one of the first tag information and the second tag information has been added.