JP6869809B2 - Image estimator - Google Patents

Description

The present invention relates to an image estimation device for classifying images such as photographs.

Patent Document 1 describes a system that recognizes an object in a photo, automatically adds a tag based on the image recognition result to the photo, and classifies the photo using the tag.

Japanese Unexamined Patent Publication No. 2015-501982

Here, it may be desired to distinguish and organize images of a specific group that shares a living space, such as a family, from other images. However, the system as described in Patent Document 1 described above simply performs tagging based on the image recognition result of the object, for example, when there is a person's face in the photograph, the tagging to that effect is performed. It is only possible to effectively distinguish an image of a specific group that shares a living space such as a family from other images.

The present invention has been made in view of the above circumstances, and an object of the present invention is to effectively distinguish and organize images of a specific group sharing a living space from other images.

The image estimation device according to one aspect of the present invention is similar to an acquisition unit that acquires a plurality of images and acquires information indicating an imaging location of each of the plurality of images, and a plurality of faces included in the plurality of images. It includes a subject identification unit that identifies a plurality of subjects by grouping based on the degree, and an estimation unit that estimates the relationship between the plurality of subjects identified by the subject identification unit according to the information indicating the imaging location.

In the image estimation device according to one aspect of the present invention, the relationship between a plurality of subjects whose faces are detected is estimated according to the information indicating the image capturing location. As a result, for example, a plurality of subjects included in an image captured at a specific place (for example, at home) are considered to be members of the same group (for example, a family) having the specific place as a common living space. It is possible to estimate and so on. This makes it possible to effectively distinguish and organize images of a specific group such as a family member who shares a living space with other images in consideration of the relationship between the subjects.

According to the present invention, images of a specific group sharing a living space can be effectively distinguished and organized from other images.

It is a block diagram which shows the functional structure of the family estimation system which concerns on this embodiment. It is a figure which shows the hardware configuration of the family estimation apparatus shown in FIG. It is a table showing each information stored in the data storage unit, FIG. 3A is image file management information, FIG. 3B is face image file management information, and FIG. 3C is face group management information. 3 (d) is a table showing home registration management information. It is a figure which shows the display image of a family candidate in a data display terminal. It is a flowchart which shows a series of processing of the family estimation method performed by the family estimation apparatus shown in FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same reference numerals are used for the same or equivalent elements, and duplicate description is omitted.

FIG. 1 is a block diagram showing a functional configuration of the family estimation system 1 according to the present embodiment. The family estimation system 1 is a system that groups only the images showing the family from the images (photographs) captured by the user, and distinguishes the images showing the family from other images and organizes them. The family estimation system 1 includes a family estimation device 10 and a data display terminal 20.

The family estimation device 10 is a server configured to be able to communicate with the data display terminal 20, and is tagged with an acquisition unit 11, a subject identification unit 12, a family candidate registration unit 13 (estimation unit), a notification unit 14, and a notification unit 14. A unit 15 and a data storage unit 16 (activity location storage unit) are provided. In the following description, for convenience, an example in which the family estimation device 10 communicates with one data display terminal 20 will be described, but in reality, the family estimation device 10 communicates with a plurality of data display terminals 20. It is configured to be possible. The family estimation device 10 is composed of, for example, the hardware shown in FIG.

FIG. 2 is a diagram showing a hardware configuration of the family estimation device 10. As shown in FIG. 2, the family estimation device 10 physically includes one or a plurality of processors 1001, a memory 1002 which is a main storage device, a storage 1003 such as a hard disk or a semiconductor memory, and a data transmission / reception device such as a network card. It is configured as a computer system including a communication device 1004, an input device 1005, and an output device 1006 such as a display. Each function shown in FIG. 1 loads input device 1005, output device 1006, and communication under the control of processor 1001 by loading predetermined computer software on hardware such as memory 1002 shown in FIG. This is realized by operating the device 1004 and reading and writing data in the memory 1002 and the storage 1003.

The details of each function of the family estimation device 10 will be described with reference to FIG. 1 again.

The acquisition unit 11 acquires a plurality of images (a plurality of images captured by the data display terminal 20) from the data display terminal 20, and also acquires information indicating an imaging location of each of the plurality of images. More specifically, the acquisition unit 11 acquires the user account ID, which is information that uniquely identifies the user of the data display terminal 20, from the data display terminal 20, in addition to the image and the information indicating the image capturing location of the image. .. The acquisition unit 11 generates image file management information for each image based on the acquired information, and stores the image file management information and the image in the data storage unit 16.

FIG. 3A is a diagram showing an example of image file management information I1 stored in the data storage unit 16 by the acquisition unit 11. As shown in FIG. 3A, in the image file management information I1, the file ID, the user account ID, the data storage destination, the latitude, the longitude, the face detection status flag of the preprocessing, the face detection result flag of the preprocessing, And the family tag flag is associated. The file ID is an ID that uniquely identifies the stored image. As described above, the user account ID is information that uniquely identifies the user of the data display terminal 20. The data storage destination is information indicating the storage destination of the image. Latitude and longitude are latitudes and longitudes indicating the image capture location. The face detection status flag of the preprocessing, the face detection result flag of the preprocessing, and the family tag flag are flags for which either "0" or "1" is set, and the image file management information I1 is set by the acquisition unit 11. When it is generated, "0" is set as the initial value. Details of these flags will be described later.

The subject identification unit 12 groups a plurality of faces included in the plurality of images acquired by the acquisition unit 11 based on the degree of similarity, and identifies a plurality of subjects. The subject identification unit 12 performs preprocessing, face detection processing, and grouping processing (face recognition processing) in this order.

In the preprocessing, the subject identification unit 12 rotates the image captured by the data display terminal 20 and stored in the data storage unit 16. At the time of imaging on the data display terminal 20, it is conceivable that the user tilts the data display terminal 20 at various angles to image the subject. For example, the orientation of the subject's face differs by 90 degrees between the case where the long side of the data display terminal 20 is set to the vertical direction and the subject is imaged, and the case where the long side of the data display terminal 20 is set to the horizontal direction and the subject is imaged. It will be. Further, at the time of imaging, it is conceivable that the subject itself tilts his face. Even if the subject is the same, if the orientation of the face is different between different images, the faces may be grouped as different subjects. As a response, the subject identification unit 12 rotates the images so that the orientations of the faces between the different images are as common as possible. The subject identification unit 12 uses, for example, a face detection technique (details will be described later), which has been well known in the past, to make relative facial features (face parts such as eyes, nose, cheekbones, and chin) in the image. The target position is acquired, and the image is rotated so that the directions in which the parts face each other are in a predetermined direction. In the face detection process described later, it is necessary to detect each face part of all the subjects in the image, but in the preprocessing, a part of the face part (tilt can be grasped) of at least one subject. Parts) should be detected. When the preprocessing is completed, the subject identification unit 12 changes the "face detection status flag of the preprocessing" in the image file management information I1 (see FIG. 3A) described above to "1". When a face is appropriately detected in the preprocessing, the "face detection result flag of the preprocessing" in the image file management information I1 (see FIG. 3A) described above is changed to "1". For example, when the tilt of the data display terminal 20 at the time of imaging can be grasped, the subject identification unit 12 simply considers the tilt of the data display terminal 20 at the time of shooting without using the face detection technique. Each image may be rotated in a direction in which the tilt is corrected.

In the face detection process, the subject identification unit 12 detects the face of the subject for each preprocessed image stored in the data storage unit 16. The subject identification unit 12 detects the faces of all the subjects in the image by using the face detection technique which has been well known in the past. The subject identification unit 12 determines a face region from the image and obtains the positions of facial features (face parts such as eyes, nose, cheekbones, and chin). In addition, the subject identification unit 12 estimates the age and gender of the detected face. The subject identification unit 12 estimates the age and gender of the detected face by learning, for example, a large number of face patterns stored in the data storage unit 16 or an external database. In estimating age, for example, the face of a child has larger eyes and a higher position than the face of an adult, the nose and mouth are smaller, the contour of the face is rounded, and the chin is smaller. .. The subject identification unit 12 generates face image file management information for each detected face, and stores the face image file management information in the data storage unit 16. Further, the subject identification unit 12 cuts out the detected face image and stores it in the data storage unit 16.

FIG. 3B is a diagram showing an example of face image file management information I2 stored in the data storage unit 16 by the subject identification unit 12. As shown in FIG. 3B, the face ID, the image file management ID, the face size, the estimated age, the estimated gender, and the face group ID are associated with each other in the face image file management information I2. The face ID is an ID that uniquely identifies the detected face (the face that has been cut out and stored in the data storage unit 16). The image file management ID is an ID that uniquely identifies the image in which the face is detected, and is an ID common to the file ID of the image file management information I1 described above. The face size is the detected rectangular area of the face (the area of the rectangle surrounding the face). The estimated age is the age estimated from the face in the face detection process. The estimated sex is the sex estimated from the face in the face detection process. The face group ID is an ID that uniquely identifies the subject, which is input after the grouping process described later.

In the grouping process (face recognition process), the subject identification unit 12 collates a plurality of faces detected in the face detection process with each other by using a face recognition technique that has been well known in the past, and makes similar faces the same. Group as the face of the subject. First, based on the estimated age of the face image file management information I2 of the data storage unit 16, the subject identification unit 12 uses each face image stored in the data storage unit 16 as a face image of an adult subject and a child subject. Divide into face images. In the subject identification unit 12, a face image whose estimated age is at least a predetermined age (for example, 12 years old) is a face image of an adult (first age group), and a face image under a predetermined age is a child (second age group). Use as a face image of the subject. The subject identification unit 12 separates the face image of an adult subject and the face image of a child subject, and groups them respectively. For example, the subject identification unit 12 extracts facial features that collate with each other and directly geometrically compares them, and when the similarity of the faces exceeds a predetermined threshold value, these faces are similar (same). Group as (the face of the subject). The threshold value for determining that the faces of a child's subject are similar when grouping the faces is set higher than the threshold value for determining that the faces of an adult subject are similar. That is, in the grouping of the faces of children, it is more difficult to determine that they are similar (the faces of the same subject) than the grouping of the faces of adults.

After grouping, the subject identification unit 12 sets a face group ID for each specified subject (face group registration is performed). The subject identification unit 12 inputs the set face group ID into the above-mentioned face image file management information I2 (see FIG. 3B). Further, the subject identification unit 12 generates face group management information for each subject (for each face group ID), and stores the face group management information in the data storage unit 16.

FIG. 3C is a diagram showing an example of face group management information I3 stored in the data storage unit 16 by the subject identification unit 12. As shown in FIG. 3C, the face group management information I3 is associated with the face group ID, the family candidate flag, and the user-selected family flag. The face group ID is an ID that uniquely identifies the subject. The family candidate flag and the user-selected family flag are flags for which either "0" or "1" is set, and are used as initial values when the face group management information I3 is generated by the subject identification unit 12. "0" is set. Details of these flags will be described later.

The family candidate registration unit 13 (estimation unit) estimates the relationship between a plurality of subjects specified by the subject identification unit 12 according to the information indicating the imaging location. Specifically, the family candidate registration unit 13 is a member of the user's family for a plurality of subjects whose imaging location is within a predetermined range from the user's home and whose face is included in the image. Presumed to be (make up a family). The family candidate registration unit 13 acquires information about the user's home (information about the location of the home) by referring to the home registration management information stored in the data storage unit 16.

FIG. 3D is a diagram showing an example of home registration management information I4 stored in the data storage unit 16. As shown in FIG. 3D, in the home registration management information I4, the home ID, the user account ID, the latitude, and the longitude are associated with each other. The home ID is an ID that uniquely identifies the user's home. The user account ID is information that uniquely identifies the user of the data display terminal 20. Latitude and longitude are latitudes and longitudes that indicate the location of your home.

First, the family candidate registration unit 13 refers to the home registration management information I4 of the data storage unit 16 and acquires the latitude and longitude of the home associated with the user account ID of the target user. Subsequently, the family candidate registration unit 13 refers to the image file management information I1 of the data storage unit 16, and among the images associated with the user account ID of the target user, the latitude and longitude of the image imaging location of the images. Acquires all the file IDs of images that match or approximate the latitude and longitude of the home described above (that is, the file IDs of images whose imaging location is within a predetermined range from the home). The latitude and longitude of the imaging location are, for example, those that are positioned by the data display terminal 20 at the time of imaging by the data display terminal 20 (details will be described later), and the above-mentioned "approximate" range (described above, predetermined). The range) is a range in which the imaging location may be at home in consideration of the positioning accuracy of the data display terminal 20.

Subsequently, the family candidate registration unit 13 refers to the face image file management information I2 of the data storage unit 16 and acquires all the face group IDs of the face images in which the acquired file ID and the image file management ID match. The family candidate registration unit 13 presumes that all of the plurality of subjects specified by the acquired face group ID are members of the user's family, and registers them as family candidates (group composition candidates). Specifically, the family candidate registration unit 13 changes the family candidate flag associated with the acquired face group ID to "1" in the face group management information I3 (see FIG. 3C). A subject presumed to be a member of the user's family is registered as a family candidate.

The notification unit 14 notifies the user of a plurality of subjects (presumed to be members of the family) registered as family candidates by the family candidate registration unit 13. The notification unit 14 notifies the user of the family candidate by displaying an image relating to the subject which is the family candidate. The notification unit 14 transmits the image to the data display terminal 20 so that the image relating to the subject which is a family candidate can be displayed on the data display terminal 20 used by the user.

First, the notification unit 14 refers to the face group management information I3 of the data storage unit 16 and acquires all the face group IDs in which the family candidate flag is “1”. Subsequently, the notification unit 14 refers to the face image file management information I2 of the data storage unit 16 and acquires all the face IDs associated with the acquired face group IDs. Regarding the adult face group ID, the notification unit 14 has the face image (data cut out by the subject identification unit 12) of the face ID having the largest face size (see FIG. 3B) among the corresponding plurality of face IDs. Only one image (face image stored in the storage unit 16) is used as an image to be displayed. On the other hand, the notification unit 14 selects two or more of the corresponding face IDs for the child's face group ID, and sets the face images of the plurality of face IDs as the images to be displayed. That is, the notification unit 14 increases the number of images displayed for the family candidate for the child subject to be larger than the number of images displayed for the family candidate for the adult subject.

The tagging unit 15 attaches a family tag (group composition tag) to all the images including the subject selected by the user among the family candidates notified to the user by the notification unit 14.

First, the tagging unit 15 acquires all face IDs related to all face images selected by the user. Subsequently, the tagging unit 15 refers to the face image file management information I2 of the data storage unit 16, acquires the face group ID associated with the acquired face ID, and associates the acquired face group ID with the acquired face group ID. Acquire all the image file management IDs. Subsequently, the tagging unit 15 changes the user selection family flag associated with the acquired face group ID to "1" for the face group management information I3 of the data storage unit 16. Finally, the tagging unit 15 changes the family tag flag of the image whose acquired image file management ID and the file ID match with respect to the image file management information I1 of the data storage unit 16 to "1". As a result, it is possible to attach a family tag to all the images in which the subject selected by the user among the family candidates appears. By adding the family tag in this way, for example, when the user wants to search for an image of the family, it is possible to effectively search for an image that is likely to show the family.

As described above, the data storage unit 16 (activity location storage unit) includes image file management information I1 (see FIG. 3A), face image file management information I2 (see FIG. 3B), and face group management information. Stores I3 (see FIG. 3C) and home registration management information I4 (see FIG. 3D), which is information about the user's home location (information indicating the activity location of a specific group). The information about the user's home location stored in the home registration management information I4 is based on the information transmitted from the data display terminal 20.

Next, the details of each function of the data display terminal 20 will be described with reference to FIG. The data display terminal 20 is a communication terminal configured to be able to communicate with the family estimation device 10, and is, for example, a smartphone or a tablet terminal. The data display terminal 20 includes a camera and a touch panel as a hardware configuration. The data display terminal 20 includes an imaging unit 21, a data storage unit 22, a communication unit 23, and a family candidate display unit 24.

The imaging unit 21 is a function of capturing an image of a subject by controlling a camera. The imaging unit 21 stores an image such as a captured photograph and an imaging location (latitude / longitude) of the image in the data storage unit 22. The data storage unit 22 stores the image captured by the image pickup unit 21, the image capture location of the image, and the information about the user's home (information about the home location) of the data display terminal 20. The communication unit 23 communicates with the family estimation device 10. The communication unit 23 transmits the image stored in the data storage unit 22, the image capturing location of the image, and information about the home location to the family estimation device 10 (specifically, the acquisition unit 11). The communication unit 23 receives the face image related to the family candidate from the family estimation device 10 (specifically, the notification unit 14) and outputs the face image to the family candidate display unit 24. The communication unit 23 transmits the face image selected by the user among the face images related to the family candidate to the family estimation device 10 (specifically, the tagging unit 15). The family candidate display unit 24 displays the face image related to the family candidate output from the data storage unit 22 on the display of the data display terminal 20 so that the user can select it.

FIG. 4 is a screen image displayed on the display of the data display terminal 20 by the family candidate display unit 24. The display of the data display terminal 20 is a touch panel. In the following description, as a premise, it is assumed that a plurality of images captured by the data display terminal 20 are already stored in the data storage unit 16 of the family candidate display unit 24.

The data display terminal 20 first displays the home setting screen as shown in FIG. 4A. In the state where the home setting screen is displayed, the home location is set by the operation of the display by the user, and the information about the home location is transmitted to the family estimation device 10. Subsequently, as shown in FIG. 4 (b), the data display terminal 20 displays a message regarding the necessity of family setting (the message "Would you like to put together the family photos" in FIG. 4 (b)). To. When the user operates the display to instruct the family setting in response to the message, a face image relating to the family candidate is displayed as shown in FIG. 4 (c). The face image related to the family candidate is transmitted by the notification unit 14 of the family estimation device 10 and displayed on the display by the family candidate display unit 24. As described above, a plurality of facial images relating to family candidates for the subject of the child are displayed. When the user selects an image of the family in the display, a check mark is added to the image of the family selected by the user as shown in FIG. 4 (c). All the facial images selected by the user are transmitted to the tagging unit 15 of the family estimation device 10. Then, as shown in FIG. 4D, a family tag is attached to the image including the subject selected by the user, and the image is displayed separately from other images.

Next, with reference to FIG. 5, a series of processes of the family estimation method performed by the family estimation device 10 will be described. FIG. 5 is a flowchart showing an example of a series of processes of the family estimation method performed by the family estimation device 10.

First, the acquisition unit 11 acquires a plurality of images from the data display terminal 20 (step S1). The acquisition unit 11 acquires the user account ID, which is information indicating the imaging location of each image and information uniquely identifying the user of the data display terminal 20, together with the plurality of images. The acquisition unit 11 generates image file management information for each image based on the acquired information, and stores the image file management information and the image in the data storage unit 16.

Subsequently, the subject identification unit 12 performs preprocessing (step S2). In the preprocessing, the subject identification unit 12 rotates the image captured by the data display terminal 20 and stored in the data storage unit 16.

Subsequently, the subject identification unit 12 performs face detection processing (step S3). The subject identification unit 12 detects the face of the subject for each image after preprocessing stored in the data storage unit 16. The subject identification unit 12 detects the faces of all the subjects in the image by using the face detection technique which has been well known in the past. Further, the subject identification unit 12 estimates the age and gender of the detected face. The subject identification unit 12 cuts out the detected face image and stores (saves) it in the data storage unit 16 (step S4).

Subsequently, the subject identification unit 12 performs a grouping process (steps S5 to S8). The subject identification unit 12 first determines whether or not each face image stored in the data storage unit 16 is a face image of an adult subject, based on the estimated age of the face image file management information I2 of the data storage unit 16. (Step S5). The face image determined to be an adult face image in step S5 is subjected to face recognition processing between the adult face images (step S6), and is determined to be a child face image instead of an adult face image in step S5. With respect to the face image, face recognition processing is performed between the face images of the children (step S7). For example, the subject identification unit 12 extracts facial features that collate with each other and directly geometrically compares them, and when the similarity of the faces exceeds a predetermined threshold value, these faces are similar (same). Group as (the face of the subject) and identify multiple subjects. The threshold value for determining that the faces of a child's subject are similar when grouping the faces is set higher than the threshold value for determining that the faces of an adult subject are similar.

After the processing of steps S6 and S7, the face group ID is set for each specified subject by the subject identification unit 12, and the face group is registered (step S8). The subject identification unit 12 inputs the set face group ID into the above-mentioned face image file management information I2 (see FIG. 3B). Further, the subject identification unit 12 generates face group management information for each subject (for each face group ID), and stores the face group management information in the data storage unit 16.

Subsequently, the family candidate registration unit 13 determines whether or not there is an image (whether there is a home photograph) whose imaging location is within a predetermined range from the user's home (step S9). In step S9, when it is determined that there is an image whose imaging location is within a predetermined range from the user's home, the family candidate registration unit 13 registers a subject whose face is included in the image as a family candidate. (Step S10). Specifically, the family candidate registration unit 13 changes the family candidate flag associated with the acquired face group ID to "1" in the face group management information I3 (see FIG. 3C). A subject presumed to be a member of the user's family is registered as a family candidate.

Subsequently, the notification unit 14 displays an image relating to the subject which is a family candidate, and notifies the user of the family candidate (step S11). Then, when the user makes a selection (selection to the effect that he / she is a family member) in response to the notification, the tagging unit 15 adds a family tag to the image including the selected subject. (Step S13). If it is determined in step S9 that there is no image whose imaging location is within a predetermined range from the user's home, or if it is determined in step S12 that there is no selection by the user, the process ends. To do.

Next, the operation and effect of the family estimation device 10 according to the present embodiment will be described.

The family estimation device 10 according to the present embodiment is similar to the acquisition unit 11 that acquires a plurality of images and acquires information indicating the imaging location of each of the plurality of images, and a plurality of faces included in the plurality of images. The family candidate registration unit 13 that estimates the relationship between the subject identification unit 12 that identifies a plurality of subjects based on the degree and the plurality of subjects identified by the subject identification unit 12 according to the information indicating the imaging location. And.

In such a family estimation device 10, the relationship between a plurality of subjects whose faces are detected is estimated according to the information indicating the image capturing location. This makes it possible to presume that a plurality of subjects included in an image captured at home are members of a family. This makes it possible to effectively distinguish and organize family images from other images in consideration of the relationship between the subjects.

The family estimation device 10 includes a data storage unit 16 that stores home registration management information I4 (see FIG. 3D), which is information about the user's home location, and the family candidate registration unit 13 has an imaging location at home. It is presumed that all of the plurality of subjects whose images include faces within a predetermined range are members of the family. By using the information about the home location registered in advance, it is possible to estimate the members of the family living at home with high accuracy and easily.

The family estimation device 10 includes a notification unit 14 that notifies the user of a plurality of subjects estimated to form a family by the family candidate registration unit 13 as family candidates, and a subject selected by the user among the family candidates. The image is provided with a tagging unit 15 for adding a family tag to the image. As a result, the user is notified of the subjects that may form a family, and the image including the subject actually selected by the user who received the notification is distinguished from other images. Can be done. By adopting the configuration selected by the user, the family image can be more accurately and easily distinguished from other images.

The family candidate registration unit 13 separates the face of the adult subject and the face of the child subject into grouping, and sets a threshold value for determining that the faces of the child subject are similar to each other when grouping the faces of the child subject. The notification unit 14 notifies the user of the family candidate by displaying an image related to the family candidate so as to be higher than the threshold for determining that the faces are similar to each other when grouping the faces of the children. The number of images displayed for the family candidate is made larger than the number of images displayed for the family candidate for an adult subject. In general, it is more difficult to determine the similarity of a child's face than that of an adult's face, and even if they are not the same person, they are often determined to be similar (misrecognized). In this regard, the face of the child's subject is grouped separately from the face of the adult's subject, and the threshold value in the similarity determination is set higher than the value of the adult's subject to make the child's subject a higher value. It is possible to suppress misrecognition of the face. Furthermore, when receiving a selection from a user (selection of an image of a specific group), by increasing the number of images displayed for a child's subject, it is tentatively misrecognized at the time of grouping. However, the user can appropriately select the subjects that actually form the group. That is, it is possible to prevent the group composition tag from being attached to the images of the subjects constituting the group.

The block diagram used in the description of the above embodiment shows a block of functional units. These functional blocks (components) are realized by any combination of hardware and / or software. Further, the means for realizing each functional block is not particularly limited. That is, each functional block may be realized by one physically and / or logically coupled device, or directly and / or indirectly by two or more physically and / or logically separated devices. It may be physically (for example, wired and / or wirelessly) connected and realized by these plurality of devices.

For example, the family estimation device 10 in the above embodiment may function as a computer that performs processing of the family estimation device 10 in the above embodiment. FIG. 2 is a diagram showing an example of the hardware configuration of the family estimation device 10 according to the present embodiment. The family estimation device 10 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

In the following description, the word "device" can be read as a circuit, a device, a unit, or the like. The hardware configuration of the family estimation device 10 may be configured to include one or more of the devices shown in FIG. 1, or may be configured not to include some of the devices.

Each function in the family estimation device 10 is performed by loading predetermined software (program) on hardware such as the processor 1001 and the memory 1002, so that the processor 1001 performs an calculation, and the communication device 1004 communicates, the memory 1002, and the storage 1003. It is realized by controlling the reading and / or writing of the data in.

Processor 1001 operates, for example, an operating system to control the entire computer. The processor 1001 may be composed of a central processing unit (CPU) including an interface with a peripheral device, a control device, an arithmetic unit, a register, and the like.

Further, the processor 1001 reads a program (program code), a software module, and / or data from the storage 1003 and / or the communication device 1004 into the memory 1002, and executes various processes according to these. As the program, a program that causes a computer to execute at least a part of the operations described in the above-described embodiment is used. For example, the acquisition unit 11 of the family estimation device 10 may be realized by a control program stored in the memory 1002 and operated by the processor 1001, and may be realized for other functional blocks as well. Although it has been described that the various processes described above are executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. Processor 1001 may be mounted on one or more chips. The program may be transmitted from the network via a telecommunication line.

The memory 1002 is a computer-readable recording medium, and is composed of at least one such as a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), and a RAM (Random Access Memory). May be done. The memory 1002 may be referred to as a register, a cache, a main memory (main storage device), or the like. The memory 1002 can store a program (program code), a software module, or the like that can be executed to carry out the musical instrument sound recognition method according to the above embodiment.

The storage 1003 is a computer-readable recording medium, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, an optical magnetic disk (for example, a compact disk, a digital versatile disk, a Blu-ray). It may consist of at least one (registered trademark) disk), smart card, flash memory (eg, card, stick, key drive), floppy (registered trademark) disk, magnetic strip, and the like. The storage 1003 may be referred to as an auxiliary storage device. The storage medium described above may be, for example, a database, server, or other suitable medium that includes memory 1002 and / or storage 1003.

The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via a wired and / or wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, or the like.

The input device 1005 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives an input from the outside. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may have an integrated configuration (for example, a touch panel).

Further, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. Bus 1007 may be composed of a single bus, or may be composed of different buses between devices.

Further, the family estimation device 10 includes hardware such as a microprocessor, a digital signal processor (DSP: Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). It may be configured by, and a part or all of each functional block may be realized by the hardware. For example, processor 1001 may be implemented on at least one of these hardware.

Although the present invention has been described in detail above, it is clear to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented as an amended or modified embodiment without departing from the spirit and scope of the present invention determined by the description of the claims. Therefore, the description of the present specification is for the purpose of exemplification and does not have any limiting meaning to the present invention.

For example, the image of the family is described as a photograph, but the present invention is not limited to this, and the image may be another still image or moving image.

Further, as an example of the image estimation device, a family estimation device that estimates an image of a family has been described, but the present invention is not limited to this, and the image estimation device may estimate an image of a specific group other than the family. .. The specific group may be, for example, a place of activity to some extent, a colleague of a company, a classmate of a school, or the like. Further, when estimating an image of a specific group, the result of age estimation of the subject may be reflected. That is, for example, when estimating an image of a classmate at school, an image of a subject that seems to be of the same age as the classmate may be used as a group composition candidate in consideration of the age of the subject.

Further, although it has been described that the family estimation device 10 performs the face detection process, the present invention is not limited to this, and even if the face detection process is performed by an external face recognition engine and the detection result is used by the image estimation device, for example. Good.

Further, the image estimation device may determine an image to be notified to the user in consideration of the imaging date and time of a plurality of images. That is, the above-mentioned acquisition unit 11 acquires information indicating the imaging date and time of each of the plurality of images, and the notification unit 14 is only a family candidate whose number of images whose imaging location is within a predetermined range from the home is equal to or greater than a predetermined number. Is displayed, and among the images whose imaging location is within a predetermined range from the home, the images related to the family candidates are displayed to the user in order from the family candidate having the largest number of images having different shooting dates and times. May be good. The shooting date and time is associated with the image at the time of imaging by the data display terminal 20, for example. As a result, subjects that frequently exist at home (that is, subjects that are not visitors and are likely to be family members) can be preferentially selected as candidates for selection from users, and the user selects a family member. It is possible to increase the ease of use.

The order of the processing procedures, flowcharts, and the like of each aspect / embodiment described in the present specification may be changed as long as there is no contradiction. For example, the methods described herein present elements of various steps in an exemplary order, and are not limited to the particular order presented.

The input / output information and the like may be stored in a specific location (for example, a memory) or may be managed by a management table. Input / output information and the like can be overwritten, updated, or added. The output information and the like may be deleted. The input information or the like may be transmitted to another device.

The determination may be made by a value represented by 1 bit (0 or 1), by a boolean value (Boolean: true or false), or by comparing numerical values (for example, a predetermined value). It may be done by comparison with the value).

Each aspect / embodiment described herein may be used alone, in combination, or switched with execution. Further, the notification of the predetermined information (for example, the notification of "being X") is not limited to the explicit one, and may be implicitly (for example, the notification of the predetermined information is not performed). Good.

Software, whether referred to as software, firmware, middleware, microcode, hardware description language, or by any other name, is an instruction, instruction set, code, code segment, program code, program, subprogram, software module. , Applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, features, etc. should be broadly interpreted.

Further, software, instructions, and the like may be transmitted and received via a transmission medium. For example, the software uses wired technology such as coaxial cable, fiber optic cable, twisted pair and digital subscriber line (DSL) and / or wireless technology such as infrared, wireless and microwave to websites, servers, or other When transmitted from a remote source, these wired and / or wireless technologies are included within the definition of transmission medium.

The information, signals, etc. described herein may be represented using any of a variety of different techniques. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description are voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may be represented by a combination of.

The terms described herein and / or the terms necessary for understanding the present specification may be replaced with terms having the same or similar meanings.

The terms "system" and "network" as used herein are used interchangeably.

Further, the information, parameters, etc. described in the present specification may be represented by an absolute value, a relative value from a predetermined value, or another corresponding information. ..

The names used for the above parameters are not limited in any way. Further, mathematical formulas and the like using these parameters may differ from those expressly disclosed herein.

The terms "connected", "coupled", or any variation thereof, mean any direct or indirect connection or connection between two or more elements, and each other. It can include the presence of one or more intermediate elements between two "connected" or "combined" elements. The connection or connection between the elements may be physical, logical, or a combination thereof. As used herein, the two elements are by using one or more wires, cables and / or printed electrical connections, and, as some non-limiting and non-comprehensive examples, radio frequencies. By using electromagnetic energies such as electromagnetic energies with wavelengths in the region, microwave region and light (both visible and invisible) regions, they can be considered to be "connected" or "coupled" to each other.

The phrase "based on" as used herein does not mean "based on" unless otherwise stated. In other words, the statement "based on" means both "based only" and "at least based on".

Any reference to elements using designations such as "first", "second" as used herein does not generally limit the quantity or order of those elements. These designations can be used herein as a convenient way to distinguish between two or more elements. Thus, references to the first and second elements do not mean that only two elements can be adopted there, or that the first element must somehow precede the second element.

As long as "including", "comprising", and variations thereof are used within the scope of the present specification or claims, these terms are as comprehensive as the term "comprising". Intended to be targeted. Furthermore, the term "or" as used herein or in the claims is intended not to be an exclusive OR.

A plurality of devices are also included herein unless it is indicated in the context or technically that there is only one device.

10 ... Family estimation device, 11 ... Acquisition unit, 12 ... Subject identification unit, 13 ... Family candidate registration unit (estimation unit), 14 ... Notification unit, 15 ... Tagging unit, 16 ... Data storage unit (Activity location storage unit) ..

Claims (2)

An acquisition unit that acquires a plurality of images and acquires information indicating the imaging location of each of the plurality of images.
A subject identification unit that identifies a plurality of subjects by grouping the plurality of faces included in the plurality of images based on the degree of similarity.
An estimation unit that estimates the relationship between the plurality of subjects specified by the subject identification unit according to the information indicating the imaging location, and an estimation unit.
An activity location storage unit that stores information indicating the activity location of a specific group,
A notification unit that notifies the user of the plurality of subjects estimated to form the specific group by the estimation unit as group composition candidates, and a notification unit.
A tag-giving unit that attaches a group composition tag to an image including the subject selected by the user among the group composition candidates is provided.
The estimation unit estimates that each of the plurality of subjects whose face is included in the image whose imaging location is within a predetermined range from the activity location constitutes the specific group.
The subject identification part is
The grouping is performed separately for the face of the subject presumed to be the first age group who is older than the predetermined age and the face of the subject presumed to be the second age group who is less than the predetermined age.
The threshold value for determining that the faces of the subjects of the second age group are similar is larger than the threshold value for determining that the faces of the subjects of the first age group are similar. Higher
The notification unit
By displaying the image related to the group composition candidate, the user is notified of the group composition candidate, and the group composition candidate is notified.
An image estimation device that increases the number of images displayed for the group composition candidate for the subject of the second age group to be larger than the number of images displayed for the group composition candidate for the subject of the first age group. The acquisition unit acquires information indicating the shooting date and time of each of the plurality of images, and obtains information indicating the shooting date and time.
The notification unit
By displaying the image related to the group composition candidate, the user is notified of the group composition candidate, and the group composition candidate is notified.
Only the group composition candidates in which the number of images whose imaging location is within a predetermined range from the activity location is equal to or greater than a predetermined number are displayed, and the imaging location is within a predetermined range from the activity location. of the the number of shooting date different images often said population structure candidate in order to display an image according to the population structure candidate claim 1 Symbol placement image estimating apparatus.

Images