JP7267227B2 - Image processing device, method for operating image processing device, and program for operating image processing device - Google Patents

Description

The technology of the present disclosure relates to an image processing device, an image processing device operating method, and an image processing device operating program.

Conventionally, as described in Japanese Patent Application Laid-Open No. 2002-200310, trimming is performed to cut out a region including a specific person's face from an image. In Patent Document 1, face images of specific persons such as a user (photographer of an image) and related persons are registered in advance. By comparing the face included in the image to be trimmed with a pre-registered face image, the face of a specific person included in the image to be trimmed is extracted. In Patent Document 1, trimming is performed on an image to be trimmed in accordance with the position of the extracted face of a specific person. In Patent Document 1, an image to be trimmed is an image taken by the user himself/herself.

JP 2007-272685 A

By the way, recently, a service has been provided in which a plurality of users upload photographed images to an image distribution server on a network, and a user group to which access authority is given is permitted to browse the uploaded images, print the photographs, and the like. . A plurality of users constituting a user group are participants in a common event such as an athletic meet, a party, a wedding ceremony, a presentation, a concert, or a sports match.

According to such service providers, for example, for a first user who is one of a user group, an image taken by a second user different from the first user (hereinafter referred to as a second user image) , there is a demand to improve usability and attract many users to the service by providing recommended images. Also, for users constituting a couple, a couple, a family, a club, a circle, etc., an image containing not one specific person such as the first user himself, but a plurality of images such as the first user and his family, for example, is used. There is also a desire to similarly attract more users to the service by presenting an image with a higher added value that includes a specific person of a person as a recommended image.

The second user image is, of course, not an image taken for the first user. Therefore, the probability that the second user image includes not only the face of the specific person of the first user but also the face of the specific person of the second user is very high. In other words, it can be said that the second user image is an image of low value to the first user as it is. There has been a strong demand for a technique for presenting such a second user image to the first user as a recommended image with higher added value.

The technology of the present disclosure is an image processing device capable of presenting to a first user an image cut out from an image belonging to a second user based on related personal information of a first user, and an operating method of the image processing device. , to provide an operating program for an image processing apparatus.

In order to achieve the above object, an image processing apparatus of the present disclosure includes at least one processor. create related person information that is information on a related person related to the first user, acquire a second user image belonging to a second user different from the first user, and obtain a second user image that includes the face of the related person based on the related person information; A second user image containing the face of the related person is extracted, and a region containing the face of the related person is cut out from the second user image containing the face of the related person.

A method of operating an image processing apparatus according to the present disclosure creates related person information, which is information on related persons related to a first user, based on a first user image that belongs to the first user and includes a face. an information creation step; an image acquisition step of acquiring a second user image belonging to a second user different from the first user; and a second user image including the face of the related person based on the related person information. and a trimming step of cutting out an area including the face of the related person from the second user image including the face of the related person.

An operation program of an image processing apparatus of the present disclosure creates related person information, which is information of a related person related to a first user, based on a first user image that belongs to the first user and includes a face. An information creation unit, an image acquisition unit that acquires a second user image belonging to a second user different from the first user, and a second user image that includes the face of the related person based on related person information. The computer is caused to function as an extraction unit that extracts and a trimming unit that cuts out a region including the face of the related person from the second user image including the face of the related person.

According to the technology of the present disclosure, an image processing device capable of presenting to a first user an image cut out from an image belonging to a second user based on related personal information of a first user, and an image processing device of the image processing device An operating method and an operating program for the image processing apparatus can be provided.

1 is a diagram showing an image distribution server and various devices connected to the image distribution server; FIG. It is a figure which shows the inside of image DB and user information DB. FIG. 4 is a diagram for explaining the structure of a shared folder; FIG. FIG. 4 is a diagram showing an image file; It is a figure which shows user basic information. It is a figure which shows related person information. 2 is a block diagram showing a computer that constitutes an image distribution server and user terminals; FIG. 4 is a block diagram showing a processing section of a CPU of the image distribution server; FIG. 4 is a flow chart showing a procedure of information creation processing executed by an information creation unit; FIG. 4 is an explanatory diagram of information creation processing executed by an information creation unit; FIG. 4 is an explanatory diagram of information creation processing executed by an information creation unit; 4 is a flowchart showing a procedure of extraction processing executed by an extraction unit; FIG. 4 is an explanatory diagram of extraction processing executed by an extraction unit; FIG. 4 is an explanatory diagram of extraction processing executed by an extraction unit; It is a figure which shows a 1st face arrangement map. It is a figure which shows a 2nd face arrangement map. It is a figure which shows the 3rd face arrangement map. It is a figure which shows a 4th face arrangement map. 6 is a flow chart showing a procedure of trimming processing executed by a trimming unit; 6 is a flow chart showing a procedure of trimming processing executed by a trimming unit; FIG. 5 is an explanatory diagram of trimming processing executed by a trimming unit; FIG. 5 is an explanatory diagram of trimming processing executed by a trimming unit; FIG. 5 is an explanatory diagram of trimming processing executed by a trimming unit; FIG. 5 is an explanatory diagram of trimming processing executed by a trimming unit; FIG. 5 is an explanatory diagram of trimming processing executed by a trimming unit; FIG. 10 is a diagram showing the contents of recommended image information; 4 is a block diagram showing a processing unit of a CPU of the user terminal; FIG. It is a figure which shows an image display screen. FIG. 10 is a diagram showing a recommended image information display screen; 4 is a flow chart showing a processing procedure of an image distribution server; 4 is a flow chart showing a processing procedure of an image distribution server; FIG. 9 is an explanatory diagram of another example of trimming processing executed by the trimming unit; FIG. 10 is a diagram showing another example of the content of recommended image information; It is explanatory drawing of the information preparation process performed by the information preparation part in 2nd Embodiment. FIG. 11 is an explanatory diagram of another example of information creation processing executed by an information creation unit in the second embodiment; It is a figure which shows the aspect which produces related person information for every specific period. FIG. 12 is a diagram showing a third embodiment in which a score related to the face of a person other than a related person is calculated with a value opposite to the positive/negative value of the score related to the face of the related person; FIG. 38A is a diagram for explaining the effect of the third embodiment, and FIG. 38A shows a case where the score for the face of a person other than the related person is not calculated with a value opposite to the positive or negative value of the score for the face of the related person. Clipped images and FIG. 38B respectively show clipped images when the score for the face of a person other than the related person is calculated with the positive/negative value of the score for the face of the related person. FIG. 10 is a diagram showing a manner in which a face arrangement map to be used is selected according to the sum of the number of faces of related persons and the number of faces of persons other than related persons; FIG. 14 is a diagram showing an outline of a fourth embodiment for creating a face layout map based on an image with a face; 4 is a flowchart showing the procedure of map creation processing executed by a map creation unit; 4 is a flowchart showing the procedure of map creation processing executed by a map creation unit; FIG. 4 is an explanatory diagram of map creation processing executed by a map creation unit; FIG. 4 is an explanatory diagram of map creation processing executed by a map creation unit; 9 is a flow chart showing another example of the procedure of map creation processing executed by the map creation unit; FIG. 9 is an explanatory diagram of another example of map creation processing executed by the map creation unit;

[First embodiment]
In FIG. 1, a plurality of user terminals 11 are connected to an image distribution server 10, which is an example of an "image processing apparatus" according to the technology of the present disclosure. A user terminal 11 is a terminal possessed by each of a plurality of users 12 . The image distribution server 10 and user terminal 11 are connected via a network 13 . The network 13 is, for example, a WAN (Wide Area Network) such as the Internet or a public communication network. In the image distribution server 10, the users 12 are identified, for example, by user IDs (Identification Data) assigned to each user 12. FIG. Note that when one user ID is used by multiple users 12 , multiple users 12 corresponding to one user ID may be treated as one user 12 . Also, the user terminal 11 is, for example, a terminal device associated with each user ID. A plurality of user terminals 11 may be associated with one user ID.

The user terminal 11 has a function of transmitting/receiving the image file 27 (see FIG. 2) of the image 36 (see FIG. 4) via the network 13 . The user terminal 11 also has a function of displaying an image 36 . The user terminal 11, which is, for example, a personal computer, captures an image 36 captured by a photographing device such as a digital camera 15 or an image 36 obtained by reading a film photograph or the like with a scanner 16, and stores an image file 27 of the captured image 36 as The image is uploaded to the image distribution server 10 via the network 13. - 特許庁Also, the user terminal 11 is, for example, a smart phone, a tablet terminal, or the like that incorporates a photographing device, and uploads an image file 27 of an image 36 photographed by the photographing device to the image distribution server 10 via the network 13 .

An image database (hereinafter abbreviated as DB (Data Base)) server 20 and a user information DB server 21 are connected to the image distribution server 10 via a network (not shown) such as a LAN (Local Area Network). there is The image distribution server 10 transmits the image file 27 from the user terminal 11 to the image DB server 20 . The image DB server 20 has an image DB 22 . The image DB server 20 stores and manages the image files 27 from the image distribution server 10 in the image DB 22 . The image DB server 20 also transmits the image files 27 accumulated in the image DB 22 to the image distribution server 10 in response to a request from the image distribution server 10 .

The image distribution server 10 transmits user information 30 (see FIG. 2) that is information about the user 12 to the user information DB server 21 . User information DB server 21 has user information DB 23 . The user information DB server 21 accumulates and manages the user information 30 from the image distribution server 10 in the user information DB 23 . Further, the user information DB server 21 transmits related person information 32 to the image distribution server 10, which is included in the user information 30 accumulated in the user information DB 23, in response to a request from the image distribution server 10. FIG. Note that the image distribution server 10 may be connected to the image DB server 20 and the user information DB server 21 via the network 13 . Also, the image DB server 20 and the user information DB server 21 may be one DB server having both the image DB 22 and the user information DB 23 . Furthermore, the image distribution server 10 may include an image DB server 20 and a user information DB server 21 . In other words, the “image processing device” according to the technology of the present disclosure may be configured with the image distribution server 10 , the image DB server 20 , and the user information DB server 21 .

As shown in FIG. 2, the image DB 22 is provided with a personal folder 25 and a shared folder 26 . The personal folder 25 is a folder unique to one user 12 that is individually associated with each user 12 . For this reason, personal folders 25 are provided for the number of users 12 . An image file 27 uploaded from the user terminal 11 of each user 12 is stored in the personal folder 25 . The image file 27 contains an image 36 as will be described later. Therefore, an image 36 belonging to each user 12 is stored in the personal folder 25 . In other words, the “image belonging to the user” is the image 36 associated with each user 12, for example, the image 36 included in the image file 27 transmitted from the user terminal 11 of the user 12 to the image distribution server 10. be. In other words, it is the image 36 included in the image file 27 received by the image distribution server 10 from the user terminal 11 . The “image belonging to the user” includes, for example, the image 36 photographed by the user 12 with the photographing device and the image 36 read by the scanner 16 by the user 12 . In addition, the “images belonging to the user” include images 36 transferred to the user 12 from other users 12 such as friends and family, images 36 downloaded by the user 12 from Internet sites, and the like.

A user ID is associated with the personal folder 25 included in the image DB 22, and is indicated by, for example, "(U1)" after "personal folder". "Personal folder (U1)" means the personal folder 25 of the user 12 whose user ID is "U1". That is, the “image belonging to the user” indicates the image 36 associated with the user ID of each user 12 .

As also shown in FIG. 3 , the shared folder 26 is a folder shared by a user group 28 composed of a plurality of users 12 , unlike the personal folder 25 . Therefore, the shared folder 26 is associated with the user IDs of the users 12 who share the shared folder 26 . Only the user 12 associated with the user ID is given access authority to the shared folder 26 . Image files 27 of images 36 from a plurality of users 12 are stored in the shared folder 26 . A shared folder 26 may be created for each event, for example. In addition, the shared folder 26 may be manually created by any one of the users 12 for each event, or may be created by analyzing the images 36 sent from the users 12 so that each image 36 It may be created automatically by identifying the corresponding event. Multiple users 12 are associated with the shared folder 26 . A plurality of users 12 associated with the shared folder 26 may be set by, for example, a specific administrative user among the users 12 associated with the shared folder 26, or an event or image 36 associated with the shared folder 26 may be set. Users 12 with images 36 similar to may be automatically set. Events include, for example, athletic meets, parties, weddings, recitals, concerts, sports games, and the like.

In FIG. 3, a plurality of users 12 having user IDs "U5", "U10", "U15", etc. who participated in the event "wedding ceremony 1" share a shared folder named "shared folder (wedding ceremony 1)". 26 shows how an image file 27 of an image 36 belonging to itself is uploaded. Note that the shared folder 26 is created by, for example, one user 12 out of the plurality of users 12 forming the user group 28 . A shared folder 26 may be created by the organizer of the event. Although the name of the shared folder 26 is an event name associated with the shared folder 26 by the user 12 or the image distribution server 10 in this embodiment, the name is not limited to this. An identification number or a management number that identifies each shared folder 26 may be used as the name of the shared folder 26 .

User basic information 31 and related person information 32 are stored as user information 30 in the user information DB server 21 . The user basic information 31 is basic information of each user 12 . The related person information 32 is information of related persons 45 (see FIG. 6) related to each user 12 . The user basic information 31 and related person information 32 and the personal folder 25 are associated with each other by the user ID. It should be noted that the "user basic information (U1)" and "related person information (U1)" are similar to the "personal folder (U1)" and refer to the user basic information 31 and related person information of the user 12 whose user ID is "U1". It means information 32.

As shown in FIG. 4, image file 27 includes image information 35, image 36, and a user ID. The image information 35 includes items such as an image ID for identifying each image 36, the date and time when the image 36 was taken, the location of the image taken, the resolution, and the tag. The shooting location field records the latitude and longitude detected by the GPS (Global Positioning System) function of the shooting device. The item of resolution records horizontal and vertical dpi (dots per inch) of the pixels of the imaging device mounted on the imaging device. In the tag item, a word that simply expresses the subject of the image 36 is recorded. Tags may be obtained by manual input by user 12 and/or analysis of image 36 . The image information 35 includes, in addition to these, the model of the photographing device that photographed the image 36, the number of pixels in the horizontal and vertical directions of the image 36, the exposure value, the ISO (International Organization for Standardization) sensitivity, the focal length, and the number of flashes. It has various items such as presence/absence.

As shown in FIG. 5, the basic user information 31 includes items such as a user ID, name, sex, date of birth, age, location, shared folder, and face image of the user 12 . The prefecture in which the user 12 resides is recorded in the item of whereabouts. A user ID is registered in the shared folder item, and the shared folder 26 to which the user has access authority is recorded. A face image 40 of the user 12 is recorded in the face image item. The face image 40 of the user 12 is registered by the user 12 himself/herself, for example. As used herein, the term "face" refers to regions of the human body including, for example, the head, ears, eyes, nose, mouth, cheeks, chin, and the like.

As shown in FIG. 6, the related person information 32 includes the user ID and the face image 40 of the related person 45. As shown in FIG. The face image 40 of the related person 45 may be one representative image, or may be multiple images.

Related persons 45 include a main person 45M and a sub-person 45S. The main person 45M is a person who is relatively highly related to the user 12 with the user ID. The sub-person 45S is a person whose relevance to the user 12 with the user ID is lower than that of the main person 45M. The mode of sorting the related person 45 into the main person 45M and the sub-person 45S is an example of a mode of "giving each related person a related rank indicating the degree of relevance to the first user" according to the technology of the present disclosure. be.

In FIG. 7, the computer that constitutes the image distribution server 10 and the user terminal 11 basically have the same configuration, and include a storage device 50, a memory 51, a CPU (Central Processing Unit) 52, a communication section 53, a display 54, and a An input device 55 is provided. These are interconnected via bus lines 56 .

The storage device 50 is a hard disk drive that is built in the computer that constitutes the image distribution server 10 and the user terminal 11 or that is connected via a cable or network. Alternatively, the storage device 50 is a disk array in which a plurality of hard disk drives are connected. The storage device 50 stores a control program such as an operating system, various application programs (hereinafter abbreviated as AP (Application Program)), and various data associated with these programs. A solid state drive may be used instead of the hard disk drive. At least one of the image DB server 20 and the user information DB server 21 may be included in the storage device 50 .

The memory 51 is a work memory for the CPU 52 to execute processing. The CPU 52 loads a program stored in the storage device 50 into the memory 51 and executes processing according to the program, thereby comprehensively controlling each section of the computer.

The communication unit 53 is a network interface that controls transmission of various information via the network 13 or the like. The display 54 displays various screens. Various screens are provided with an operation function by a GUI (Graphical User Interface). A computer that constitutes the image distribution server 10 and the user terminal 11 accepts input of operation instructions from the input device 55 through various screens. The input device 55 is a keyboard, mouse, touch panel, microphone for voice input, or the like.

In the following description, each part of the computer that constitutes the image distribution server 10 is denoted by a suffix "A", and each part of the computer that constitutes the user terminal 11 is denoted by a suffix "B". distinguish between

In FIG. 8 , the user terminal 11 transmits an image distribution request 57 to the image distribution server 10 . Image delivery request 57 includes, for example, the user ID of user 12 . In response to the image distribution request 57 , the image distribution server 10 distributes the image file 27 in the personal folder 25 specified by the user ID to the user terminal 11 that sent the image distribution request 57 . Further, when there is a shared folder 26 designated by the user ID, the image distribution server 10 sends the image file 27 in the shared folder 26 together with the image file 27 in the personal folder 25 to the image distribution request 57. is delivered to the user terminal 11 that is the source of the transmission. Note that the number of image files 27 to be distributed to the user terminal 11 may be one.

Also, the user terminal 11 transmits a recommended image distribution request 58 to the image distribution server 10 . The recommended image distribution request 58 includes the user ID of the user 12 and the name of the shared folder 26 requesting the recommended image. The recommended image distribution request 58 is sent from the user terminal 11 by the user 12 who sends the recommended image distribution request based on the images 36 belonging to the users 12 other than the user 12 in the shared folder 26 specified by the name. This is a request for presenting an image to the image distribution server 10 . In the present embodiment, the recommended image is the image 36 belonging to the user 12 other than the user 12 who sent the recommended image distribution request 58 in the shared folder 26 specified by the recommended image distribution request 58. 25 are clipped images 110 (see FIG. 25) that are clipped based on related person information 32 corresponding to user 12 who has transmitted recommended image distribution request 58. FIG. In response to the recommended image distribution request 58 , the image distribution server 10 distributes recommended image information 59 including the recommended image clipped image 110 to the user terminal 11 that sent the recommended image distribution request 58 .

Requests (for example, the image distribution request 57 and the recommended image distribution request 58 ) transmitted from the user terminal 11 to the image distribution server 10 may include the terminal ID of the user terminal 11 . For example, by receiving the terminal ID, the image distribution server 10 sends an image (for example, the image file 27) or information (for example, the recommended image information 59) corresponding to the request to the user terminal 11 corresponding to the terminal ID. deliver. Here, the terminal ID is not limited to an ID unique to the user terminal 11 such as a MAC (Media Access Control) address, but may be an IP (Internet Protocol) address, port number, or other ID that identifies the user terminal 11 in data transmission/reception. including.

Here, the user 12 who has transmitted the recommended image distribution request 58 from the user terminal 11 is an example of the "first user" according to the technology of the present disclosure. Also, the user 12 other than the user 12 who transmitted the recommended image distribution request 58 from the user terminal 11 is an example of the "second user different from the first user" according to the technology of the present disclosure. In the following description, the first user is denoted by reference numeral 12F, and the second user is denoted by reference numeral 12S. An image 36 belonging to the first user 12F, which has an image file 27 in the personal folder 25 of the first user 12F, is referred to as a first user image 36F. Further, an image 36 that has an image file 27 in the personal folder 25 of the second user 12S and belongs to the second user 12S is referred to as a second user image 36S.

An operating program 65 is stored in the storage device 50A of the image distribution server 10 . The operation program 65 is an AP for causing the server computer that configures the image distribution server 10 to function as an "image processing apparatus" according to the technique of the present disclosure. That is, the operation program 65 is an example of the "image processing apparatus operation program" according to the technology of the present disclosure. In addition to the operating program 65, a face layout map set 66 is also stored in the storage device 50A.

When the operation program 65 is activated, the CPU 52A of the image distribution server 10 cooperates with the memory 51 and the like to create an information generation unit 70, a request reception unit 71, an image acquisition unit 72, an information acquisition unit 73, an extraction unit 74, It functions as a read/write (hereinafter abbreviated as RW (Read Write)) control section 75 , a trimming section 76 and a distribution control section 77 . The CPU 52A is an example of a "processor" according to the technology of the present disclosure.

The information creating unit 70 creates related person information 32 . The information creation unit 70 transmits the first image acquisition request 80 to the image DB server 20 . The first image acquisition request 80 requests acquisition of the image file 27 in the personal folder 25 of the first user 12F for which the related person information 32 has not been created. Further, the first image acquisition request 80 is issued when a predetermined time (hereinafter referred to as a first set time) has passed since the related person information 32 was created last time, or when a predetermined number of images (hereinafter referred to as a (referred to as the first number) of image files 27 are newly transmitted, in at least one case, acquisition of the image files 27 in the personal folder 25 of the first user 12F is requested. The first image acquisition request 80 includes the user ID of the first user 12F for whom the related personal information 32 has not been created, the user ID of the first user 12F for whom the first set time has passed since the previous related personal information 32 was created, Alternatively, the user ID of the first user 12F who has sent the first number of image files 27 is included. The timing of requesting acquisition of the image file 27 in the personal folder 25 of the first user 12F for which the related person information 32 has not been created by the first image acquisition request 80 is, for example, This is the timing when the number of image files 27 stored in is equal to or greater than a predetermined number. Alternatively, it may be the timing when the first user 12F installs the image viewing AP 115 (see FIG. 27) on the user terminal 11. FIG. Note that the first set time is, for example, one month, half a year, one year, or the like. Also, the first number is, for example, a natural number of 1 or more.

The image DB server 20 reads from the image DB 22 the image file 27 including the first user image 36F in the personal folder 25 of the first user 12F in response to the first image acquisition request 80, and distributes the read image file 27 as an image. Send to server 10 . The information creating unit 70 acquires the image file 27 including the first user image 36F, which is transmitted from the image DB server 20 in response to the first image acquisition request 80. FIG. The information creation unit 70 creates the related person information 32 based on the first user image 36F of the acquired image file 27 . The information creating unit 70 transmits the created related person information 32 to the user information DB server 21 . If the related personal information 32 has not been created, the user information DB server 21 newly registers the related personal information 32 from the information creating unit 70 in the user information DB 23 . On the other hand, if the related personal information 32 has already been created, the user information DB server 21 rewrites the related personal information 32 stored in the user information DB 23 with the related personal information 32 from the information creation unit 70 .

The request reception unit 71 receives various requests from the user terminal 11 . Upon receiving various requests from the user terminal 11 , the request receiving portion 71 outputs instructions corresponding to the various requests to the image obtaining portion 72 or the information obtaining portion 73 . For example, when receiving the image distribution request 57 or the recommended image distribution request 58, the request reception unit 71 sends an instruction corresponding to the image distribution request 57 or the recommended image distribution request 58 to the image acquisition unit 72 or the information acquisition unit 73. Output.

The image acquisition unit 72 transmits a second image acquisition request 81 to the image DB server 20 in response to instructions corresponding to various requests output from the request reception unit 71 . The second image acquisition request 81 requests acquisition of at least one of the image file 27 in the personal folder 25 and the image file 27 in the shared folder 26 specified by the user ID of the image distribution request 57 or the recommended image distribution request 58. It is something to do.

The image DB server 20 reads the image file 27 of the personal folder 25 and the image file 27 of the shared folder 26 in response to the second image acquisition request 81 from the image DB 22 and transmits the read image file 27 to the image distribution server 10 . The image acquisition unit 72 acquires the image file 27 transmitted from the image DB server 20 in response to the second image acquisition request 81 . The image acquisition section 72 outputs the acquired image file 27 to the distribution control section 77 . The image acquisition unit 72 also outputs the image files 27 in the shared folder 26 among the acquired image files 27 to the extraction unit 74 . Note that if there is no shared folder 26 specified by the user ID of the image distribution request 57 , only the image file 27 in the personal folder 25 is acquired by the image acquisition unit 72 .

The shared folder 26 also includes an image file 27 of a second user image 36S belonging to a user 12 other than the first user 12F, that is, the second user 12S. Therefore, by acquiring the image file 27 in the shared folder 26, the image acquiring unit 72 acquires the second user image 36S belonging to the second user 12S.

The information acquisition unit 73 transmits an information acquisition request 82 to the user information DB server 21 in response to instructions corresponding to various requests output from the request reception unit 71 . The information acquisition request 82 requests acquisition of the related person information 32 of the user 12 having the user ID included in the recommended image distribution request 58, that is, the first user 12F. The user information DB server 21 reads the related person information 32 corresponding to the information acquisition request 82 from the user information DB 23 and transmits the read related person information 32 to the image distribution server 10 . The information acquisition unit 73 acquires the related person information 32 transmitted from the user information DB server 21 in response to the information acquisition request 82 . The information acquisition unit 73 outputs the acquired related person information 32 to the extraction unit 74 and the trimming unit 76 .

Based on the related person information 32 of the first user 12F, the extraction unit 74 extracts the related person 45 of the first user 12F from the image 36 included in the image file 27 in the shared folder 26 specified by the recommended image delivery request 58. The second user image 36S including the face 90R (see FIG. 14) is extracted. The extraction section 74 outputs the extracted second user image 36S to the trimming section 76 .

The RW control unit 75 controls storage of various information in the storage device 50A and reading of various information in the storage device 50A. For example, the RW control section 75 reads the face arrangement map set 66 from the storage device 50A and outputs the read face arrangement map set 66 to the trimming section 76 . The face arrangement map set 66 is composed of a first face arrangement map 85a, a second face arrangement map 85b, a third face arrangement map 85c, and a fourth face arrangement map 85d. Note that the first face layout map 85a, the second face layout map 85b, the third face layout map 85c, and the fourth face layout map 85d may be collectively referred to as the face layout map 85 below.

Using the face arrangement map 85, the trimming unit 76 trims the related person of the first user 12F from the second user image 36S including the face 90R of the related person 45 of the first user 12F extracted by the extraction unit 74. A region containing 45 faces 90R is cut out. The trimming unit 76 outputs the clipped image 110 (see FIG. 25) to the distribution control unit 77 . Note that the trimming unit 76 may enlarge the clipped image 110 to form a clipped image 110E (see FIG. 25), and output the clipped image 110E to the distribution control unit 77. FIG.

The distribution control unit 77 controls distribution of images or information output from the image acquisition unit 72 or the trimming unit 76 and corresponding to various requests received by the request reception unit 71 to the user terminal 11 . The distribution control unit 77 controls, for example, distribution of the image file 27 corresponding to the image distribution request 57 output from the image acquisition unit 72 to the user terminal 11 that is the transmission source of the image distribution request 57 . Further, the distribution control unit 77 controls distribution of the recommended image information 59 corresponding to the recommended image distribution request 58 output from the trimming unit 76 to the user terminal 11 that is the transmission source of the recommended image distribution request 58 . conduct. The recommended image information 59 includes, for example, a clipped image 110E. A mode of distributing this recommended image information 59 to the user terminal 11 is an example of a mode of "presenting the clipped image to the first user" according to the technology of the present disclosure.

9 to 11 show the procedure of information creation processing (step ST100A) for creating related person information 32 executed in information creation section . The information creating section 70 first transmits the first image acquisition request 80 to the image DB server 20 (step ST1001). Then, information creating section 70 acquires image file 27 in personal folder 25 transmitted from image DB server 20 in response to first image acquisition request 80 (step ST1002).

The information creating unit 70 uses face recognition technology to extract a face image 91 including a face 90 from the first user image 36F included in the obtained image file 27 (step ST1003). The image with a face 91 extracted in step ST1003 is an example of a "first user image that satisfies a predetermined setting condition" according to the technology of the present disclosure.

Next, information creating section 70 recognizes the person with face 90 included in image with face 91 (step ST1004). In other words, the information creating unit 70 classifies the images 91 with faces according to persons. Recognition or classification of a person is performed, for example, by the following method. That is, the feature amounts of all the faces 90 included in the face-containing image 91 are calculated, and the similarity of the calculated feature amounts is calculated for all the faces 90 by round-robin. Then, faces 90 whose similarity is within a predetermined range are recognized or classified as faces 90 of the same person. FIG. 10 shows an example in which persons such as "P1", "P2", "P3", . . . are recognized. In the present embodiment, after extracting the image with face 91 from the first user image 36F, the person with the face 90 included in the image with face 91 is recognized, but the present invention is not limited to this. Without extracting the image 91 with a face, all of the first user images 36F may be recognized or classified as persons.

For each recognized person, the information creating unit 70 calculates the number of images with faces 91 including the face 90 of each person, or images with faces including the face 90 of each person in the first user image 36F included in the personal folder 26. A ratio of 91 is calculated (step ST1005). Then, according to the selection condition 96, the related person information 32 of the first user 12F is created (step ST1006). The selection condition 96 is information indicating the degree of association with the first user 12F. Also, the selection conditions 96 may include different conditions according to the rank of association with the first user 12F. For example, the first selection conditions 93 for selecting the main person 45M and the second 2 selection conditions 94 . The first selection condition 93 is a condition that is relatively stricter than the second selection condition 92, for example, a condition that requires a relatively large number of sheets or a large ratio. The selection condition 96 is, for example, a condition related to the number of face images 91 including the face 90 of each person, as shown in Table 92 of FIG.

The information generator 70 selects the main person 45M from among the recognized persons according to the first selection condition 93 and registers the face image 40 of the selected main person 45M in the related person information 32 . The first selection condition 93 is that the person who has the largest number of face images 91 including the face 90 is set as the main person 45M. Therefore, the information creating unit 70 selects the person who has the largest number of face images 91 including the face 90 as the main person 45M. FIG. 10 shows an example in which the person "P1" having the largest number of face images 91 including the face 90 is selected as the main person 45M.

The information creating unit 70 selects the sub-person 45S from among the recognized persons according to the second selection condition 94, and registers the face image 40 of the selected sub-person 45S in the related person information 32. FIG. The second selection condition 94 is that the sub-person 45S is the person who has the second largest number of face images 91 including the face 90 after the main person 45M. Therefore, the information creating unit 70 selects the person who has the second largest number of face images 91 including the face 90 after the main person 45M as the sub person 45S. FIG. 10 shows an example in which a person "P2" having the second largest number of face images 91 including a face 90 after the person "P1" is selected as the sub-person 45S.

In the present embodiment, two or more types of related persons 45 (main person 45M and sub-person 45S) with different degrees of relatedness are created as the related person information 32 of the user, but the present invention is not limited to this. As the related person information 32 of the user, only one type of related person 45 (for example, the main person 45M) having a certain degree of relatedness or higher may be created, or three or more types of related ranks having different degrees of relatedness may be created. A related person 45 may be selected.

The information creating unit 70 may also calculate the ratio of the face images 91 including the face 90 of each person in the first user image 36</b>F included in the personal folder 26 . As a modified example, the information creating unit 70 calculates a first ratio and a second ratio as the ratio of the face-containing image 91 including the face 90 of each person, as shown in Table 95 of FIG. 11 . The first ratio is the ratio of the number of face images 91 including each person's face 90 to the total number of face images 91 extracted in step ST1003. For example, consider a case where the total number of images 91 with faces is 1000 sheets. In this case, if there are 200 face images 91 including the face 90 of the person "P1", the first ratio of the person "P1" is 200/1000=0.20 as shown in Table 95. Similarly, the first ratio of the person “P2” with 150 face images 91 is 0.15, and the first ratio of the person “P3” with 80 face images 91 is 0.08. , and the first ratio of the person "P4" having 60 face images 91 is 0.06.

The second ratio is the ratio of the number of face images 91 including the face 90 of each person to the number of face images 91 including the face 90R of the main person 45M. The number of face images 91 including the face 90R of the person "P1" who is the main person 45M is 200 sheets. Therefore, the second ratio for person “P1” is 100/100=1.0, the second ratio for person “P2” is 0.75, and the second ratio for person “P3” is 100/100=1.0, as shown in Table 95. The second proportion is 0.4, and the second proportion for person "P4" is 0.3. Note that the face-containing image 91 including the face 90R of the main person 45M is also an example of the "first user image that satisfies a predetermined setting condition" according to the technology of the present disclosure.

The first selection condition 93 stores, for example, the first threshold for the first ratio, and the second selection condition 94 stores, for example, the second threshold for the second ratio. The information creating unit 70 selects a person whose first ratio is equal to or higher than the first threshold as the main person 45M, selects a person whose second ratio is equal to or higher than the second threshold as the sub-person 45S, and registers them as related person information 32. You may In FIG. 11, 0.2 is set as the first threshold, and 0.4 is set as the second threshold. Persons "P2" and "P3" who are 0.4 or more are selected as sub-persons 45S. The selection condition 96 is stored in the storage device 50 , read by the RW control unit 75 and output to the information creation unit 70 . Also, the first selection condition 93 and the second selection condition 94 may be conditions corresponding to different ratios as shown in FIG. 11, or may be conditions corresponding to different numbers of sheets.

The information creation unit 70 transmits the related person information 32 created in this way to the user information DB server 21 (step ST1007). Thus, the information creation processing by the information creation unit 70 is completed.

12 to 14 show the procedure of extraction processing (step ST300) for extracting the second user image 36S including the face 90R of the related person 45 of the first user 12F, which is executed by the extraction unit 74. FIG. Extraction unit 74 first extracts second user image 36S from image file 27 in shared folder 26 specified by recommended image distribution request 58 (step ST3001). More specifically, the extraction unit 74 extracts the user 12 ( That is, the user 12 other than the first user 12F) is identified as the second user 12S. Then, the image 36 of the image file 27 including the identified user ID of the second user 12S is extracted as the second user image 36S.

In FIG. 13, the user 12 with the user ID "U1" is targeted for the shared folder 26 (named "Alumni Association 1") of the user group 28 consisting of two users 12 with the user IDs "U1" and "U2". A case where a recommended image distribution request 58 is transmitted from the user terminal 11 of . In this case, the user 12 with the user ID "U1" is the first user 12F, and the user 12 with the user ID "U2" is the second user 12S. Therefore, the extraction unit 74 extracts the image 36 belonging to the user 12 with the user ID "U2" from the image file 27 in the shared folder 26 as the second user image 36S.

The extracting unit 74 extracts an image with a face 91 including the face 90 from the second user image 36S extracted in step ST3001, as in step ST1003 of the information creation process (step ST3002). Next, the extraction unit 74 recognizes or classifies the person with the face 90 included in the image with a face 91 (step ST3003), as in step ST1004 of the information creation process. FIG. 14 shows an example of recognition of persons such as "P1", "P2", "P3", "P4", . Note that in the present embodiment, after extracting the image with face 91 from the second user image 36S, the person with the face 90 included in the image with face 91 is recognized, but the present invention is not limited to this. Without extracting the image with face 91, the person may be recognized or classified for all of the second user images 36S.

Based on the person recognition result in step ST3003 and the related person information 32 of the first user 12F from the information acquisition unit 73, the extraction unit 74 extracts the face of the related person 45 of the first user 12F from the image with face 91. An image with face 91R including 90R is extracted (step ST3004). The image with face 91R is subject to trimming processing in the trimming unit 76, which will be described later. Therefore, in the following description, the image with a face 91R is referred to as the target image 91R.

The extraction unit 74 determines whether or not there is a face 90 matching the face 90R of the related person 45 of the first user 12F among the faces 90 of the persons recognized in step ST3003. If there is a face 90 that matches the face 90R of the related person 45 of the first user 12F among the faces 90 of the persons recognized in step ST3003, the extraction unit 74 extracts all faces that include the face 90. The presence image 91 is extracted. Then, the extracted image with face 91 is treated as the target image 91R. Note that the extracting unit 74 extracts, from the extracted face-containing images 91, the face-containing images 91 that include two or more faces 90 that match the face 90R of the related person 45 of the first user 12F as the target image 91R. may

In FIG. 14, based on the related person information 32 of the first user 12F whose user ID is "U1", the image with face 91, which is the second user image 36S, of the first user 12F whose user ID is "U1", It illustrates how to extract a target image 91R including the faces 90R of at least two related persons 45. FIG.

The extraction section 74 outputs the target image 91R thus extracted to the trimming section 76 (step ST3005). Thus, the extraction processing by the extraction unit 74 is completed.

As shown in FIG. 15, the first face placement map 85a represents appropriate placement positions of the faces 90 of two persons in the image 36. FIG. The first face layout map 85a is composed of a plurality of divided cells 100 arranged horizontally and vertically. As shown in "divisional cell coordinates" in table 101, 1024 divisional cells 100 are arranged horizontally and vertically, for example.

Each divided cell 100 is assigned an evaluation value between 0 and 1. FIG. Specifically, as shown by the hatching gradation, a relatively high value is assigned to the divided cells 100 in which the face 90 should be placed compared to the divided cells 100 in which the face 90 should not be placed. ing. For example, an evaluation value of 1 is assigned to the divided cell 100 at coordinates (200, 300) in which the face 90 is to be placed. On the other hand, 0 is assigned as an evaluation value to the divided cell 100 at the coordinates (1023, 1023) in which the face 90 should not be placed.

Similarly, FIG. 16 shows a second face placement map 85b representing appropriate placement positions of three human faces 90 within image 36. FIG. FIG. 17 also shows a third face placement map 85c representing appropriate placement positions of faces 90 of four or more persons in image 36. FIG. Further, FIG. 18 shows a fourth face placement map 85d representing appropriate placement positions of a person's face 90 within the image 36. FIG. In this way, a plurality of face layout maps 85 are prepared for each number of faces 90, including a first face layout map 85a, a second face layout map 85b, a third face layout map 85c, and a fourth face layout map 85d. there is

The first face arrangement map 85a is created based on the following equations (1) and (2) according to the Gaussian function of Gaussian distribution. Equation (1) represents the evaluation value for the horizontal direction, and Equation (2) represents the evaluation value for the vertical direction. The evaluation value assigned to each divided cell 100 of the first face arrangement map 85a is the product of the horizontal evaluation value and the vertical evaluation value obtained by the formulas (1) and (2).

Note that C is the coefficient of the Gaussian function, μ is the mean, and σ is the standard deviation ( ^σ2 is the variance). "21" attached below C, μ, and σ in equation (1) means the first face counted from the left of the faces 90 of the two persons. Similarly, "22" means the second from the left of the faces 90 of the two persons. The values of C, μ, and σ are, for example, C ₂₁ =C ₂₂ =0.9, C=1, μ ₂₁ =0.3, μ ₂₂ =0.7, μ=1/3, σ ₂₁ =σ ₂₂ = σ = 0.15.

The second face arrangement map 85b is created based on Equations (3) and (4) below. Equation (3) represents the evaluation value for the horizontal direction, and Equation (4) represents the evaluation value for the vertical direction. Equation (4) is the same as Equation (2). The evaluation value assigned to each divided cell 100 of the second face arrangement map 85b is the product of the horizontal evaluation value and the vertical evaluation value obtained by the formulas (3) and (4).

"31" attached below C, μ, and σ in equation (3) means the first one counted from the left among the faces 90 of the three persons. Similarly, "32" means the second from the left of the three human faces 90, and "33" means the third from the left of the three human faces 90. be. The values of C, μ, and σ are, for example, C ₃₁ =C ₃₂ =C ₃₃ =0.9, C=1, μ ₃₁ =0.25, μ ₃₂ =0.5, μ ₃₃ =0.75, μ = 1/3, σ ₃₁ = σ ₃₂ = σ ₃₃ = 0.1, σ = 0.15.

Also, the third face arrangement map 85c is created based on the following equations (5) and (6). Equation (5) represents the evaluation value for the horizontal direction, and Equation (6) represents the evaluation value for the vertical direction. Equation (6) is the same as Equations (2) and (4). The evaluation value assigned to each divided cell 100 of the third face arrangement map 85c is the product of the horizontal evaluation value and the vertical evaluation value obtained by the formulas (5) and (6).

"4" attached below C, μ, and σ in equation (5) means that it relates to faces 90 of four or more persons. Values of C, μ, and σ are, for example, C ₄ =C=1, μ ₄ =0.5, μ=1/3, σ ₄ =0.5, and σ=0.15.

Also, the fourth face arrangement map 85d is created based on the following equations (7) and (8). Equation (7) represents the evaluation value for the horizontal direction, and Equation (8) represents the evaluation value for the vertical direction. Equation (8) is the same as Equations (2), (4) and (6). The evaluation value assigned to each divided cell 100 of the fourth face arrangement map 85d is the product of the horizontal evaluation value and the vertical evaluation value obtained by the formulas (7) and (8).

"1" attached below C, μ, and σ in equation (7) means that it relates to the face 90 of one person. The values of C, μ, and σ are, for example, C ₁ =C=1, μ ₁ =0.5, μ=1/3, σ ₁ =σ=0.15.

Thus, the face arrangement map 85 is created based on equations (1) to (8) according to Gaussian functions. Therefore, the face arrangement map 85 has a symmetric distribution of evaluation values.

19 to 25 show the procedure of the trimming process (step ST400) for cutting out the area including the face 90R of the related person 45 from the target image 91R, which is executed by the trimming unit 76. FIG. The trimming unit 76 first creates a face layout map 85 corresponding to the number of faces 90R of the related person 45 included in the target image 91R for one of the target images 91R extracted by the extraction unit 74. , face arrangement map set 66 (step ST4001).

Specifically, when the number of faces 90R of the related person 45 included in the target image 91R is two, the trimming unit 76 selects the first face layout map 85a. When the number of faces 90R of the related person 45 included in the target image 91R is three, the second face arrangement map 85b is selected. If the number of faces 90R of the related person 45 included in the target image 91R is four or more, the third face layout map 85c is selected. If the number of faces 90R of the related person 45 included in the target image 91R is one, the fourth face arrangement map 85d is selected.

Trimming section 76 shapes face layout map 85 in accordance with the lowest resolution (step ST4002). The minimum resolution is the resolution required to ensure the minimum image quality as a product when the clipped image 110E clipped by the trimming process is photo printed. The minimum resolution is represented, for example, by the number of pixels. Note that the minimum resolution may be determined in advance according to the resolution of the target image 91R. Also, the resolution for shaping the face position map 85 is not limited to the lowest resolution.

FIG. 21 shows an example of step ST4002. More specifically, the case where the resolution of the target image 91R is 100 dpi×100 dpi, the minimum resolution is 500 pixels×500 pixels, and the first face layout map 85a of 100 inches×100 inches is shaped is exemplified. In this case, the minimum required number of inches of face layout map 85 to cover the lowest resolution is (500×500)/100=2500=50×50. Therefore, the trimming unit 76 trims the first face layout map 85a of 100 inches by 100 inches to 50 inches by 50 inches.

The trimming unit 76 applies the face arrangement map 85 after shaping to the target image 91R. As also shown in FIG. 22, the trimming unit 76 calculates a score for each application position while changing the application position of the face layout map 85 in the target image 91R (step ST4003). The trimming unit 76 stores the calculated score for each application position.

FIG. 22 illustrates a case where the first face arrangement map 85a is applied to a target image 91R in which the faces 90 of four persons are captured and the second and third persons counted from the left are the related persons 45. ing. If the application position of the first face layout map 85a does not include the face 90R of the related person 45, the score is naturally 0 points. On the other hand, when the application position of the first face arrangement map 85a is a position including the face 90R of the related person 45, the score is 30 points, 40 points, or the like. In addition, in FIG. 22, the face 90R of the related person 45 is hatched with oblique lines in order to distinguish it from the face 90 of another person. The same applies to subsequent figures.

FIG. 20 shows a more detailed procedure for score calculation in step ST4003. First, prior to score calculation, the trimming unit 76 sets the pixel value of the face region of the main person 45M in the target image 91R to the first set value, sets the pixel value of the face region of the sub-person 45S to the second set value, and sets the pixel value of the face region of the sub-person 45S to the second set value. The pixel values of the regions are replaced with 0 (step ST40031).

As shown in Table 105, FIG. 23 illustrates a case where the first set value is set to 1 and the second set value is set to 0.8. Also, FIG. 23 illustrates a case where the second person from the left is the main person 45M, and the third person from the left is the sub-person 45S. In this case, the trimming unit 76 replaces the pixel value of the area of the face 90R of the main person 45M with 1 and the pixel value of the area of the face 90R of the sub-person 45S with 0.8. Also, the trimming unit 76 replaces the pixel values of other regions with zero. The other areas are the background area of the person, the torso area of the main person 45M and the sub-person 45S, and the face 90 and torso area of the person who is neither the main person 45M nor the sub-person 45S. In this way, the trimming unit 76 calculates the score by adding the related rank by changing the set value for replacing the pixel value of the area of the face 90R between the main person 45M and the sub-person 45S. become. The set value set for the face area of the related person 45 may be larger than the set value set for areas other than the face area of the related person 45 . Therefore, for example, the first setting value set for the main person 45M and the second setting value set for the sub-person 45S may be the same value.

The trimming section 76 multiplies the evaluation value of each divided cell 100 of the face layout map 85 by the pixel value of the pixel of the target image 91R at the position corresponding to each divided cell 100 (step ST40032). Then, the multiplication results are integrated, and the integration result is output as a score (step ST40033).

After calculating the score for each application position, trimming section 76 enlarges face layout map 85 (step ST4005). Then, the process of step ST4003 is executed again. Trimming section 76 repeats the process of step ST4003 until the size of face layout map 85 becomes the same as the size of target image 91R (NO in step ST4004).

In FIG. 24, as in FIG. 22, the first face arrangement map 85a is applied to the target image 91R in which the faces 90 of four persons are captured and the second and third persons counted from the left are the related persons 45. This is an example of a case where The trimming section 76 performs the process of step ST4003 while gradually enlarging the first face layout map 85a.

If the size of face layout map 85 is the same as the size of target image 91R (YES in step ST4004), trimming section 76 ends the processing of step ST4003. Then, the target image 91R is cut out according to the face layout map 85 at the position where the highest score among the stored scores is applied, and the cutout image 110 is obtained (step ST4006). The trimming unit 76 repeats the processing of steps ST4001 to ST4006 after changing the target image 91R until the clipped image 110 is generated for all the target images 91R (NO in step ST4007). Although the trimming unit 76 gradually enlarges the face layout map 85 in the present embodiment, the present invention is not limited to this. The face layout map 85 may have only one size defined by a specific number of pixels or resolution. In this case, a score may be obtained for the face arrangement map 85 of one size, and the target image 91R may be cut out at the position where the highest score among the stored scores is applied, and the cutout image 110 may be obtained.

In FIG. 25, as in FIG. 22 and the like, the first face arrangement map 85a is applied to the target image 91R in which the faces 90 of four persons are shown and the second and third persons counted from the left are the related persons 45. An example of application is shown. The trimming unit 76 cuts out the target image 91R according to the face layout map 85 at the application position with the highest score of 300 points, and uses it as a cutout image 110 . Then, the clipped image 110 is enlarged to the size of the original target image 91R to obtain a clipped image 110E. It should be noted that if the maximum score is less than the predetermined threshold, the extraction of the target image 91R may be stopped. Further, in the present embodiment, the trimming unit 76 cuts out the clipped image 110 from the target image 91R in accordance with the face layout map 85 at the application position with the highest score, but the present invention is not limited to this. For example, the trimming unit 76 may cut out a plurality of cut-out images 110 having different sizes of the face arrangement map 85 at application positions where the score exceeds a predetermined set value.

When the clipped image 110E is generated for all the target images 91R, the trimming section 76 outputs the generated clipped image 110E as a recommended image (step ST4008). Thus, the trimming process by the trimming section 76 is completed.

As shown in FIG. 26 , the distribution control unit 77 distributes the combination of the clipped image 110E and the score as the recommended image information 59 to the user terminal 11 that sent the recommended image distribution request 58 . Further, when there are a plurality of clipped images 110E as illustrated, the distribution control unit 77 preferentially displays the clipped image 110E according to the score. As an example of the aspect of "preferentially displaying according to the score", as shown in FIG. 26, there is an aspect in which a clipped image 110E with a high score is displayed in a higher order. The display size of the clipped image 110E with a high score may be displayed relatively larger than the display size of the clipped image 110E with a low score.

In FIG. 27, the storage device 50B of the user terminal 11 stores an image viewing AP 115 . When the image browsing AP 115 is executed and a web browser dedicated to the image browsing AP 115 is activated, the CPU 52B of the user terminal 11 functions as the browser control section 120 in cooperation with the memory 51 and the like. Browser control unit 120 controls the operation of the web browser. The browser control unit 120 receives the image file 27 from the image distribution server 10, the recommended image information 59, and the like. The browser control unit 120 generates various screens such as an image display screen 125 (see FIG. 28) displaying the image 36 of the image file 27 and a recommended image information display screen 130 (see FIG. 29) displaying the recommended image information 59. and displayed on the display 54B.

In addition, the browser control unit 120 accepts various operation instructions input from the input device 55B by the user 12 through various screens. The operation instruction includes an image distribution instruction to the image distribution server 10, a recommended image distribution instruction, and the like. The image distribution instruction also serves as an instruction to activate the image viewing AP 115, for example. Browser control unit 120 transmits a request corresponding to the operation instruction to image distribution server 10 . Specifically, browser control unit 120 transmits image distribution request 57 to image distribution server 10 in response to an image distribution instruction to image distribution server 10 . In addition, the browser control unit 120 transmits a recommended image distribution request 58 to the image distribution server 10 in response to the recommended image distribution instruction to the image distribution server 10 .

FIG. 28 shows an image display screen 125 displaying an image 36 included in an image file 27 in the shared folder 26. FIG. On the image display screen 125, a list of a plurality of images 36 is displayed, for example, in order of photographing date and time. Information indicating the shared folder 26 shared by the user 12 is also displayed on the image display screen 125 . The image display screen 125 displays information indicating the shared folder 26 shared by the user 12 and a shared folder designation button 126 for designating the shared folder 26 . In the present embodiment, the shared folder designation button 126 has a star shape, but it is not limited to this and may have any shape. When the shared folder designation button 126 is selected on the image display screen 125, a popup context menu 127 is displayed. A recommended image distribution instruction button 128 and a cancel button 129 are displayed on the context menu 127 . When the recommended image distribution instruction button 128 is selected, a recommended image distribution instruction is input to the browser control unit 120 and a recommended image distribution request 58 is transmitted from the browser control unit 120 to the image distribution server 10 . The recommended image distribution request 58 includes information on the designated shared folder 26 . When the cancel button 129 is selected, the display of the context menu 127 disappears.

FIG. 29 shows a recommended image information display screen 130 that displays recommended image information 59 . A clipped image 110E is displayed on the recommended image information display screen 130 as a recommended image. The clipped images 110E are displayed according to the display order given by the distribution control unit 77 according to the scores.

A score is displayed next to the clipped image 110E as indicated by reference numeral 131 as a recommended score. Also, a save button 132 and a print order button 133 are displayed beside the clipped image 110E. When the save button 132 is selected, the clipped image 110E is saved in the personal folder 25. FIG. When the print order button 133 is selected, it becomes possible to order photo prints of the clipped image 110E.

Next, the operation of the above configuration will be described with reference to the flow charts of FIGS. 30 and 31. FIG. When the operation program 65 is activated, the CPU 52A of the image distribution server 10, as shown in FIG. It functions as a control section 75 , a trimming section 76 and a distribution control section 77 .

As shown in FIG. 30, when there is a first user 12F who has not created related personal information 32 and it is time to create the first related personal information 32 of the first user 12F, the previous related personal information 32 is created. If there is a first user 12F for whom the first set time has elapsed since then, and it is time to update the related personal information 32 of the first user 12F, or if the first number of image files 27 newly transmitted is If there is a user 12F and it is time to update the related personal information 32 of the first user 12F (YES in step ST10), the information creating section 70 performs the information creating process shown in FIGS. 9 to 11 (step ST100A). is executed. As a result, new related person information 32 is created. Step ST100A is an example of an "information creation step" according to the technology of the present disclosure.

As shown in FIG. 31, when the request receiving unit 71 receives the recommended image distribution request 58 (YES in step ST20), the image obtaining unit 72 transmits the second image obtaining request 81 to the image DB server 20. (Step ST21). Then, the image file 27 in the personal folder 25 and the image file 27 in the shared folder 26 transmitted from the image DB server 20 in response to the second image acquisition request 81 are acquired by the image acquisition unit 72 (step ST200). . Step ST200 is an example of an "image acquisition step" according to the technology of the present disclosure.

Subsequently, the extracting section 74 executes the extraction process (step ST300) shown in FIGS. Thereby, the second user image 36S including the face 90R of the related person 45 of the first user 12F is extracted. Step ST300 is an example of an "extraction step" according to the technology of the present disclosure.

Next, the trimming section 76 performs the trimming process (step ST400) shown in FIGS. As a result, a clipped image 110E is generated as a recommended image. Step ST400 is an example of a "trimming step" according to the technology of the present disclosure.

The clipped image 110</b>E is output from the trimming section 76 to the distribution control section 77 . As shown in FIG. 26, the clipped image 110E is made into the recommended image information 59 together with the score by the distribution control unit 77. FIG. Also, the clipped images 110E are assigned a display order according to the score. The recommended image information 59 is distributed by the distribution control unit 77 to the user terminal 11 that sent the recommended image distribution request 58 (step ST22).

As described above, the CPU 52A of the image distribution server 10 has the information creation section 70, the image acquisition section 72, the extraction section 74, and the trimming section . The information creation unit 70 creates information on the related person 45 related to the first user 12F based on the first user image 36F belonging to the first user 12F and the face image 91 including the face 90. Related person information 32 is created. By acquiring the image file 27 in the shared folder 26, the image acquiring unit 72 acquires the second user image 36S belonging to the second user 12S different from the first user 12F. Based on the related person information 32, the extraction unit 74 extracts the target image 91R, which is the second user image 36S including the face 90R of the related person 45 of the first user 12F. The trimming unit 76 cuts out a region including the face 90R of the related person 45 from the target image 91R including the face 90R of the related person 45 to obtain a cutout image 110E. Therefore, it is possible to present to the first user 12F the clipped image 110 clipped from the second user image 36S belonging to the second user 12S based on the related person information 32 of the first user 12F. As a result, it is possible to attract many users 12 to the service.

If the related person information 32 is created by the user 12, there may be a user 12 who feels that the work of registering the related person 45 is troublesome and neglects the registration. Since the clipped image 110 based on the related personal information 32 cannot be presented to such a user 12, the provider of the service by the image distribution server 10 loses an opportunity. However, with the technology of the present disclosure, the related person information 32 can be created without relying on the user 12, so opportunity loss can be prevented.

The information creating unit 70 creates related person information 32 including two or more related persons 45 . The extraction unit 74 extracts the target image 91R including the faces 90R of at least two of the two or more related persons 45. FIG. The trimming unit 76 cuts out a region including at least two faces 90R from a target image 91R including at least two faces 90R. For this reason, for the first user 12F who constitutes a couple, a married couple, a family, a club, a circle, etc., a clipped image with a higher added value that includes a plurality of persons such as the first user 12F and his family, for example. 110E can be presented.

As shown in FIG. 26, the distribution control unit 77 distributes the recommended image information 59 to the user terminal 11 so that the target image 91R, which is the second user image 36S including the face 90R of the related person 45, is displayed. A clipped image 110E clipped from is presented to the first user 12F. Therefore, the first user 12F can browse the clipped image 110E, save the clipped image 110E in his or her personal folder 25, or order a photo print to enjoy the clipped image 110E.

As shown in FIG. 25, the trimming unit 76 cuts out a region including the face 90R of the related person 45 using the face arrangement map 85. FIG. As shown in FIGS. 15-18, face placement map 85 represents the placement position of face 90 in image 36 . A relatively high evaluation value is assigned to the divided cell 100 in which the face 90 is to be arranged among the plurality of divided cells 100 of the face arrangement map 85 . Therefore, a clipped image 110E in which the face 90R is arranged at an appropriate position can be obtained, and the added value of the clipped image 110E can be further increased.

As shown in FIGS. 15 to 18, a plurality of face layout maps 85 are prepared for each number of faces 90R of the related person 45. FIG. The trimming unit 76 selects one face layout map 85 from the plurality of face layout maps 85 according to the number of faces 90R of the related person 45 included in the target image 91R extracted by the extraction unit 74 . Therefore, the trimming process can be performed using the face arrangement map 85 that matches the number of faces 90R of the related person 45, and the added value of the clipped image 110E can be further increased.

As shown in FIGS. 22 and 24, the trimming unit 76 calculates a score according to the evaluation value for each application position while changing the application position of the face arrangement map 85 in the target image 91R, and calculates the calculated score. A region including the face 90R of the related person 45 is cut out based on . Therefore, a clipped image 110E in which the face 90R is arranged at an appropriate position can be obtained, and the added value of the clipped image 110E can be further increased.

As shown in FIG. 6, the related persons 45 include a main person 45M who is relatively highly related to the first user 12F and a sub-person 45S who is less related to the first user 12F than the main person 45M. including. By distributing the related persons 45 in this way and assigning a related rank indicating the degree of relatedness to the first user 12F, it is possible to calculate a score taking into consideration the related rank as shown in FIG.

As shown in FIGS. 10 and 11, the information creation unit 70 selects the main person 45M and the sub-person 45S based on the number of face images 91, which are the first user images 36F including the face 90. Allocate. In addition, the information creating unit 70 determines the main person 45M and the sub-person based on the ratio of the number of face images 91 including the face 90 to the number of first user images 36F that satisfy a predetermined setting condition. 45S is distributed. Therefore, the main person 45M and the sub-person 45S can be clearly sorted, and the reliability of the related person information 32 can be improved.

As shown in FIG. 26, the distribution control unit 77 distributes the recommended image information 59 to the user terminal 11, thereby presenting the set of the clipped image 110E and the score to the first user 12F. Therefore, the first user 12F can confirm by the score how well the clipped image 110E is an image in which the face 90R is arranged at an appropriate position.

Also, when there are a plurality of clipped images 110E, the plurality of clipped images 110E are rearranged according to the scores and presented to the first user 12F. Therefore, the first user 12F can grasp the clipped image 110E with a high score at a glance.

10 and 11, the selection condition 96 for assigning the related rank to the person recognized in the first user image 36F includes the condition related to the number of images for each recognized person and the recognition It may be based on at least one of the conditions related to the proportion of each person in the first user image 36F. Also, the condition related to the number of sheets or the condition related to the ratio may be changed according to the related rank. For example, the first selection condition 93, which is the selection condition 96 for the main person 45M, may be a condition related to the number of images, and the second selection condition 94, which is the selection condition 96 for the sub person 45S, may be a condition related to the ratio.

Note that the main person 45M is not limited to the person with the largest number of face images 91 including the face 90 . For example, a person whose number of face images 91 including the face 90 is equal to or greater than a predetermined third threshold (third threshold≦number of images) may be selected as the main person 45M. Alternatively, a person whose ratio of the first ratio and the second ratio is equal to or higher than a predetermined fourth threshold (fourth threshold≦ratio) may be selected as the main person 45M. In these cases, a plurality of persons may be selected as the main person 45M.

Similarly, the sub-person 45S is not limited to the person selected according to the second selection condition 94 illustrated in the first embodiment. For example, a person whose number of face images 91 including the face 90 is less than the third threshold and equal to or greater than a predetermined fifth threshold (fifth threshold≦number of images<third threshold) is selected as the sub-person 45S. good too. Alternatively, a person whose ratio of the first ratio and the second ratio is less than the fourth threshold and equal to or greater than a predetermined sixth threshold (sixth threshold≦ratio<fourth threshold) may be selected as the sub-person 45S. good.

Further, the "first user image that satisfies a predetermined setting condition", which is the basis for calculating the ratio, is the image with face 91 extracted in step ST1003 of the example, and the image with face that includes the face 90R of the main person 45M. It is not limited to the image 91 . For example, all the first user images 36F may be set as "first user images that satisfy predetermined setting conditions". Also, the image with face 91 including the face 90 of the first user 12F may be set as the "first user image that satisfies a predetermined setting condition". By doing so, the related person information 32 can be created based on the face 90 of the user 12 who is included in the same image 36 as the first user 12F and who is considered to have a high degree of intimacy.

The aspect of assigning the relation rank indicating the degree of relevance to the first user 12F is not limited to the aspect of assigning to the main person 45M and the sub-person 45S of the first embodiment. For example, a person with the largest number of face images 91 including a face 90 is given a relation rank of 1, a person with the next largest number of images is given a relation rank of 2, and a person with the third largest number of images is given a relation rank of 3, and so on. A relation rank may be given in order of the number of face images 91 including the face 90 . Similarly, a related rank may be assigned in order of percentage, such as the first percentage.

FIG. 32 shows an example of replacing the pixel values of the target image 91R with the set values when the relation rank is assigned as described above. The target image 91R includes the faces 90R of the related person 45_1 with the related rank 1, the related person 45_2 with the related rank 2, and the related person 45_3 with the related rank 3. Further, as shown in Table 135, the set value for the region of the face 90R of the related person 45_1 with the related rank 1 is 1, the set value for the region of the face 90R of the related person 45_2 with the related rank 2 is 0.8, and the related rank 3. are set to 0.6, . In addition, 0 is set as the set value for other areas. In this case, the trimming unit 76 sets the pixel value of the area of the face 90R of the related person 45_1 to 1, the pixel value of the area of the face 90R of the related person 45_2 to 0.8, and the pixel value of the area of the face 90R of the related person 45_3 to Substitute 0.6. Also, the trimming unit 76 replaces the pixel values of other regions with zero. Note that the number of related ranks in the related person information 32 may be one, or three or more. In other words, the number of set values corresponding to the related rank may be one, or three or more. Similarly, the number of selection conditions 96 for assigning a related rank to the person recognized in the first user image 36F shown in FIGS. 10 and 11 may be one, or three or more.

In the first embodiment, the pixel values of the area of the face 90R of the related person 45 in the target image 91R are replaced with positive set values, and the pixel values of other areas are replaced with 0, but the present invention is not limited to this. . Only the pixel values in the center area of the face 90R of the related person 45 may be replaced with positive set values, and the pixel values in other areas may be replaced with zero. The central area is an elliptical area including, for example, the nose and the glabella.

As shown in FIG. 33, the clipped image 110E is assigned a display order according to the score for each related person 45, such as a related person 45_1 with a related rank of 1, a related person 45_2 with a related rank of 2, and so on. good too.

In the first embodiment described above, the face layout map 85 is shaped to match the lowest resolution and then gradually enlarged, but the present invention is not limited to this. Conversely, the face arrangement map 85 may be shaped according to the target image 91R and then gradually reduced toward the lowest resolution. In addition, the target image 91R is cut out to form the cutout image 110, and the cutout image 110 is enlarged to the size of the original target image 91R to form the cutout image 110E, but the present invention is not limited to this. The clipped image 110 may have a predetermined size instead of the size of the original target image 91R. A predetermined size is, for example, a size corresponding to a photo print.

Note that the timing of creating the related person information 32 is not limited to the timing illustrated above. The related person information 32 may be created when the recommended image distribution request 58 is received by the request receiving unit 71 .

[Second embodiment]
In the first embodiment described above, in the information creation process, the related person information 32 is created for all the first user images 36F included in the personal folder 25, but the invention is not limited to this. As in the second embodiment shown in FIGS. 34 and 36, the first user images 36F used to create the related person information 32 may be narrowed down to specific images.

FIG. 34 is an example of creating the related person information 32 based only on the first user image 36F related to a specific event. Differences from the information creation process of the first embodiment shown in FIGS. 9 and 10 will be mainly described. The information creation process (step ST100B) in this case is performed until the transmission of the first image acquisition request 80 (step ST1001) and the acquisition of the image file 27 in the personal folder 25 (step ST1002). This is the same as the creation process (step ST100A). In this case, instead of extracting the image with face 91 from the first user image 36F included in the image file 27 (step ST1003) in the information creation process (step ST100A) of the first embodiment, the personal folder 25, the event image 140 related to the specific event is extracted from the first user image 36F included in the acquired image file 27 (step ST1010), and the image with face 91 is extracted from the event image 140. (step ST1011). Then, person recognition of the face included in the image with face 91 extracted from the event image 140 is performed (step 1004). Since the procedure after step 1004 is the same as the information creation process (step ST100A) of the first embodiment, description thereof is omitted.

FIG. 34 shows an example of extracting event images 140 related to specific events such as weddings, birthdays, and Christmas from words registered in tags of image information 35 . Event images 140 include those that do not include a person's face 90 (for example, a photo showing a cake tagged "Christmas"). The information creating section 70 extracts the face-containing image 91 including the face 90 from the event image 140 .

In this way, the information creation unit 70 creates the related person information 32 based only on the event image 140, which is the first user image 36F related to the specific event. A person who has a higher relationship with the first user 12F participates in the specific event. Therefore, it is possible to avoid a situation in which a person who is frequently shown in the first user image 36F but who actually has relatively low relevance to the first user 12F is erroneously recognized as the related person 45. FIG. The reliability of the related person information 32 can be enhanced. Moreover, the load of creating the related person information 32 can be reduced as compared with the case where all the first user images 36F are targeted.

Note that specific events are not limited to the exemplary weddings, birthdays, Christmas, and the like. Funerals, local festivals, trips, Hinamatsuri, Boy's Festival, etc. are also acceptable. Event images 140 related to events such as birthdays, Christmas, Hinamatsuri, and Boy's Festival may be extracted from the shooting date and time instead of the words registered in the tags. Further, the event shown in the first user image 36F may be estimated by image analysis, and if the estimated event is a specific event, the first user image 36F may be extracted as the event image 140. FIG.

The procedures of steps ST1010 and ST1011 may be reversed. That is, the event image 140 may be extracted from the face-containing image 91 after extracting the face-containing image 91 from the first user image 36F.

FIG. 35 is an example of creating the related person information 32 based only on the first user image 36F relating to a specific period. Differences from the information creation process of the first embodiment shown in FIGS. 9 and 10 will be mainly described. The information creation process (step ST100C) in this case includes transmission of the first image acquisition request 80 (step ST1001) and acquisition of the image file 27 in the personal folder 25 (step ST1002) is the same as the information creation process (step ST100A) of the first embodiment. In this case, instead of extracting the image with face 91 from the first user image 36F included in the image file 27 (step ST1003) in the information creation process (step ST100A) of the first embodiment, the personal folder After acquiring the image file 27 in 25, the period image 142 relating to the specific period is extracted from the first user image 36F included in the acquired image file 27 (step ST1020), and the image with face 91 is extracted from the period image 142. (Step ST1021). Then, person recognition of the face included in the image with face 91 extracted from the period image 142 is performed (step 1004). Since the procedure after step 1004 is the same as the information creation process (step ST100A) of the first embodiment, description thereof is omitted.

FIG. 35 shows an example in which the first user image 36</b>F whose photographing date and time of the image information 35 is 2019 is extracted as the period image 142 . The information creating unit 70 extracts the face-containing image 91 including the face 90 from the period image 142 .

In this way, the information creating unit 70 creates the related person information 32 based only on the period image 142, which is the first user image 36F related to the specific period. Therefore, the personal relationship of the first user 12</b>F, which changes daily, can be reflected in the related person information 32 . Moreover, as in the case of targeting the event image 140, the load of creating the related person information 32 can be reduced more than the case of targeting all the first user images 36F.

In addition, the specific period is not limited to the illustrated one year. It may be five years, ten years, or the like. In addition, by performing image analysis on the first user image 36F whose shooting date and time is unknown, such as an old photograph read by the scanner 16, the shooting date and time are estimated, and the period image 142 is extracted based on the estimated shooting date and time. good too.

You may reverse the procedure of step ST1020 and step ST1021. That is, the period image 142 may be extracted from the face-containing image 91 after extracting the face-containing image 91 from the first user image 36F.

The total number of event images 140 or the total number of period images 142 may be used as the “number of first user images satisfying a predetermined setting condition” used to calculate the ratios such as the first ratio and the second ratio.

For example, the aspect shown in FIG. 34 and the aspect shown in FIG. 35 may be combined, such as creating the related person information 32 based on the first user image 36F related to a specific event in the last five years.

As shown in FIG. 36, when the related person information 32 is created for each specific period and the extraction unit 74 extracts the target image 91R from the second user image 36S, the related person information 32 may be used differently. FIG. 36 shows an example of creating related person information 32 for each year. In this case, for example, the related person information 32 of 2019 is used when the extraction unit 74 extracts the target image 91R from the second user image 36S of 2019. By doing so, it is possible to extract the target image 91R that reflects the human relationship of the first user 12F more, and as a result, it is possible to further increase the added value of the clipped image 110E.

In addition, in the aspect of creating the related person information 32 for all the first user images 36F in the first embodiment, the following may be performed. That is, the event image 140 is further extracted from the image with face 91 extracted from the first user image 36F. As for the event image 140, in step ST1005 for calculating the number of face images 91 including the face 90 for each person whose face 90 has been recognized, the number is calculated as, for example, five instead of one. Alternatively, the period images 142 are extracted from the face images 91 extracted from the first user image 36F, and the period images 142 are calculated as, for example, five in step ST1005. Event images 140 and period images 142 may be extracted from face images 91 extracted from first user image 36F, and the number of event images 140 and period images 142 calculated in step ST1005 may be greater than one. . In this way, it is possible to create more reliable related person information 32 while targeting all first user images 36F.

[Third embodiment]
In the third embodiment shown in FIGS. 37 and 38, the score for the face 90 of the person other than the related person 45 is calculated as the positive/negative value of the score for the face 90R of the related person 45 .

FIG. 37 shows how the pixel values of the target image 91R are replaced with the set values in the third embodiment. The target image 91R includes faces 90 of a total of five persons. Among the five persons, the two persons in the middle are the related persons 45, and the faces 90 of the persons other than the related persons 45 are shown in the upper center of the faces 90R of the two related persons 45. As shown in Table 145, the setting value is set to 1 for the area of the face 90R of the related person 45, while -1 is set to the area of the face 90 of the person other than the related person 45. ing. 0 is set in other areas. In this case, the trimming unit 76 sets the pixel value of the area of the face 90R of the related person 45 to 1, the pixel value of the area of the face 90 of the person other than the related person 45 to -1, and the pixel value of the other areas to 0. Replace. By replacing the pixel value of the area of the face 90 of the person other than the related person 45 with -1, the score of the face 90 of the person other than the related person 45 is positive or negative with the score of the face 90R of the related person 45. Calculated as the opposite, negative value.

By calculating the score of the face 90 of the person other than the related person 45 as a negative value in this way, the following effects can be obtained. First, let us consider a case where the score for the face 90 of a person other than the related person 45 is not calculated as a negative value, as in the first embodiment. In this case, in a composition where the distance between the face 90R of the related person 45 and the face 90 of the person other than the related person 45 is relatively close, such as the target image 91R shown in FIG. 37, as shown in FIG. 38A, A clipped image 110E clipped according to the face layout map 85a includes a part of the face 90 of a person other than the related person 45, which does not look good.

On the other hand, if the score for the face 90 of the person other than the related person 45 is calculated as a negative value, the score at the application position including the face 90 of the person other than the related person 45 is the face 90 of the person other than the related person 45. is calculated to be lower than the score at the application position that does not include Therefore, as shown in FIG. 38B, the probability of cutting out an area that does not include the face 90 of a person other than the related person 45 increases. Therefore, the appearance of the cutout image 110E can be improved, and the added value of the cutout image 110E can be further increased.

Note that, contrary to the above, when the score for the face 90R of the related person 45 is calculated as a negative value by, for example, setting a negative set value in the region of the face 90R of the related person 45, the trimming unit 76 calculates the score of the face 90 of the person other than the related person 45 as a positive value by, for example, setting a positive set value in the area of the face 90 of the person other than the related person 45 . Further, when calculating the score of the face 90R of the related person 45 as a negative value, the trimming unit 76 selects the face layout map 85 arranged at the application position where the score is the lowest point or equal to or less than a predetermined set value. , the clipped image 110 may be generated.

As a method for improving the appearance of the clipped image 110E, the following method may be adopted. That is, when the distance between the face 90R of the related person 45 and the face 90 of the person other than the related person 45 is less than the predetermined seventh threshold, the number of faces 90R of the related person 45 and the number of related person The face arrangement map 85 to be used is selected according to the total number of faces 90 of persons other than 45. Then, a setting value equivalent to that of the face 90R of the related person 45 is set for the area of the face 90 of the person other than the related person 45. FIG.

FIG. 39 illustrates a case where the distance between the faces 90R1 and 90R2 of two related persons 45 and the face 90 of one person other than the related person 45 is less than the seventh threshold pL. The distance between the faces 90R1 and 90R2 of the two related persons 45 and the face 90 of one person other than the related person 45 is, for example, c1 and c2 as the centers of the faces 90R1 and 90R2 of the related person 45. Assuming that the center of the face 90 of one person other than the related person 45 is c3, the length of the line segment L1 connecting c1 and c3 and the length of the line segment L2 connecting c2 and c3 are provided. If the length of at least one of the line segment L1 and the line segment L2 is less than the seventh threshold pL, the trimming unit 76 trims the one person other than the related matter 45 to be similar to the related person 45 as described below. can be treated as

Although the trimming unit 76 should select the first face arrangement map 85a according to the number of faces 90R of the related person 45, the number of faces 90R of the related person 45 and the number of faces 90 of persons other than the related person 45 are The second face layout map 85b is selected according to the total number of three. In this way, the entire face 90 of the person other than the related person 45 can be included in the clipped image 110E without being partially cut off as shown in FIG. 38A. In FIG. 39, the distance between the related person 45 and the person other than the related person 45 is the length of the line segment connecting the centers of the faces 90, but it is not limited to this. For example, the length of a line segment connecting the centers of gravity of the face 90, the length of a line segment connecting specific parts such as eyes and nose included in the face 90, and the length of the shortest line segment connecting the contours of the face 90. It may be at least one of

37 and 38B, the aspect shown in FIG. 38A, and the aspect shown in FIG. 39 may be configured so that the user 12 can select them. Alternatively, the clipped image 110E may be generated in each of these three modes, and the three clipped images 110E may be presented to the user 12. FIG.

[Fourth embodiment]
In the above-described first embodiment, the face arrangement map 85 created in advance based on the formulas (1) to (8) is used, but the present invention is not limited to this. As in the fourth embodiment shown in FIGS. 40 to 46, a face layout map 85 may be created based on an image 91 with a face.

In FIG. 40, in the fourth embodiment, the CPU 52A of the image distribution server 10 functions as a map creation section 150 in addition to each section of the first embodiment. In addition to the image DB server 20 and the user information DB server 21 , a map DB server 151 is connected to the image distribution server 10 . Note that the map DB server 151 may be configured as the same DB server as at least one of the image DB server 20 and the user information DB server 21 , or may be included in the image distribution server 10 .

The map creator 150 creates a face layout map 85 for each user 12 . The map creating unit 150 transmits the created face layout map 85 to the map DB server 151 . A map DB server 151 has a map DB 152 . The map DB server 151 stores and manages the face layout map 85 from the map creating unit 150 in the map DB 152 . The map DB 152 stores the face layout map 85 in association with the user ID.

The map creation unit 150 transmits a third image acquisition request 153 to the image DB server 20 . The third image acquisition request 153 requests acquisition of the image file 27 in the personal folder 25 of the first user 12F for which the face layout map 85 has not been created. The third image acquisition request 153 is an image file in the personal folder 25 of the first user 12F after a predetermined time (hereinafter referred to as a second set time) has passed since the face arrangement map 85 was created last time. 27 is requested. The third image acquisition request 153 is the user ID of the first user 12F for whom the face arrangement map 85 has not been created or the user ID of the first user 12F for whom the second set time has passed since the previous face arrangement map 85 was created. including. The timing of requesting acquisition of the image file 27 in the personal folder 25 of the first user 12F for which the face arrangement map 85 is not created by the third image acquisition request 153 is, for example, This is the timing when the number of image files 27 stored in is equal to or greater than a predetermined number. Alternatively, it may be the timing at which the first user 12F installs the image viewing AP 115 on the user terminal 11 . The second set time is, for example, one month, half a year, one year, or the like. The third image acquisition request 153 may be transmitted when a predetermined second number of image files 27 are newly transmitted.

The image DB server 20 reads the image file 27 including the first user image 36F in the personal folder 25 in response to the third image acquisition request 153 from the image DB 22 and transmits the read image file 27 to the image distribution server 10. . The map creation unit 150 acquires the image file 27 including the first user image 36F transmitted from the image DB server 20 in response to the third image acquisition request 153. FIG. The map creation unit 150 creates a face layout map 85 based on the first user image 36F included in the acquired image file 27. FIG. If the face layout map 85 has not been created, the map DB server 151 newly registers the face layout map 85 from the map creating unit 150 in the map DB 152 . On the other hand, when the face layout map 85 has already been created, the map DB server 151 rewrites the face layout map 85 stored in the map DB 152 with the face layout map 85 from the map creation unit 150 . The map DB server 151 also transmits the face layout map 85 to the trimming unit 76 .

41 to 44 show the procedure of map creation processing (step ST500A) for creating face layout map 85 executed in map creation section 150. FIG. Map creation section 150 first transmits third image acquisition request 153 to image DB server 20 (step ST5001). Then, map creation section 150 acquires image file 27 in personal folder 25 transmitted from image DB server 20 in response to third image acquisition request 153 (step ST5002).

Map creation section 150 uses face recognition technology to extract face-containing image 91 including face 90 from first user image 36F included in acquired image file 27 (step ST5003). Next, map creation section 150 evaluates the image quality of extracted image with face 91 (step ST5004).

FIG. 43 illustrates a case where an image quality evaluation value is calculated as an image quality evaluation result. The image quality evaluation value is, for example, a value indicating the blurring degree of the outline of the face 90 obtained by performing edge detection on the outline of the face 90 . Note that the image quality evaluation value may be, for example, a value corresponding to the degree of exposure (overexposed highlights, blocked out shadows), or a value comprehensively capturing a plurality of indexes representing image quality such as the degree of blurring and the degree of exposure. may

Map creating section 150 selects face-containing images 91 whose evaluated image quality is equal to or higher than the set level (step ST5005). FIG. 43 illustrates a case where the setting level is 100. In FIG. In this case, images 91 with faces whose image quality evaluation value is less than 100 are excluded.

Map creating section 150 creates face layout map 85 based on face-containing image 91 having image quality equal to or higher than the set level selected in step ST5005 (step ST5006).

FIG. 42 shows a more detailed procedure for creating face layout map 85 in step ST5006. First, the map creating section 150 replaces the pixel values of the area of the person's face 90 in the image with face 91 with 1, and replaces the pixel values of other areas with 0 (step ST50061). Next, map creating section 150 integrates the replaced pixel values of each image with face 91 for each pixel (step ST50062). Map creation section 150 divides the integrated value of the pixel values by the number of face images 91, and sets the obtained numerical value as the evaluation value of face layout map 85 (step ST50063).

The map creation unit 150 creates the first face layout map 85a from the face image 91 including the faces 90 of two persons. Also, the map creating unit 150 creates the second face layout map 85b from the face-containing image 91 including the faces 90 of three persons. Further, the map creating unit 150 creates a third face arrangement map 85c from the face-containing image 91 including the faces 90 of four or more persons. Furthermore, the map creating unit 150 creates a fourth face arrangement map 85d from the face image 91 including the face 90 of one person.

FIG. 44 shows an example of creating the first face arrangement map 85a from 100 images 91 with faces containing the faces 90 of two persons. In an image 91 with a face, faces 90 are arranged at various positions. For this reason, the integrated value of the pixel values calculated in step ST50062 takes various values as indicated within the dashed circle 155 . Similarly, the evaluation values calculated in step ST50063 also have various values as indicated within the dashed circle 156 . Therefore, the face arrangement map 85 does not have a symmetric distribution of evaluation values as in the first embodiment, but has a biased distribution of evaluation values.

Map creating section 150 transmits face layout map 85 created in this way to map DB server 151 (step ST5007). The map creating unit 150 registers, for example, the face layout map 85 created based on the first user image 36F of the first user 12F in the map DB server 151 in association with the user ID of the first user 12F. Thus, the map creating process by the map creating unit 150 ends. Note that, in the present embodiment, the map creation unit 150 selects the image with a face 91 whose evaluated image quality is equal to or higher than the set level, but the present invention is not limited to this. For example, the face layout map 85 may be created for all of the images with faces 91 included in the first user image 36F.

Thus, by creating the face arrangement map 85 based on the image 91 with a face, it is possible to create the face arrangement map 85 adapted to the actual photographing tendency. In addition, since the image with a face 91 used to create the face layout map 85 is an image with image quality equal to or higher than a predetermined set level, the reliability of the face layout map 85 can be enhanced.

Since the first user image 36F is used as the image with face 91 used to create the face layout map 85, the face layout map 85 adapted to the photographing tendency of the first user 12F can be created. In other words, the face placement map 85 can be customized for the first user 12F.

45 and 46 show an example of using the first user image 36F selected by the first user 12F to create the face arrangement map 85. FIG. The map creation process (step ST500B) in this case is different from the map creation process (step ST500A) in that an image with a face 91 having a selection history by the first user 12F is selected (step ST5010). The selection history is stored with items prepared in the image information 35 . In the selection history item, for example, "yes" is stored for the first user image 36F uploaded by the first user 12F to a SNS (Social Networking Service) specializing in image posting.

FIG. 46 shows steps ST5002, ST5003, and ST5010. The map creating unit 150 selects images with faces 91 with a selection history of "yes" and excludes images with faces 91 with a selection history of "no".

The first user image 36F selected by the first user 12F is an image preferred by the first user 12F. Therefore, if the first user image 36F selected by the first user 12F is selected and used to create the face arrangement map 85, the face arrangement map 85 reflecting the arrangement position of the face 90 preferred by the first user 12F can be created. can be created.

Although the first user image 36F uploaded to the SNS is the first user image 36F selected by the first user 12F, it is not limited to this. Prior to creation of the face arrangement map 85, a list of the first user images 36F is displayed on the user terminal 11, and the first user images 36F used for creation of the face arrangement map 85 are selected from the first user images 36F displayed as a list. may be selected by the first user 12F.

The face layout map 85 may be created based on not only the first user image 36F but also other images 36. FIG. Another image 36 is, for example, a second user image 36S included in the image file 27 in the shared folder 26. FIG. The face layout map 85 may be created based on all the images 36 stored in the image DB 22 . In this way, by creating the face arrangement map 85 based on not only the first user image 36F but also other images 36, it is possible to create a face arrangement map 85 adapted to more general shooting tendencies. When the face arrangement map 85 is created based on another image 36, an image with a face 91 having an image quality equal to or higher than a predetermined set level may be selected, or an image with a face selected by the user 12 may be selected. 91 may be selected.

If the number of face-containing images 91 is less than a predetermined threshold value, creation of the face arrangement map 85 may be stopped. In this case, the face layout map 85 created in advance based on the equations (1) to (8) exemplified in the first embodiment is used. Alternatively, a target image 91R including the face 90R of the related person 45 may be selected from the image 91 with a face, and the face layout map 85 may be created based on the selected target image 91R.

The face arrangement map 85 for the portrait image 36 and the face arrangement map 85 for the landscape image 36 may be created separately. Also, the face layout map 85 may be created for each aspect ratio of the image 36 such as 1:1, 4:3, and 16:9.

A face layout map 85 may be created for each event such as a wedding ceremony or an athletic meet. Further, the face layout map 85 may be created separately for a case where a person is moving in a foot race of an athletic meet, etc., and a case where a person is stationary in a group photo or the like.

The face arrangement map 85 may be created for each related rank of the related person 45, such as for the related person 45_1 assigned the related rank 1, for the related person 45_2 assigned the related rank 2, and the like.

Note that the timing of creating the face layout map 85 is not limited to the timing illustrated above. The face layout map 85 may be created when the request receiving unit 71 receives the recommended image distribution request 58 .

The related person information 32 may be created using the first user image 36F whose image quality is equal to or higher than a predetermined set level. In this case, "the number of first user images that satisfy a predetermined setting condition", which is the basis for calculating the ratios such as the first ratio and the second ratio, is the first user image whose image quality is equal to or higher than a predetermined setting level. It is good also as all the number of sheets of 36F.

For example, the trimming process may be performed using the method disclosed in Japanese Patent Application Laid-Open No. 2007-272685 without using the face layout map 85 . However, in this case, the clipped image 110E may be generated with a resolution lower than the lowest resolution. On the other hand, if the face arrangement map 85 is used, the clipping of the target image 91R can be stopped when the highest score is less than the predetermined threshold, so that the clipped image 110E having a resolution lower than the lowest resolution can be obtained. is not likely to be generated.

The target image 91R may be limited to the target image 91R including the main person 45M. Also, the related person information 32 may be configured to be modifiable by the user 12 .

Among the plurality of users 12 forming the user group 28 , there may be a user 12 who does not store the image file 27 in the shared folder 26 but has access authority to the shared folder 26 .

Various modifications are possible for the hardware configuration of the computer that constitutes the image distribution server 10 . For example, the image distribution server 10 can be composed of a plurality of server computers separated as hardware for the purpose of improving processing capability and reliability. For example, the functions of the request reception unit 71, the image acquisition unit 72, the information acquisition unit 73, and the distribution control unit 77, and the functions of the information creation unit 70, the extraction unit 74, the RW control unit 75, and the trimming unit 76 are combined into two. It is distributed to each server computer. In this case, the image distribution server 10 is composed of two server computers.

In this manner, the hardware configuration of the computer of the image distribution server 10 can be appropriately changed according to required performance such as processing power, safety, and reliability. Furthermore, not only the hardware but also application programs such as the operation program 65 can of course be duplicated or distributed and stored in multiple storage devices for the purpose of ensuring safety and reliability. be.

A part or all of the functions of the processing units of the image distribution server 10 may be performed by the user terminal 11 .

In each of the above embodiments, for example, the information creation unit 70, the request reception unit 71, the image acquisition unit 72, the information acquisition unit 73, the extraction unit 74, the RW control unit 75, the trimming unit 76, the distribution control unit 77, and the map creation unit As a hardware structure of the processing unit (Processing Unit) that executes various processes such as 150, various processors shown below can be used. In addition to the CPU 52A, which is a general-purpose processor that executes software (operation program 65) and functions as various processing units, the various processors include an FPGA (Field Programmable Gate Array) whose circuit configuration can be changed after manufacture. Programmable Logic Device (PLD), which is a processor, and/or ASIC (Application Specific Integrated Circuit). etc. are included.

One processing unit may be configured with one of these various processors, or a combination of two or more processors of the same or different type (for example, a combination of a plurality of FPGAs and/or a CPU and combination with FPGA). Also, a plurality of processing units may be configured by one processor.

As an example of configuring a plurality of processing units with a single processor, first, as represented by computers such as clients and servers, a single processor is configured by combining one or more CPUs and software. There is a form in which a processor functions as multiple processing units. Secondly, as typified by System On Chip (SoC), etc., there is a mode of using a processor that realizes the function of the entire system including multiple processing units with a single IC (Integrated Circuit) chip. be. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

Furthermore, more specifically, as the hardware structure of these various processors, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined can be used.

The technology of the present disclosure can also appropriately combine various embodiments and/or various modifications described above. Moreover, it is needless to say that various configurations can be employed without departing from the scope of the present invention without being limited to the above embodiments. Furthermore, the technology of the present disclosure extends to storage media that non-temporarily store programs in addition to programs.

The above description and illustration are detailed descriptions of the parts related to the technology of the present disclosure, and are merely examples of the technology of the present disclosure. For example, the above descriptions of configurations, functions, actions, and effects are descriptions of examples of configurations, functions, actions, and effects of portions related to the technology of the present disclosure. Therefore, unnecessary parts may be deleted, new elements added, or replaced with respect to the above-described description and illustration without departing from the gist of the technology of the present disclosure. Needless to say. In addition, in order to avoid complication and facilitate understanding of the portion related to the technology of the present disclosure, the descriptions and illustrations shown above require no particular explanation in order to enable implementation of the technology of the present disclosure. Descriptions of common technical knowledge, etc., that are not used are omitted.

As used herein, "A and/or B" is synonymous with "at least one of A and B." That is, "A and/or B" means that only A, only B, or a combination of A and B may be used. In addition, in this specification, when three or more matters are expressed by connecting with "and/or", the same idea as "A and/or B" is applied.

All publications, patent applications and technical standards mentioned herein are expressly incorporated herein by reference to the same extent as if each individual publication, patent application and technical standard were specifically and individually noted to be incorporated by reference. incorporated by reference into the book.

10 Image distribution server (image processing device)
11 User terminal 12 User 12F First user 12S Second user 13 Network 15 Digital camera 16 Scanner 20 Image database server (image DB server)
21 User information database server (user information DB server)
22 Image Database (Image DB)
23 User information database (user information DB)
25 personal folder 26 shared folder 27 image file 28 user group 30 user information 31 user basic information 32 related person information 35 image information 36 image 36F first user image 36S second user image 40 face image 45, 45_1, 45_2, 45_3 related person 45M Main person 45S Sub person 50, 50A, 50B Storage device 51 Memory 52, 52B CPU
52A CPU (processor)
53 communication unit 54, 54B display 55, 55B input device 56 bus line 57 image distribution request 58 recommended image distribution request 59 recommended image information 65 operation program (image processing apparatus operation program)
66 face layout map set 70 information creation unit 71 request reception unit 72 image acquisition unit 73 information acquisition unit 74 extraction unit 75 read/write control unit (RW control unit)
76 trimming unit 77 distribution control unit 80 first image acquisition request 81 second image acquisition request 82 information acquisition request 85 face arrangement map 85a first face arrangement map 85b second face arrangement map 85c third face arrangement map 85d fourth face arrangement Map 90 Faces 90R, 90R1, 90R2 Related person's face 91 Face-containing image 91R Target image (face-containing image containing the related person's face)
92, 95, 101, 105, 135, 145 Table 93 First selection condition 94 Second selection condition 96 Selection condition 100 Divided cells 110, 110E Clipped image 115 Image viewing application program (image viewing AP)
120 Browser control unit 125 Image display screen 126 Shared folder designation button 127 Context menu 128 Recommended image distribution instruction button 129 Cancel button 130 Recommended image information display screen 131 Recommended score 132 Save button 133 Print order button 140 Event image 142 Period image 150 map creation unit 151 map database server (map DB server)
152 Map DB (Map Database)
153 Third image acquisition request 155, 156 Circles c1 to c3 Face centers L1, L2 Line segment pL connecting the centers of faces Seventh threshold values ST10, ST20, ST21, ST22, ST500A, ST500B, ST1001, ST1002, ST1003, ST1004, ST1005, ST1006, ST1007, ST1010, ST1011, ST1020, ST1021, ST3001, ST3002, ST3003, ST3004, ST3005, ST4001, ST4002, ST4003, ST4004, ST4005, ST4006, ST4007, ST4008, ST 5001, ST5002, ST5003, ST5004, ST5005, ST5006, ST5007, ST5010, ST40031, ST40032, ST40033, ST50061, ST50062, ST50063 Step ST100A, ST100B, ST100C Step (information creation step)
ST200 step (image acquisition step)
ST300 step (extraction step)
ST400 step (trimming step)

Claims (22)

comprising at least one processor;
The processor
Based on the first user image belonging to the first user and including the face, related person information is created, which is information about a related person related to the first user,
Acquiring a second user image belonging to a second user different from the first user,
extracting the second user image containing the face of the related person based on the related person information;
A face arrangement map representing the arrangement position of the face in the image, from the second user image including the face of the related person, to the divided cell in which the face is to be arranged, among a plurality of divided cells. Using a face arrangement map assigned a relatively high evaluation value to the
Image processing device. The processor
creating the related person information including two or more of the related persons;
extracting the second user image containing the faces of at least two of the two or more related persons;
The image processing apparatus according to claim 1, wherein a region including the faces of the at least two persons is cut out from the second user image including the faces of the at least two persons. The processor
3. The image processing apparatus according to claim 1, wherein a clipped image clipped from the second user image is presented to the first user. The processor
4. The image processing apparatus according to any one of claims 1 to 3, wherein the related person information is created based only on the first user image related to a specific event. The processor
5. The image processing apparatus according to any one of claims 1 to 4, wherein the related person information is created based only on the first user image relating to a specific period. A plurality of face arrangement maps are prepared for each number of faces,
The processor
6. The face arrangement map according to the number of faces of the related persons included in the extracted second user image is selected from among the plurality of face arrangement maps. 2. The image processing device according to item 1 . The processor
While changing the application position of the face arrangement map in the second user image, a score corresponding to the evaluation value is calculated for each application position, and an area including the face of the related person is calculated based on the calculated score. 7. The image processing apparatus according to any one of claims 1 to 6, which cuts out the . The processor
When creating the related person information, giving each related person a related rank indicating the degree of relevance to the first user;
8. The image processing apparatus according to claim 7 , wherein the score is calculated taking into consideration the related rank. The processor
9. The image processing apparatus according to claim 8 , wherein the relation rank is given based on the number of the first user images containing a face. The processor
Claim 8 or 9 , wherein the relation rank is given based on a ratio of the number of the first user images containing a face to the number of the first user images satisfying a predetermined setting condition. The described image processing device. The processor
11. The image processing according to any one of claims 7 to 10 , wherein the score for the face of the person other than the related person is calculated as a positive/negative value of the score for the face of the related person. Device. The processor
The image processing device according to any one of claims 7 to 11 , wherein a set of the clipped image clipped from the second user image and the score is presented to the first user. When there are a plurality of clipped images clipped from the second user image, the plurality of clipped images are rearranged according to the score and presented to the first user. 12. An image processing device according to any one of 12 . The processor
14. The image processing apparatus according to any one of claims 1 to 13 , wherein the face arrangement map is created based on an image with a face containing a face. 15. The image processing apparatus according to claim 14 , wherein the image with a face is an image having an image quality equal to or higher than a predetermined set level. 16. The image processing apparatus according to claim 14 , wherein the image with face is the first user image. 17. The image processing apparatus according to claim 16 , wherein the image with face is the first user image selected by the first user. An information creation step of creating related person information, which is information of a related person related to the first user, based on the first user image belonging to the first user and including the face;
an image acquisition step of acquiring a second user image belonging to a second user different from the first user;
an extracting step of extracting the second user image including the face of the related person based on the related person information;
A face arrangement map representing the arrangement position of the face in the image, from the second user image including the face of the related person, to the divided cell in which the face is to be arranged, among a plurality of divided cells. a trimming step of cutting out a region containing the face of the related person using a face arrangement map assigned a relatively high evaluation value for the face arrangement map ;
A method of operating an image processing apparatus comprising: an information creation unit that creates related person information, which is information about a related person related to the first user, based on a first user image that belongs to the first user and includes a face;
an image acquisition unit that acquires a second user image belonging to a second user different from the first user;
an extraction unit that extracts the second user image including the face of the related person based on the related person information;
A face arrangement map representing the arrangement position of the face in the image, from the second user image including the face of the related person, to the divided cell in which the face is to be arranged, among a plurality of divided cells. As a trimming unit that cuts out an area containing the face of the related person using a face arrangement map to which a relatively high evaluation value is assigned ,
An operating program for an image processing device that causes a computer to function. comprising at least one processor;
The processor
Related person information, which is information on a related person related to the first user, based only on the first user image belonging to the first user and including the face and relating to a specific period. and create
Acquiring a second user image belonging to a second user different from the first user,
extracting the second user image containing the face of the related person based on the related person information;
cutting out a region containing the face of the related person from the second user image containing the face of the related person;
There are a plurality of said specific periods,
creating the related person information for each of the plurality of specific periods;
When extracting the second user image containing the face of the related person, according to the acquisition period of the second user image, selectively use the related person information for each specific period,
Image processing device. Related person information, which is information on a related person related to the first user, based only on the first user image belonging to the first user and including the face and relating to a specific period. an information creating step of creating a
an image acquisition step of acquiring a second user image belonging to a second user different from the first user;
an extracting step of extracting the second user image including the face of the related person based on the related person information;
a trimming step of cutting out an area including the face of the related person from the second user image including the face of the related person;
with
There are a plurality of said specific periods,
In the information creating step, the related person information is created for each of the plurality of specific periods;
In the extracting step, when extracting the second user image including the face of the related person, the related person information for each specific period is selectively used according to the acquisition period of the second user image. ,
A method of operating an image processing device. Related person information, which is information on a related person related to the first user, based only on the first user image belonging to the first user and including the face and relating to a specific period. an information creation unit that creates
an image acquisition unit that acquires a second user image belonging to a second user different from the first user;
an extraction unit that extracts the second user image including the face of the related person based on the related person information;
A trimming unit that cuts out an area containing the face of the related person from the second user image containing the face of the related person,
make your computer work
There are a plurality of said specific periods,
The information creating unit creates the related person information for each of the plurality of specific periods,
When extracting the second user image including the face of the related person, the extraction unit selectively uses the related person information for each specific period according to the acquisition period of the second user image. ,
An operating program for an image processing device.

Images