JP6797871B2 - program - Google Patents

Description

The present invention relates to a program.

A large number of digital photo data taken by users using digital still cameras (hereinafter also referred to as DSCs) and multifunctional mobile phones equipped with camera functions can be used for PCs owned by users and storage services existing on the Internet. It will be saved.

When the user searches for where the desired image is stored from among a large number of stored digital photo data, it is necessary to perform a huge amount of image browsing operations. This is a factor that greatly impairs usability in the viewability and searchability of images.

Therefore, Patent Document 1 proposes a method of creating one or a plurality of layout candidates by changing the parameters of the past layout stored in the layout table.

JP 2009-245071

By the way, there are many so-called failed images such as blur images in the image group taken by the user, but the user may store the images without erasing them.

In Patent Document 1, when an image is selected from an image group including a failed image to create a layout image, if the failed image is selected, a layout image with low satisfaction for the user is created.

In view of the above circumstances, it is an object of the present invention to provide an image processing device, an image processing method, and a program that enhance user satisfaction with a layout image.

The program of the present invention for solving the above-mentioned problems is a program of a predetermined application, and is an image in which a first person is included in a computer and a second person different from the first person. A step of acquiring an image including, and a plurality of images including the image, and analyzing and processing the acquired plurality of images to obtain a plurality of person's faces and objects other than the person's face included in the plurality of images. It recognized, identifying the face of the recognized person per person, and a step to display the first facial image corresponding to the face of the identified first person on the basis of the analysis on a display unit Based on the step of receiving a predetermined user instruction related to the first face image while the first face image is displayed on the display unit and a user input different from the predetermined user instruction. Based on the steps of determining a theme related to a predetermined event as the layout theme, the step of setting the first person as the first main character based on the predetermined user instruction, and the determined theme. , and setting the second person as second hero from among the templates held by theme, selecting a template based on the determined theme, based on the predetermined user instruction based on the set second hero set based on the set first hero and the theme Te, the selected template based on the theme, the image including the first person It is characterized in that a step of generating a layout image in which an image including the second person is arranged is executed.

In the present invention, an effect that can increase the user satisfaction with the layout image.

It is a hardware block diagram of the image processing apparatus which concerns on Embodiment 1. FIG. It is a software block diagram which concerns on Embodiment 1. It is a flowchart of image analysis processing which concerns on Embodiment 1. It is a flowchart of image analysis processing which concerns on Embodiment 1. It is a flowchart of the person group generation processing which concerns on Embodiment 1. It is a flowchart of the automatic layout proposal processing which concerns on Embodiment 1. It is a figure which shows the display example of the person group of Embodiment 1. FIG. It is a figure which shows the display example of the image group of Embodiment 1. FIG. It is a figure which shows the example of the storage format of the image analysis result. It is a figure which shows the example of the UI for manually inputting a favorite degree, and the example of the UI for manually inputting event information. It is a figure which shows the UI example for manually inputting a person attribute information. It is a figure which shows the example of the storage format of the person attribute information. It is a figure which shows an example of a layout template. It is a figure which shows the example of the holding form of the layout template of FIG. It is a figure which shows an example of a layout template. It is a figure which shows the example of the holding form of the layout template of FIG. It is a flowchart of the automatic layout generation processing in Embodiment 1. It is a flowchart of unnecessary image filtering processing in Embodiment 1. It is explanatory drawing of the automatic trimming process, the explanatory drawing of the calculation method of the brightness appropriateness, and the explanatory drawing of the calculation method of the saturation appropriateness. It is explanatory drawing of the trimming chipping determination processing. It is a figure which shows the display example of the automatic layout generation result which concerns on Embodiment 1. It is a figure which shows the determined theme and the holding example of the hero information. It is a figure which shows the determined theme and the holding example of the hero information. It is a figure which shows the holding example of the generated layout information.

Hereinafter, the first embodiment of the present invention will be described in order to automatically generate a layout output using the input image group. This is merely an example of one embodiment, and the present invention is not limited to the following embodiments.

FIG. 1 is a block diagram showing a hardware configuration example of the image processing device of the first embodiment.

In FIG. 1, the image processing device 115 includes a CPU 100, a ROM 101, a RAM 102, a secondary storage device 103, a display device 104, an input device 105, an IF 107, an IF 108, and a wireless LAN 109. Further, it includes an internal imaging device 106. These are connected to each other by a control bus / data bus 110.

The image processing device 115 is, for example, a computer. The CPU 100 (central processing unit) executes the information processing described in the first embodiment according to a program. The ROM 101 stores programs such as the following applications executed by the CPU 100. The RAM 102 provides a memory for temporarily storing various information when the program is executed by the CPU 100. The secondary storage device 103 is a hard disk or the like, and is a storage medium for storing an image file, a database for storing an image analysis result, or the like. The display device 104 is, for example, a display, and is a device that presents the processing result of the first embodiment, the UI (User Interface) shown below, and the like to the user. The display device 104 may have a touch panel function. The input device 105 is a mouse, keyboard, or the like for the user to input an instruction or the like for image correction processing.

Further, the image captured by the internal imaging device 106 is stored in the secondary storage device 103 after undergoing predetermined image processing. The image processing device 115 can also read image data from an external imaging device 111 connected via an interface (IF108). Further, the wireless LAN (Local Area Network) 108 is connected to the Internet 113. The image processing device 115 can also acquire image data from an external server 114 connected to the Internet 113.

The printer 112 for outputting an image or the like is connected to the image processing device 115 via the IF 107. The printer 112 is further connected to the Internet, and print data can be exchanged via the wireless LAN 109.

FIG. 2 is a block diagram of the software configuration such as the application in the present embodiment.

First, the image data acquired by the hardware is usually in a compressed format such as JPEG (Joint Photography Expert Group). Therefore, the image codec unit 200 decompresses the compressed format and converts it into a so-called RGB point sequential bitmap data format. The converted bitmap data is transmitted to the display / UI control unit 201 and displayed on a display device 104 such as a display.

The bitmap data is further input to the image sensing unit 203 (application), in which various analysis processes of the image are performed, which will be described in detail later. Various attribute information of the image obtained as a result of the above analysis process is stored in the above-mentioned secondary storage device 103 in the database unit 202 according to a predetermined format. In the following, image analysis processing and sensing processing will be treated as synonymous.

The scenario generation unit 204 (application) generates layout conditions to be automatically generated according to various conditions input by the user, and the layout generation unit 205 (application) automatically generates layouts according to the above scenario. Perform processing.

In the generated layout, the rendering unit 206 generates bitmap data for display, the bitmap data is sent to the display / UI control unit 201, and the result is displayed on a display device 104 such as a display. On the other hand, the rendering result is further sent to the print data generation unit 207, which is converted into printer command data and sent to the printer.

3 to 6 are flowcharts of basic image processing of the application of the present embodiment. Specifically, FIGS. 3 and 4 show the flow of the image sensing unit 203, which is a process of acquiring a plurality of image data groups, performing analysis processing on each of them, and storing the results in a database. It shows the flow. FIG. 5 shows a flow of processing for grouping face information that seems to be the same person based on the detected face position information. FIG. 6 shows a flow of processing for determining a scenario for creating a layout based on image analysis information and various information input by the user, and automatically generating a layout based on the scenario. There is.

In S301 of FIG. 3, an image data group including one or more image data is acquired. The image data group is acquired, for example, by the user connecting an image pickup device or a memory card in which the captured image is stored to the image processing device 115 and reading the captured image from these. Further, the image data group taken by the internal imaging device and stored in the secondary storage device may be acquired. Alternatively, the image data group may be acquired from a location other than the image processing device 115, such as an external server 114 connected on the Internet via a wireless LAN.

When the image data group is acquired, the thumbnail group is displayed on the UI as shown in FIGS. 8 (a) and 8 (b). As shown in 801 of FIG. 8A, thumbnails 802 of images may be displayed for each folder in the secondary storage device 103, or as shown in FIG. 8B, for each date in UI901 such as a calendar. Image data may be managed in. By clicking the date portion 902, the images taken on the same day are displayed in a thumbnail list as shown in FIG. 8A.

Next, in S302, each image is decoded. Specifically, the application searches for a newly saved image that has not yet been subjected to sensing processing, and the image codec unit 200 converts the compressed data into bitmap data for each extracted image.

Next, in S303, various sensing processes are executed on the above-mentioned bitmap data. The sensing process referred to here includes various processes as shown in Table 1 below. In the present embodiment, face detection, image feature analysis, and scene analysis are given as examples of sensing processing, and the results of data types as shown in Table 1 are calculated for each.

Hereinafter, each sensing process will be described.

Since the overall average brightness and average saturation, which are the basic features of the image, may be obtained by a known method, detailed description thereof will be omitted. For the average brightness, the RGB component may be converted into a known luminance color difference component (for example, YCbCr component) for each pixel of the image (conversion formula omitted), and the average value of the Y component may be obtained. Further, the average saturation may be calculated as follows for each pixel for the CbCr component, and the average value of S below may be obtained.

Further, the average hue (AveH) in the image is a feature amount for evaluating the hue of the image. The hue of each pixel can be obtained using a known HIS conversion formula, and AveH can be obtained by averaging them over the entire image.

Further, these feature quantities may be calculated for the entire image as described above, or for example, the image may be divided into regions of a predetermined size and calculated for each region.
Next, the face detection process of a person will be described. As a method for detecting the face of a person used in the present embodiment, a known method can be used.

In the method described in JP-A-2002-183731, the eye region is detected from the input image, and the periphery of the eye region is set as the face candidate region. For this face candidate region, the brightness gradient for each pixel and the weight of the brightness gradient are calculated, and these values are compared with the preset ideal gradient of the face reference image and the weight of the gradient. At that time, if the average angle between the gradients is equal to or less than a predetermined threshold value, it is determined that the input image has a face region.

Further, in the method described in Japanese Patent Application Laid-Open No. 2003-30667, the position of the eye can be detected by first detecting the skin color region in the image and then detecting the human rainbow-colored pixel in the region. it can.

In the method described in JP-A-8-63597, first, the degree of matching between a template having a plurality of face shapes and an image is calculated. The template having the highest matching degree is selected, and if the matching degree having the highest matching degree is equal to or higher than a predetermined threshold value, the area in the selected template is set as the face candidate area. By using the same template, the position of the eyes can be detected.

Further, in the method described in Japanese Patent Application Laid-Open No. 2000-105829, first, the nose image pattern is used as a template, the entire image or a designated area in the image is scanned, and the most matching position is output as the nose position. .. Next, the area above the nose position of the image is considered to be the area where the eyes exist, and the eye image pattern is used as a template to scan the eye-existing area for matching, and the degree of matching is higher than a certain threshold. Find the eye existence candidate position set, which is a set of. Then, the continuous region included in the eye existence candidate position set is divided into clusters, and the distance between each cluster and the nose position is calculated. The organ position can be detected by determining the cluster with the shortest distance as the cluster in which the eyes exist.

As other methods for detecting the face of a person, JP-A-8-7734, JP-A-2001-216515, JP-A-5-197793, JP-A-11-53525, JP-A-2000-132688, JP-A-2000-235648, JP-A-11- Methods of detecting face and organ positions as described in 250267 can be mentioned. Further, the face detection process of a person may be performed by the method described in Japanese Patent No. 2541688, and the method is not particularly limited.

By the face detection process of a person, the number of human faces and the coordinate position of each face can be acquired for each input image. Further, by knowing the coordinate position of the face in the image, it is possible to analyze the feature amount of the face region. For example, by obtaining the average YCbCr value of the pixel values included in the face region for each face region, the average brightness and the average color difference of the face region can be obtained.

In addition, scene analysis processing can be performed using the feature amount of the image. The scene analysis process can be performed by, for example, the methods disclosed in JP-A-2010-251999 and JP-A-2010-273144 disclosed by the applicant. By the scene analysis process, it is possible to acquire an ID for distinguishing a shooting scene such as a landscape (Landscape), a night view (Nightscape), a person (Portrait), an underexposure (Underexposure), and others (Others).

In the present embodiment, the sensing information is acquired by the above sensing process, but other sensing information may be used.

The sensing information acquired as described above is stored in the database unit 202. The storage format in the database unit 202 may be described and stored in, for example, a general-purpose format (for example, XML: eXtensible Markup Language) as shown in FIG.

FIG. 9 shows an example in which the attribute information for each image is described by dividing it into three categories.

The first BaseInfo tag is a tag for storing information added to an image file acquired in advance as image size and shooting information. This includes an identifier ID for each image, a storage location where the image file is stored, an image size, a shooting date and time, and the like.

The second SensInfo tag is a tag for storing the result of the image analysis process described above. The average brightness, average saturation, average hue and scene analysis result of the entire image are stored, and information on the face position and complexion of a person existing in the image is stored.

The third UserInfo tag is a tag that can store the information input by the user for each image, and the details will be described later.

The method of storing the image attribute information in the database is not limited to the above. It may be stored in any other known format.

In S305 of FIG. 3, it is determined whether or not the image subjected to the processing of S302 and S303 described above is the last image. If it is the last image, the process proceeds to S306, and if it is not the last image, the process returns to S302.

In S306, a process of generating a group for each person is performed using the face position information detected in S303. By automatically grouping the faces of people in advance, it is possible to streamline the work of the user naming each person afterwards.

The person group formation here is executed by the processing flow of FIG. 5 using a known personal recognition method.

The personal recognition process is mainly executed by extracting the feature amounts of organs such as eyes and mouth existing in the face and comparing the similarity of their relationships. Since the personal recognition process is disclosed in, for example, Japanese Patent No. 3469031, detailed description here will be omitted.

FIG. 5 is a basic flowchart of the person group generation process S306.

First, in S501, the images stored in the secondary storage device are sequentially read out and decoded. Further, the database unit 202 is accessed by S502 to acquire the number of faces included in the image and the position information of the faces. Next, in S504, a normalized face image for performing personal recognition processing is generated.

Here, the normalized face image is a face image obtained by cutting out faces existing in various sizes, orientations, and resolutions in the image, and converting and cutting them so that all of them have a predetermined size and orientation. Since the positions of organs such as eyes and mouth are important for personal recognition, it is desirable that the size of the normalized facial image is such that the above organs can be reliably recognized. By generating the normalized face image in this way, it is not necessary to deal with faces having various resolutions in the feature amount detection process.

Next, in S505, the facial feature amount is calculated from the normalized face image. The facial feature amount here is characterized by including the position and size of organs such as eyes, mouth, and nose, as well as the contour of the face.

Further, in S506, it is determined whether or not the face feature amount is similar to the face feature amount of the database (hereinafter referred to as the face dictionary) in which the face feature amount is stored in advance for each person's identifier (dictionary ID). The similarity is calculated by comparing, for example, the feature amount managed inside the dictionary ID with the newly input feature amount. The feature amount used here is information such as the position of the organs such as eyes, nose, and mouth that are held, and the distance between the organs. The degree of similarity is as high as the above-mentioned feature quantities are similar, and if they are not similar, a low value is taken, for example, a value of 0 to 100 can be taken. Then, in the determination of whether or not they are similar, the calculated similarity is compared with the threshold value held in advance, and if the similarity is higher than the threshold value, it is determined that the person is the same person as the dictionary ID. On the other hand, if the similarity is lower than the threshold value, it is determined that they are not the same person. As for the threshold value for such similarity determination, only one fixed value may be held for all dictionary IDs, or a different threshold value may be held for each dictionary ID. Good.

If the determination in S506 is Yes, the process proceeds to S509, and the feature amount of the face is added to the dictionary ID of the same person as the same person.

If the determination in S506 is No, the process proceeds to S508, and it is determined that the face currently being evaluated is a person different from the person registered in the face dictionary so far, a new dictionary ID is issued, and the face dictionary is issued. Add to. The processes S502 to S509 are applied to all the face areas detected from the input image group to group the appearing characters.

As shown in the XML format of FIG. 12, the result of the person group generation process is described by using an ID tag for each face and stored in the above-mentioned database.

In the above embodiment, as shown in FIG. 3, the person group generation process is executed after the sensing process of all the images is completed, but other methods may be used. For example, as shown in FIG. 4, even if the operation of executing the sensing process in S403 for one image and then performing the grouping process S405 using the face detection position information is repeated, the same result can be obtained. Can be generated.

In addition, each person group obtained by the person group generation process is displayed by UI701 as shown in FIG. In the same 7, 702 represents a representative face image of the person group, and next to it, there is an area 703 for displaying the name of the person group. Immediately after the automatic person grouping process is completed, the person names are displayed as "No name 1", "No name 2", etc., as shown in the figure. These person names will be referred to as "person IDs" below. Further, 704 is a plurality of facial images included in the person group. As will be described later, in the UI 701 of FIG. 7, a person name can be input by designating the area 703 of "No name X", and information such as a birthday and a relationship can be input for each person.

Further, the above sensing process may be executed by utilizing a background task of the operating system. In this case, the user can continue the sensing process of the image group even if he / she is performing another work on the computer.

In the present embodiment, the user can manually input various attribute information related to the image.

Table 2 shows a list of examples of the attribute information (hereinafter referred to as manual registration information). The manual registration information is largely divided into information set for each image and information set for the person group-processed by the above processing.

First, there is a user's favorite degree as attribute information to be set for each image. The favorite level is a manual step-by-step input by the user as to whether or not the image is liked. For example, as shown in FIG. 10A, a dialog is displayed in which the desired thumbnail image 1302 can be selected with the mouse pointer 1303 on the UI 1301 and the favorite degree can be input by right-clicking. The user can select the number of ★ in the menu according to his / her preference. In this embodiment, the higher the degree of favorite, the larger the number of ★.

Further, the favorite degree may be set automatically without being manually set by the user. For example, suppose that the user clicks on a desired image file from the image thumbnail list display state shown in FIG. 8A to transition to the one image display screen. The number of transitions may be measured and the favorite degree may be set according to the number of transitions. For example, the more times the image is viewed, the more the user determines that he / she likes the image.

Further, as another example, the number of prints may be set to the favorite degree. For example, when a printing act is performed, it is naturally determined that the image is liked and the favorite degree is set to be high. In this case, the number of prints is measured, and the degree of favorite is set according to the number of prints.

As described above, the favorite degree may be set manually by the user, the favorite degree may be set according to the number of views, or the favorite degree may be set according to the number of prints. Each of these settings and measured information is individually stored in the UserInfo tag of the database unit 202 in the XML format as shown in FIG. For example, the favorite degree is stored in the FavoriteRate tag, the number of views is stored in the ViewingTimes tag, and the number of prints is stored in the PrintingTimes tag.

Further, as another information to be set for each image, event information can be mentioned. The event information includes, for example, a family trip "travel", a graduation ceremony "graduation", and a wedding "wedding".

As shown in FIG. 10B, the event may be specified by designating a desired date on the calendar with the mouse pointer 1402 or the like and inputting the event name of the day. The specified event name will be included in the XML format shown in FIG. 9 as a part of the attribute information of the image. In the format of FIG. 9, the event name and the image are associated with each other by using the Event tag in the UserInfo tag. In the following, "linking" refers to associating.

Next, the attribute information of the person will be described.

FIG. 11 shows a UI for inputting attribute information of a person. In FIG. 11, 1502 shows a representative face image of a predetermined person (in this case, “father”). Reference numeral 1503 is a display area for a person name (person ID) of a predetermined person. Further, 1504 is an image (thumbnail) detected from other images and determined in S506 to have similar facial features. As described above, in FIG. 11, under the person ID 1503, a list of images 1504 determined to have similar facial features in S506 is displayed.

Immediately after the sensing process is completed, no name is input to each person group as shown in FIG. 7, but an arbitrary person name is input by pointing the mouse pointer to the "No name" part 702. be able to.

In addition, as attribute information for each person, it is possible to set the birthday of each person and the relationship seen by the user who is operating the application. When the representative face 1502 of the person in FIG. 11 is clicked, the birthday of the clicked person can be input in the first input unit 1505 as shown in the lower part of the screen. In addition, in the second input unit 1506, the relationship information of the clicked person can be input.

As described above, the input personal attribute information is managed in the database unit 202 separately from the image attribute information by the XML format as shown in FIG. 12, unlike the attribute information associated with the images so far.

In the present embodiment, layout generation processing is performed using various layout templates prepared in advance. The layout template is as shown in FIGS. 13 and 15, and includes a plurality of image arrangement frames 1702, 1902, and 1903 (hereinafter, synonymous with slots) on the paper size to be laid out.

A large number of such templates are prepared, and may be stored in the secondary storage device 103 when the software for executing the present embodiment is installed in the image processing device 115 in advance. Further, as another method, an arbitrary template group may be acquired from an external server 114 existing on the Internet connected via IF107 or wireless LAN 109.

It is assumed that these templates are described in XML in a highly versatile structured language, for example, as in the storage of sensing results described above. Examples of XML data are shown in FIGS. 14 and 16.

In these examples, first, the basic information of the layout page is described in the BASIC tag. The basic information may be, for example, the layout theme, page size, page resolution (dpi), and the like. In Example X, the Theme tag, which is the layout theme, is blank in the initial state of the template. Further, as basic information, the page size is set to A4 and the resolution is set to 300 dpi.

Further, the ImageSlot tag describes the information of the image arrangement frame described above. Two ID tags and POSITION tags are held in the Image Slot tag, and the ID and position of the image arrangement frame are described. The position information is defined in, for example, an XY coordinate system with the upper left as the origin, as shown in FIGS. 14 and 16.

In addition, the Image Slot also sets the shape of the slot and the name of the recommended person group to be arranged for each slot. For example, in the template of FIG. 13, as shown by the Shape tag of FIG. 14, it is recommended that all the slots have a rectangular “rectangle” shape, and the person group name is “MainGroup” by the PersonGroup tag.

Further, in the template of FIG. 15, as shown in FIG. 34, it is described that the slot having ID = 0 arranged in the center has a rectangular shape. In addition, it is recommended that the person group arranges "SubGroup", the slots following the other IDs = 1 and 2 have an elliptical "ellipse" shape, and the person group arranges "MainGroup".

In this embodiment, a large number of templates as described above are retained.

The application according to the present embodiment can execute analysis processing on the input image group, automatically group people, and display them on the UI. In addition, the user can see the result, input attribute information such as a name and birthday for each person group, and set a favorite degree for each image. In addition, you can have a large number of layout templates sorted by theme.

The application of the present embodiment satisfying the above conditions automatically generates a collage layout that is likely to be liked by the user at a predetermined timing, and performs a process of presenting the collage layout to the user (hereinafter referred to as a layout proposal process).

When the above conditions are satisfied, the application of the present embodiment automatically generates a collage layout that is likely to be liked by the user at a predetermined timing, and performs a process of presenting the collage layout to the user. This is called layout proposal processing.

FIG. 6 shows a basic flowchart for performing layout proposal processing.

First, in S601, a scenario for layout proposal processing is determined. The scenario includes determination of the theme and template of the proposed layout, setting of the person (hero) to be emphasized in the layout, selection information of the image group used for layout generation, and the like.

In the following, two scenarios will be illustrated to explain how to determine the scenario.

For example, suppose that two weeks ago, it was set to automatically propose a birthday layout for each person. It is assumed that the first birthday of the person "son" who is automatically grouped in FIG. 11 is near. In this case, the theme of the proposed layout is determined to be the growth record “growth”. Next, a template is selected. In this case, a template such as FIG. 15 suitable for growth recording is selected, and as shown in FIG. 22, “growth” is described in the part of the XML Theme tag. Next, set "son" as the main character "MainGroup" to pay attention to when laying out. In addition, "son" and "father" are set as "SubGroup" which is secondarily noticed when laying out. Next, a group of images to be used for the layout is selected. In the case of this example, the database is referred to, and a large number of image groups including "son" are extracted and listed from the image groups taken so far from the birthday of the person "son". The above is the scenario determination for the growth record layout.

As an example different from the above, it is assumed that if the predetermined event information is registered within one month, the layout proposal process is set to be executed. From the event information registered in FIG. 10B, for example, when it is found that a family trip was made several days ago and a large amount of the images were stored in the secondary storage device, the scenario determination department proposed a layout for the family trip. Determine the scenario to do. In this case, the theme of the proposed layout is determined to be travel "travel". Next, the template is selected. In this case, the layout as shown in FIG. 13 is selected, and as shown in FIG. 36, “travel” is described in the part of the XML Tag. Next, "son", "mother", and "father" are set as the main character "MotherGroup" to be noticed when laying out. In this way, by utilizing the characteristics of XML, a plurality of persons can be set as "MainGroup". Next, a group of images to be used for the layout is selected. In the case of this example, the database is referred to, and a large number of images associated with the above travel event are extracted and listed. The above is the scenario determination for the family travel layout.

Next, in S603 of FIG. 6, the layout automatic generation process based on the above-mentioned scenario is executed. FIG. 17 shows a detailed processing flow of the layout processing unit. Hereinafter, each processing step will be described with reference to the figure.

First, in S2101, the template information is acquired after the theme and the person group information are set, which is determined by the scenario generation process described above.

Next, in S2103, based on the image list determined in the above scenario, the feature amount of the image is acquired from the database for each image, and the image group attribute information list is generated. The image group information list referred to here has a structure in which the IMAGEINFO tags shown in FIG. 9 are arranged by the number of image lists.

As described above, in the automatic layout generation processing of the present embodiment, instead of directly handling the image data itself in this way, the attribute information that has been subjected to sensing processing for each image in advance and stored in the database is used. This is to avoid requiring a very large memory area for storing the image group if the image data itself is targeted when performing the layout generation process. That is, this makes it possible to reduce the amount of memory required for the layout generation process.

Specifically, first, in S2105, unnecessary images are filtered from the input image group by using the attribute information of the input image group. The filtering process is performed according to the flow shown in FIG. In FIG. 18, for each image, first, it is determined in S2201 whether or not the overall average brightness is included in a certain threshold value (ThY_Low and ThY_High). If not, the process proceeds to S2206, and the image of interest is removed from the layout target.

Similarly, in S2202 to S2205, it is determined whether or not the average luminance and the average color difference component are included in the predetermined threshold value indicating a good skin color region for each of the face regions included in the attention image. Only the images for which all the determinations of S2202 to S2205 are Yes are applied to the subsequent layout generation processing. Specifically, in S2202, it is determined whether or not AveY of the face region having ID = N is included in the range of predetermined threshold values (ThfY_Low and ThfY_High). In S2203, it is determined whether or not AveCh of the face region having ID = N is included in the range of predetermined threshold values (ThfY_Low and ThfY_High). In S2204, it is determined whether or not AveCr in the face region having ID = N is included in the range of predetermined threshold values (ThfY_Low and ThfY_High). In S2205, it is determined whether or not it is the last face. If it is not the last face, the process returns to S2202, and if it is the last face, the process ends.

Since the purpose of this filtering process is to remove images that are clearly unnecessary for the subsequent temporary layout creation process, it is desirable to set the above threshold value relatively gently. For example, in the determination of the overall image brightness of S2201, if the difference between ThY_High and ThY_Low is extremely small compared to the image dynamic range, the number of images determined to be Yes will decrease accordingly. In order to prevent this from happening in the filtering process of the present embodiment, the difference between the two is set as wide as possible, and the threshold value is set so that those clearly judged to be abnormal images can be removed.

Next, in S2107 of FIG. 17, a large number (L) of temporary layouts are generated using the image group that was the layout target in the above process. To generate the temporary layout, the process of arbitrarily applying the input image to the image arrangement frame of the acquired template is repeated. At this time, for example, the following parameters (image selection / arrangement / trimming) are randomly determined.

As an image selection criterion, for example, when there are N image arrangement frames in the layout, which image is selected from the image group can be mentioned. As an arrangement criterion, for example, in which arrangement frame a plurality of selected images are arranged can be mentioned. As a trimming standard, there is a trimming rate of how much trimming processing is performed when the trimming is performed. The trimming rate is represented by, for example, 0 to 100%, and the trimming is performed at a predetermined trimming rate with reference to the center of the image as shown in FIG. 19A. In FIG. 19A, 2301 shows the entire image, and 2302 shows a cutting frame when trimmed at a trimming rate of 50%.

Generate as many temporary layouts as possible based on the image selection, placement, and trimming criteria described above. Each generated temporary layout can be represented as XML in FIG. For each slot, the ID of the selected and arranged image is described in the ImageID tag, and the trimming rate is described in the TrimingRatio tag.

The number L of the temporary layouts generated here is determined according to the processing amount of the evaluation processing in the layout evaluation step described later and the performance of the image processing device 115 that processes the evaluation processing. For example, hundreds of thousands of temporary layouts are generated. It is preferable to generate a temporary layout above the street.

The amount of evaluation processing in the layout evaluation step increases or decreases depending on the degree of complexity of the layout template to be created. For example, the larger the number of slots handled in the template, the larger the evaluation processing amount, and the more complicated the layout conditions specified for each slot, the larger the evaluation processing amount. Therefore, the degree of complexity of the template to be generated may be estimated in advance, and L may be dynamically determined accordingly.

By setting L appropriately as described above, the quality of the response and layout result at the time of automatic layout creation can be optimally controlled.

The generated layouts may be stored in the secondary storage device 103 in the XML format shown in FIG. 24 by adding an ID to each of them, or may be stored in the RAM 102 using another data structure such as a structure.

Next, in S2108 of FIG. 17, a quantitative evaluation is performed on the temporary layout generated in large quantities. Specifically, each of the created L temporary layouts is evaluated using a predetermined layout evaluation amount. Table 3 shows a list of layout evaluation amounts in this embodiment. As shown in Table 3, the layout evaluation amount used in this embodiment can be mainly divided into three categories.

The first is the evaluation amount of each image. This is to judge the state such as the brightness, saturation, and the amount of blurring of the image and score it. Hereinafter, an example of scoring the present embodiment will be described. As shown in FIG. 19B, the appropriateness of brightness is set so that the score value is 100 within a predetermined range with average brightness, and the score value is lowered when the average brightness is out of the predetermined range. As shown in FIG. 19 (c), the appropriateness of saturation is set to a score value of 100 when the average saturation of the entire image is larger than a predetermined saturation value, and a score value is set to a value smaller than the predetermined value. Set to gradually lower. The details of determining the state of the amount of blurring will be described later.

The second is the evaluation of the fit between the image and the slot. Examples of the evaluation of the fit between the image and the slot include the fit of a person and the determination of lack of trimming. The goodness of fit of a person represents the goodness of fit of a person designated in a slot and a person actually existing in the image arranged in the slot. For example, it is assumed that a certain slot is a Person Group specified in XML, and "father" and "son" are specified. At this time, assuming that the two persons are shown in the image assigned to the slot, the person suitability of the slot is set to a score value of 100. If only one person is shown, the goodness of fit is a score of 50, and if both are not shown, the score is 0. The goodness of fit on the page is the average value of the goodness of fit calculated for each slot. For the chipping determination of the trimming area 2702, for example, as shown in FIG. 20, when the position 2703 of the face existing in the image is known, a score value from 0 to 100 is set according to the area of the chipped portion. calculate. If the missing area is 0, the score is 100, and conversely, if all the face areas are missing, the score value is 0.

The third is the evaluation of the balance within the layout page. Examples of the evaluation value for evaluating the balance include image similarity, variation in pixel value distribution, and variation in objects.

Image similarity will be described as an evaluation amount for evaluating the balance in the layout page. The image similarity is the similarity of each image in the layout page calculated for each temporary layout generated in large quantities. For example, if you want to create a layout for a travel theme, it may not be a good layout if all the images are too similar and have a high degree of similarity. Therefore, for example, the similarity can be evaluated by the shooting date and time. Images with similar shooting dates and times are likely to have been shot at similar locations, but if the shooting dates and times are far apart, the location and scene are likely to be different. As shown in FIG. 9, the shooting date and time can be acquired as image attribute information from the attribute information for each image stored in advance in the database unit 202. To find the similarity from the shooting date and time, perform the following calculation. For example, it is assumed that four images as shown in Table 4 are laid out in the temporary layout that is currently being focused on.

The shooting date and time information is added to each image specified by the image ID. Specifically, the year, month, day and time (year: YYYY, month: MM, day: DD, hour: HH, minute: MM, second: SS) are added as the shooting date and time. At this time, the value at which the shooting time interval is the shortest among these four images is calculated.

In this case, 30 minutes between image IDs 102 and 108 is the shortest interval. This interval is set to MinInterval and is stored in seconds. That is, 30 minutes = 1800 seconds. This MinInterval is calculated for each of the L temporary layouts and stored in the array stMinInterval [l]. Next, the maximum value MaxMinInterval value is obtained in the stMinInterval [l]. Then, the similarity evaluation value [l] of the l-th temporary layout can be obtained as follows.
Similarity [l] = 100 × stMinInterval [l] / MaxMinInterval
That is, the Simularity [l] is effective as an image similarity evaluation value because the larger the minimum shooting time interval, the closer to 100, and the smaller the minimum shooting time interval, the closer to 0.

The variation in the pixel value distribution will be described as an evaluation amount for evaluating the balance in the layout page. Here, as the variation of the pixel value distribution, the variation of the hue will be described as an example. For example, if you want to create a layout for a travel theme, and you have images of too similar colors (eg, blue in the blue sky, green in the mountains), it may not be a good layout. In this case, the one with large color variation is evaluated highly. The variance of the average hue AveH of the image existing in the l-th temporary layout of interest is calculated, and it is stored as the degree of hue variation tpColorVariance [l]. Next, the maximum value MaxColorVariety value in the tpColorVariety [l] is obtained. Then, the evaluation value ColorVariance [l] of the degree of color variation of the l-th temporary layout can be obtained as follows.
ColorVariance [l] = 100 × tpColorVariety [l] / MaxColorVariety
The ColorVariety [l] becomes a value closer to 100 as the variation in the average hue of the images arranged on the page is larger, and closer to 0 as the variation is smaller. Therefore, it can be used as an evaluation value of the degree of variation in hue.

The variation in the pixel value distribution is not limited to the above.

The variation of objects will be described as an evaluation amount for evaluating the balance in the layout page. Here, as the variation of the objects, the variation of the face size will be described as an example. For example, if you want to create a travel-themed layout, and you look at the layout results and see all the images with face sizes that are too similar, that may not be a good layout. The size of the face on the paper after layout may be small or large, and it is considered that a good layout is that they are arranged in a well-balanced manner. In this case, the one with a large variation in face size is evaluated highly. The variance value of the size of the face (diagonal distance from the upper left to the lower right of the face position) after being placed in the l-th temporary layout of interest is stored as tpFaceVariance [l]. Next, the maximum value MaxFaceVariance value in the mpFaceVariety [l] is obtained. Then, the evaluation value FaceVariance [l] of the degree of face size variation of the l-th temporary layout can be obtained as follows.
FaceVariance [l] = 100 × tpFaceVariance [l] / MaxFaceVariance
The FaceVariety [l] becomes a value closer to 100 as the variation of the face size arranged on the paper surface becomes larger, and closer to 0 as the variation is smaller. Therefore, it can be used as an evaluation value of the degree of variation in face size.

The variation of objects is not limited to those described above.

Another category is user preference evaluation.

For example, even if the layout has a low evaluation value based on the various evaluation amounts described above, it may be a good layout for the user as long as the layout includes a photograph that the user personally likes. It is preferable to evaluate using an evaluation amount based on user preference so as not to hinder the selection of such a layout.

As described above, since the user can set the favorite degree for each image in advance, the preference can be evaluated based on this. Further, for example, the preference can be automatically evaluated based on the number of browsing times and the browsing time. The information used for these evaluations can be calculated based on the information managed by the FavorteRate tag, the ViewingTimes tag, and the PrintingTimes tag.

For the first temporary layout, the average values of all images, FavoriteRateAve [l], ViewingTimesAve [l], and PrintingTimesAve [l] are obtained for each numerical value of the tag information of the images arranged in each slot.

Next, by taking the sum of these, the user's preference evaluation value UserFavor [l] can be obtained as follows.
UserFavor [l] = FavorteRateAve [l] + ViewingTimesAve [l] + PrintingTimesAve [l]
The UserFavor [l] shows a higher value as the layout uses an image with a high degree of user favorite, an image with a large number of views, and an image with a large number of prints. Therefore, in the present embodiment, it is determined that the higher the value of UserFavor [l], the higher the user preference for the layout.

The plurality of evaluation values calculated for each temporary layout as described above will be integrated below to obtain layout evaluation values for each temporary layout. The integrated evaluation value of the first temporary layout is set to EvalLayout [l], and the values of the N evaluation values (including each of the evaluation values in Table 3) calculated above are set to EvalValue [n]. At this time, the integrated evaluation value can be obtained as follows.

In the above equation, W [n] is the weight of each evaluation value for each scene shown in Table 3. This weight sets a different weight for each layout theme. For example, when the themes are compared between the growth record “growth” and the travel “travel” as shown in Table 3, it is often desirable for the travel theme to lay out as many high-quality photographs as possible in various situations. Therefore, the tendency is set to emphasize the individual evaluation value of the image and the balance evaluation value in the page. On the other hand, in the case of the growth record "growth", it is often more important whether or not the main character to be the target of the growth record is surely fitted to the slot rather than the variation of the image. For this reason, the tendency is set to emphasize the image / slot goodness-of-fit evaluation rather than the in-page balance and individual image evaluation. The importance of each theme in this embodiment is set as shown in Table 3. Using the EvalLayout [l] calculated in this way, S2109 generates a layout list LayoutList [k] for displaying the layout result. The method of creating the layout list LayoutList [k] will be described in detail later. This layout list LayoutList [k] is used to associate with the layout order l for which the evaluation has been completed, when the layout order is k-th when the rendering display is performed in S605 of FIG. In the contents of LayoutList [k], the value of the layout order value l is described. The layout list LayoutList [k] is displayed in order from the smallest k.

Returning to FIG. 6, the layout result obtained by the above processing is rendered in S605 of FIG. 6, and the result is displayed as shown in FIG. In S605, first, the layout identifier stored in LayoutList [0] is read, and the temporary layout result corresponding to the identifier is read from the secondary storage device 103 or the RAM 102. In the layout result, as described above, the template information and the assigned image name are set for each slot existing in the template. Therefore, based on this information, the layout result is rendered by using the drawing function of the OS operating on the image processing device 115, and is displayed as shown in FIG. 21. 2902.

In FIG. 21, by pressing the Next button 2904, the identifier with the next highest evaluation result, that is, the identifier of LayoutList [1], which is the runner-up score, is read, rendered in the same manner as described above, and then displayed. This allows the user to browse various variations of the proposed layout. Also, by pressing the Preview button 2903, the previously displayed layout can be redisplayed. Further, if the displayed layout is liked, the layout result 2902 can be printed out from the printer 112 connected to the image processing device 115 by pressing the print button 2905.

Here, the scoring of the amount of blur in the quantitative evaluation of the large amount (L pieces) of S2108 shown in FIG. 17 will be described in detail. In scoring the amount of blur shown in FIG. 17, first, the magnitude of blur is evaluated. The magnitude of the blur is evaluated using, for example, a cepstrum that can evaluate the magnitude (amount of blur) and the direction of the blur. As a method for calculating the amount of blur and the direction, a known method can be used, and examples thereof include the method disclosed in US7617826B2. In this method, the blurr kernel allows the magnitude and direction of blurring to be obtained. The method of calculating the amount of blur and the direction is not limited to this.

The amount of blur is not limited to that evaluated by the amount of blur and the direction of blur. For example, by calculating the amount of blur and the direction of blur, a difference from the blur effect using the depth of field is detected. May be evaluated. The blur referred to here is caused by a change in the relative relationship between the camera and the subject during the exposure time of the camera. It is possible to reduce the effect of blurring by maintaining the relative relationship between the image sensor and the optical axis during the exposure time by devising the optical system, but it may over-detect the vibration of the shooting system or the exposure time. When it is long, it cannot prevent blurring.
The amount of blur may be evaluated at once for the entire target image, or may be evaluated individually by dividing the area of the target image. Further, the weighting for each area may be changed to calculate the amount of blurring of the entire image. When dividing the area of the target image, for example, the amount of blurring according to the attribute such as the face area of the central person in the proposed story or the area of the added focus point may be evaluated. By evaluating the amount of blurring by dividing the area of the target image, the amount of blurring for each area can be calculated and the difference from the blurring effect using the depth of field can be detected.

Further, as described above, instead of calculating the amount of blurring in all areas included in the image and using it as the amount of blurring in the entire image, the amount of blurring in a predetermined area, in other words, the area of interest is calculated and the amount of blurring in the entire image is calculated. It may be an amount. By evaluating the amount of blurring in the region of interest, it is possible to reduce the frequency of erroneously determining the amount of blurring in an image aimed at the blurring effect. Next, the evaluation amount of image blur is calculated. The evaluation amount of image blur is calculated based on, for example, the magnitude of blur and the size (size) when the image is actually laid out. That is, the amount of blurring of the image is converted into the size when laying out. For example, the enlargement / reduction ratio is calculated from the size after layout and the size of the original image, and the product of the blur amount and the calculated enlargement / reduction ratio is used as the evaluation amount of image blur. The size after layout is evaluated from the viewpoint of observing the output (for example, printed matter). The output size of the image, specifically, the print size when printed on the paper surface according to the specified paper size is converted. As a result, the image laid out on the recording medium can be appropriately evaluated even if the conditions on the monitor display do not change.

For the evaluation amount of image blur, when a part of the original image is trimmed and used, the enlargement / reduction ratio is calculated from the size obtained from the trimming coordinates and the size after layout, and the amount of blur is calculated as the enlargement / reduction. It may be the product of the rate. If you want to propose a layout image that includes an image with blurring that allows face detection, you can calculate the amount of blurring in the face area from the coordinate position of face detection performed in S303, and then calculate the evaluation amount based on that amount. Good. Specifically, the product of the calculated amount of blur in the face area and the variable magnification (here, the enlargement / reduction ratio) obtained from the size after layout is used as the evaluation amount of image blur. In addition, the product of the blur amount and the enlargement / reduction ratio was used as the blur evaluation amount, but assuming that the observation distance differs depending on the layout size, the blur evaluation amount is calculated using the assumed observation distance. May be good. Assuming that the larger the layout image is, the more it is observed from a distance, the evaluation value is relatively lower than the product of the layout size and the amount of blurring.

The evaluation amount of blur calculated by the above method is normalized as an individual evaluation amount of an image and scored. In addition, in the case of individual image evaluation, when the evaluation amount of blur is combined with other evaluation amounts, it may be performed by multiplying by a weighting coefficient for each theme and then adding.

By evaluating the magnitude of the blur by the method described above, the amount of blur in the image laid out in a large size becomes large, and the individual evaluation of the image becomes low, that is, the score becomes low. On the other hand, the amount of blurring of an image laid out small is small, and the individual evaluation of the image is high, that is, the score is also high.

Therefore, a layout in which an image with a large blur is arranged in a small size has a higher evaluation than a layout in which an image with a large blur is arranged in a large size, and is easily proposed as a layout. In addition, an image with less blur is highly evaluated for a layout arranged in a large size, and is easily proposed as a layout. In the present embodiment, by evaluating the influence of the blurred image on the image after layout, it is possible to make a more preferable proposal for the user from the image data owned by the user.

In the present embodiment, it is possible to propose a suitable layout candidate to the user while suppressing the blur image from being missing from the layout candidates.

(Embodiment 2)
In the first embodiment, the evaluation of the layout image having the blurred image has been described, but in the present embodiment, the evaluation of the layout image having the image including noise will be described. The description of the same configuration as that of the first embodiment will be omitted.
In the present embodiment, a case where noise is evaluated as an individual image evaluation will be described. That is, in the present embodiment, noise scoring at the time of quantitative evaluation of a large amount (L pieces) of layout of S2108 shown in FIG. 17 will be described.

For example, when the exposure amount of the camera is small, the noise appears in the image as dark noise of the image sensor or the like.

To score the noise, first, the amount of noise is evaluated. The amount of noise can be calculated using, for example, sensitivity information (for example, ISO sensitivity) recorded at the time of shooting by the camera.

Further, for example, the noise in the low-sensitivity portion of each color filter may be calculated by the filter and used as the noise amount. Specifically, first, the frequency of noise is obtained by frequency analysis of the difference between the image after the noise reduction filter processing and the original image. Then, the frequency analysis result of this noise is used as the noise amount.

Next, the evaluation amount of noise in the image is calculated. The evaluation amount of noise in the image can be calculated by the same method as the evaluation amount of blur shown in the first embodiment. For example, it can be calculated based on the amount of noise and the size when the image is actually laid out. For the evaluation amount of noise in the image, when a part of the original image is trimmed and used, the enlargement / reduction ratio is calculated from the size obtained from the trimming coordinates and the size after layout, and the noise amount and the calculated enlargement / reduction are calculated. It may be the product of the rate.

The noise evaluation amount calculated by the above method is normalized as an image individual evaluation amount and scored. In the case of individual image evaluation, when the noise evaluation amount and other evaluation amounts are to be combined, the weighting coefficient for each theme may be multiplied and then added.

In the present embodiment, it is possible to suppress that the noise, which has low sensitivity due to the visual characteristics (VTF) due to the high frequency noise, is expanded to pseudo-decrease the frequency and become conspicuous. That is, in the present embodiment, the influence of the image containing noise on the image after layout is evaluated in consideration of the fact that the degree of noise allowed differs depending on the size of the layout. As a result, it is possible to make a more preferable proposal for the user from the image data owned by the user.

In the present embodiment, it is possible to propose a suitable layout candidate to the user while suppressing the image containing noise from being missing from the layout candidates.

(Other embodiments)
The basic configuration of the present invention is not limited to those described above. The above-described embodiment is a means for obtaining the effect of the present invention, and if the same effect as that of the present invention can be obtained even if another similar method is used or different parameters are used, the present invention is used. Included in the category.
In the present embodiment, an example is shown in which a user drags an image group with a mouse pointer to execute a combination process of a person group, but the operation is not limited to the operation with the mouse pointer.

Further, in the above-described embodiment, a person is taken as an example as an object, but the object is not necessarily a person. By recognizing pets such as dogs and cats and recognizing them, pets can be set as objects. Further, according to the shape recognition process such as edge detection, a building or an accessory can be recognized, so that the building or the accessory can be set as an object.

In the above-described embodiment, a computer has been described as an example of the image processing device, but the present invention is not limited thereto. For example, the present invention can be applied to devices that perform image processing such as printers, copiers, facsimile machines, mobile phones, PDAs, image viewers, and digital cameras.

Further, even if the present invention is applied to a system composed of a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), the present invention is a device composed of one device (for example, a printer, a copying machine, a facsimile machine, etc.). Etc.) may be applied.

The above-described embodiment is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiment is supplied to the system or device via a network or various storage media, and the computer (CPU, MPU, etc.) of the system or device reads and executes the program. It is a process to do. Further, the program may be executed by one computer or may be executed by interlocking a plurality of computers. Further, it is not necessary to realize all of the above-mentioned processes by software, and a part or all of the processes may be realized by hardware such as ASIC. Further, the CPU is not limited to one that performs all processing by one CPU, and a plurality of CPUs may perform processing while appropriately coordinating with each other.

Claims (15)

A program for a given application
On the computer
A step of acquiring a plurality of images including an image including a first person and an image including a second person different from the first person .
A step of analyzing a plurality of acquired images, recognizing a plurality of person's faces included in the plurality of images and objects other than the person's face, and identifying the recognized person's face for each person. When,
A step of displaying the first facial image corresponding to the face of the identified first person on the basis of the analysis on a display unit,
A step of receiving a predetermined user instruction related to the first face image while the first face image is displayed on the display unit, and
A step of determining a theme related to a predetermined event as a layout theme based on a user input different from the predetermined user instruction.
A step of setting the first person as the first hero based on the predetermined user instruction, and
Based on the determined theme, the step of setting the second person as the second hero, and
From the templates held for each theme, the step of selecting a template based on the determined theme, and
Based on the setting of the predetermined said set based on a user instruction first hero and the theme set on the basis of the second main character, the template selected based on the theme, the first A step of generating a layout image in which an image including the person and the image including the second person are arranged, and
A program that executes. A claim characterized in that a second face image corresponding to the face of the second person identified based on the analysis process is displayed together with the first face image corresponding to the face of the first person. Item 1. The program according to item 1. When a predetermined user instruction related to the second face image is received from the user while the first face image and the second face image are displayed on the display unit, the second face The program according to claim 2, wherein the second person is set as the second main character based on a predetermined user instruction related to an image and the determined theme. A claim characterized in that a layout image in which an image including the first person and the second person is preferentially arranged is generated based on the settings of the first hero and the second hero. The program according to any one of items 1 to 3. Based on the settings of the first hero and the second hero, the image including the first person and the second person does not include the first person and the second person. The program according to any one of claims 1 to 4, wherein a layout image is preferentially arranged over the image. When arranging an image in the template, the goodness of fit of the image to be arranged is determined with respect to a predetermined slot of the template.
The image including the first person is characterized in that the image including the first person is preferentially arranged by determining that the fit is high with respect to the predetermined slot. The program according to any one of claims 1 to 5. In the image including the first person, the predetermined score value is set higher than the image not including the first person, so that the image including the first person is preferentially arranged. The program according to any one of claims 1 to 6, wherein the program is characterized by being performed. With additional steps to evaluate the layout
When a predetermined image is enlarged and arranged in a predetermined slot in the template, the evaluation step is characterized in that the evaluation is performed based on a value that changes in connection with the enlargement processing. The program according to any one of Items 1 to 7. A claim characterized in that, when a predetermined image is enlarged and arranged in a predetermined slot in the template, a specific process is performed to make it difficult for the predetermined image to be adopted in the predetermined slot. The program according to any one of 1 to 8. The program according to any one of claims 1 to 9, wherein a screen capable of receiving an instruction related to a date from a user is displayed on the display unit. The program according to any one of claims 1 to 10, wherein the template used for the layout is acquired from an external server via the Internet. The program according to any one of claims 1 to 11, wherein the plurality of images can be acquired from an external server via the Internet. The program according to any one of claims 1 to 12, wherein the scenes of the plurality of images are analyzed and the shooting scenes of the plurality of images are distinguished. The program according to any one of claims 1 to 13, wherein in the analysis process, a person's face and a dog or a cat can be recognized as an object other than the person's face. The program according to any one of claims 1 to 14, further performing a step of transmitting data based on the generated layout image to an external device via the Internet.

Images