JP2018124776A - Information processing device, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a selection of an image as photo content other than an image selected beforehand after the photo content is created.SOLUTION: An information processing device according to the present invention includes: excluding means for excluding an image that does not match an image slot from an image group selected beforehand; automatic layout means for automatically laying out the left images not excluded in the image slot corresponding to photo content; image acquiring means for acquiring a new image that is to be replaced with the image selected as the image for replacement from the laid out images; and image determining means for not determining whether or not the selected image for replacement matches the image slot when the new image is acquired from the left images not excluded by the excluding means after the automatic layout, and for determining whether or not the selected image for replacement matches the image slot when the new image is acquired from the image other than the selected image beforehand after the automatic layout.SELECTED DRAWING: Figure 15

Description

  The present invention relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program for creating photo content.

  Conventionally, there is software for creating a photo album (hereinafter referred to as a photo album creation application). In this photo album creation application, when a user selects a folder that contains photos to be included in the album or multiple photos, the photos are analyzed, and the photos to be posted in the photo album are determined and automatically laid out. Apply processing. Such a function for automatically generating a photo album from a plurality of photos selected by the user is called an automatic layout function, and it is not necessary for the user to manually create an album from scratch. It is adopted in the creation software.

  Further, the user can individually edit a photo album created by the automatic layout function of the photo album creation application if there is an unsatisfactory portion. For example, when a user wants to change a photo set in a photo slot in a photo album to a different photo, the user manually selects the target photo slot and then manually replaces the photo. When manually exchanging photos, some conventional photo album creation applications present recommended photos as candidate images from photos selected in advance by the user (for example, Patent Document 1). . In this Patent Document 1, when creating a photo album, images designated by the user are divided into groups, the groups are associated with the pages of the photo album, and when the images in the image slot are replaced, the group corresponding to the page Candidate images are displayed.

JP 2013-33453 A

  However, in Patent Document 1, the image in the image slot is exchanged from images selected in advance by the user, and there is no mention regarding setting an arbitrary image other than the image selected in advance by the user in the album. . If an arbitrary image is replaced with an image slot, an image with a small resolution that is not suitable for an album may be selected.

  The present invention has been made in view of the above-described conventional problems, and an object thereof is to allow an image other than a preselected image to be selected as a photo content after the photo content is created.

  In order to achieve the above object, an information processing apparatus according to the present invention is an information processing apparatus that creates photo content, and includes an exclusion unit that excludes an image that does not match an image slot from an image group selected in advance by a user. , An automatic layout means for automatically laying out an image that has not been excluded by the exclusion means in an image slot corresponding to the photo content, and a replacement target from the image laid out by the automatic layout means An image acquisition unit that acquires a new image to be replaced with an image selected as the image, and the replacement when the new image is acquired from an image that is not excluded by the exclusion unit after the automatic layout process is executed. No determination is made as to whether it matches the image slot of the image selected as the target, If the new image is acquired from an image other than the image selected in advance by the user after the automatic layout process is executed, whether or not the image slot of the image selected as the replacement target is matched is determined. And an image determining means for determining.

  According to the present invention, it is possible to select an image other than a pre-selected image as a photo album after creating photo content.

It is a figure which shows the structure of a photo album creation / order system. It is a figure which shows the hardware constitutions of PC. It is a figure which shows the module structure of the software of PC. It is an example of a screen for creating / editing a photo album. It is a functional block diagram of a photo album creation application. It is a flowchart which shows the flow of a process of automatic layout. It is a figure which shows image analysis information. It is a figure which shows the process divided | segmented into a subscene. It is a figure which shows the average value and standard deviation value of an image feature-value in each scene. It is a figure explaining the process regarding scoring in an image scoring part. It is a figure explaining the process which selects the image to lay out from a scene image group. It is a figure which shows the template group used for the layout of an image. It is the figure which showed the example regarding replacement of a photograph. It is the figure which showed the example regarding replacement of a photograph. It is the figure which showed the example regarding replacement to a photograph. It is the figure which showed the example regarding replacement of a photograph. It is the figure which showed the example which cannot be replaced with another photograph. It is the flowchart which showed the flow of the process which replaces a photograph. It is the flowchart which showed the flow of the process of the photo album creation application. A case where a photograph taken with a camera is set in a photograph slot is illustrated. The case where the image read by the image reading apparatus is set in the photo slot is illustrated.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the present invention according to the scope of claims, and all combinations of features described in the present embodiments are essential to the solution means of the present invention. Is not limited.

  FIG. 1 is a diagram showing a configuration of a photo album creation / order system. The photo album creation / order system is configured by connecting information processing apparatuses 1 and 2 via a predetermined network such as the Internet. The information processing apparatuses 1 and 2 are configured by, for example, a general personal computer or a server (hereinafter sometimes abbreviated as a PC). The PCs 1 and 2 are installed with an OS equivalent to Windows 8 (registered trademark) or Windows Server 2012 (registered trademark) as an OS (Operating System).

  The PC 1 is installed in a home network for a general home (a network 4 configured by Ethernet (registered trademark) shown in FIG. 1) that is mainly installed at the home of a user (for example, a customer) and built at the home. Connected. The PC 1 includes a photo album creation application 80 and a web browser application 143.

  The photo album creation application 80 is an application for creating a photo album, and includes, for example, an executable file for Windows (that is, a file whose extension is indicated by “.exe”). The Web browser application 143 is a browser used when using the WWW (World Wide Web).

  The PC 2 is installed in the company (in FIG. 1, in the company of ABC), and is connected to an office network (network 8 made up of Ethernet (registered trademark) shown in FIG. 1) built in the company. The PC 2 includes a Web server 9 having a Web server function, and provides an ABC company Web site (that is, a shopping site 11) via the Internet. The shopping site 11 is a shopping site provided by ABC Company for users, and the user can order a photo album using the shopping site 11.

  FIG. 2 is a diagram illustrating a hardware configuration of the PC 1. The PC 1 includes a RAM 201, an HDD 202, a KBD 203, a CPU 204, an LCD 205, and an NB 206. A RAM (Random Access Memory) 201 is a main memory of the CPU 204 and functions as a work area or the like. An HDD (Hard Disk Drive) 202 is a storage device (internal storage means) that stores various data. Various data can also be stored in a portable CD-ROM or an internally installed ROM. A KBD (keyboard) 203 is an input device for inputting various settings.

  A CPU (Central Processing Unit) 204 is a central processing unit that controls the entire PC 1 and executes various programs such as an OS and application programs. Specifically, the CPU 204 reads the photo album creation application 80, the web browser application 143, and each module (software) shown in FIG. 3 stored in the HDD 202 into the RAM 201 and executes them. Accordingly, the CPU 204 realizes the functions of the photo album creation application 80, the Web browser application 143, and each module (software) shown in FIG.

  An LCD (Liquid Crystal Display) 205 is a display device (display unit) for display. An NB (Network Board) 206 is an expansion board for connecting to a network. In the PC 1, the units 201 to 206 of the PC 1 are connected to each other via a bus 207.

  FIG. 3 is a diagram illustrating a module configuration of software of the PC 1. That is, the PC 1 includes modules 301 to 309 as shown in FIG. The module 301 is an application module, and includes, for example, a photo album creation application 80, a web browser application 143, and the like.

  The module 302 is an API / DDI interface, and includes an Application Programming Interface (API) and a Device Driver Interface (DDI). The module 303 is a module as a device driver, and includes a standard driver 304 bundled with the OS as a standard and an IHV driver 305 provided from the Independent Hardware Vendor (IHV).

  A module 306 is a PnP-X control module that controls plug and play of the network. Note that “PnP-X” is an abbreviation of “Plug and Play Extensions”, which is a standard function installed in the Windows 8 OS as a series of plug-and-play extended functions that provide support for network connection devices.

  A module 307 is a WSD control module that controls a Web Service on Devices (WSD) that provides a device search mechanism on the network. The module 308 is an IP network control module that controls the IP network. A module 309 is an Ethernet control module that controls Ethernet.

  FIG. 4 is an example of a screen for creating / editing a photo album with the photo album creation application 80. In FIG. 4, a display area 401 is a display area of the photo album creation application 80, a display area 402 is a thumbnail display area of each page of the photo album, and a display area 403 is a display area of the photo album.

  In FIG. 4, when the user selects a desired thumbnail in the display area 402, the selected thumbnail (that is, the corresponding page) is in a selection state 404. FIG. 4 shows an example in which pages 3 to 4 are in the selection state 404. When pages 3 to 4 are selected by the user, the layout result of the photos on pages 3 to 4 (that is, the spread page of the photo album) Is displayed in the display area 403. The user can perform an editing operation on the display area 403.

  The photo slot 405 corresponds to the image slot of the present invention and is a slot for assigning a photo. A text box 406 is an area for inputting text. A button 407 is a button for putting the created / edited photo album into the cart. When the user selects the button 407, the created / edited photo album is uploaded to the shopping cart via the Internet. In the present embodiment, a screen as shown in FIG. 4 is displayed as a layout result in which photos are laid out in the slot 405.

  FIG. 5 is a functional block diagram of the photo album creation application. 5, FIG. 5 (a) is a functional block diagram of the entire photo album creation application 80, and FIG. 5 (b) is a functional block diagram of the automatic layout processing unit 900 constituting the photo album creation application 80. Indicated.

  First, a functional block diagram of the entire photo album creation application 80 in FIG. 5A will be described. The album data generation unit 800 generates a photo album as data according to the image (photograph) selected by the user. The album data generation unit 800 lays out an image determination unit 811 that determines whether the image selected by the user is an appropriate image for generating a photo album, and lays out the image selected by the user in a predetermined template. A layout processing unit 812. The layout processing unit 812 includes an automatic layout processing unit 900 that automatically lays out an image.

  The device control unit 820 is partially linked with the module shown in FIG. 3 described above to connect with an external storage device such as a memory or a hard disk device, an image reading device, or the like connected through various interfaces. Control communications such as data input / output. For example, the device control unit 820 performs control so as to adjust reading settings and the like in the image reading apparatus. Similarly to the device control unit 820, the network control unit 830 cooperates with the module illustrated in FIG. 3 described above to execute communication control processing with other devices connected to the network.

  The camera control unit 840 controls a camera built in or connected to the PC 1 and adjusts the shooting mode and executes shooting processing and the like. The storage control unit 850 controls to store various data in the HDD 202 of the PC 1 (for example, controls to copy and store image data stored in the external storage device). The display control unit 860 performs display control in the display device of the PC 1.

  Next, a functional block diagram of the automatic layout processing unit 900 of the photo album creation application 80 in FIG. 5B will be described. In FIG. 5B, an album creation condition designating unit 901 designates an album creation condition according to a UI operation described later to the automatic layout processing unit 900. The image acquisition unit 902 acquires the image group designated from the album creation condition designation unit 901 from, for example, the HDD 202 or the like. The image conversion unit 903 converts image data used for subsequent processing into a desired number of pixels and color information. In this embodiment, the image conversion unit 903 converts the number of short-side pixels into an analysis image of sRGB color information.

  The image analysis unit 904 executes later-described image feature amount acquisition, face detection, facial expression recognition, personal recognition, and the like from the analysis image. In addition, the image analysis unit 904 acquires the shooting date and time from data accompanying the image acquired from the HDD 202, for example, information such as Exif. The image classification unit 905 executes scene division and scene classification described later using the shooting date / time information, the number of images, and detected face information for the image group. The scene is a shooting scene such as travel, daily life, wedding, and the like.

  The main character information input unit 906 inputs the main character ID (identification information) designated by the album creation condition designation unit 901 to the image scoring unit 907. The image scoring unit 907 performs scoring on each acquired image so that an image suitable for the layout has a high score. Note that the image scoring unit 907 uses information from the image analysis unit 904 and information from the image classification unit 905 when performing scoring. In addition, the image scoring unit 907 is configured to be able to add a score for an image including the main character ID input from the main character information input unit 906.

  The spread number input unit 908 inputs the number of spreads of the album designated by the album creation condition designation unit 901 to the spread allocation unit 909. A spread assignment unit 909 divides the image group and assigns it to each spread. The spread allocation unit 909 divides the image group according to the spread number input from the spread number input unit 908, and allocates a part of the image group to each spread. The image selection unit 910 selects an image to be posted on the album with reference to the score scored by the image scoring unit 907 from a part of the image group assigned to each spread by the spread allocation unit 909. . For example, the image selection unit 910 preferentially selects an image with a high score. The template input unit 911 inputs a plurality of templates corresponding to the template information specified from the album creation condition specifying unit 901 to the image layout unit 912. The image layout unit 912 selects a template suitable for the image selected by the image selection unit 910 from a plurality of templates input from the template input unit 911, and determines an image layout.

  The image correction condition input unit 913 inputs the image correction ON / OFF condition designated by the album creation condition designation unit 901 to the image correction unit 914. When the image correction condition input from the image correction condition input unit 913 is ON, the image correction unit 914 performs, for example, dodging correction, red-eye correction, contrast correction, and the like on the image input from the image conversion unit 903. Perform image correction. The image correction unit 914 does not perform image correction on the image input from the image conversion unit 903 when the image correction condition input from the image correction condition input unit 913 is OFF. Note that the number of pixels of the image input from the image conversion unit 903 to the image correction unit 914 is changed according to the layout size determined by the image layout unit 912. The layout information output unit 915 outputs layout information for display on the display according to the image layout determined by the image layout unit 912. The layout information includes, for example, data necessary for creating an album such as bitmap data in which the selected image is laid out in the selected template.

  FIG. 6 is a flowchart showing the flow of automatic layout processing in the automatic layout processing unit 900 of the photo album creation application 80. The flowchart shown in FIG. 6 is realized, for example, when the CPU 204 reads out a program stored in the HDD 202 to the RAM 201 and executes it.

  Note that the user selects an image data group to be arranged in the album before starting the processing of FIG. Then, the photo album creation application 80 specifies image data suitable for automatic layout, which will be described later with reference to FIG. 6, from the selected image data group. Specifically, in the image data group, the image data corresponding to the condition 1 is excluded as image data not suitable for automatic layout. Note that the image data corresponding to Condition 1 is image data whose format is not compatible, image data whose size is larger than the predetermined size, image data whose size is smaller than the predetermined size, and aspect ratio that does not match the predetermined ratio. Image data. In the image data group, image data that does not correspond to condition 1 but corresponds to condition 2 is also excluded as image data that is not suitable for automatic layout. The image data corresponding to condition 2 is image data that does not include information such as Exif. The photo album creation application 80 excludes the image data corresponding to the condition 1 and the image data corresponding to the condition 2 from the image data group selected before starting the processing of FIG. 6, and uses the remaining image data. The subsequent processing of FIG. 6 is executed. That is, when the above-described processing is not executed, analysis processing or the like is executed for image data that is not suitable for automatic layout, and thus there is a possibility that efficient automatic layout processing cannot be executed. A further object of this embodiment is to execute an efficient automatic layout process.

  The image conversion unit 903 converts each image in the image group of the HDD 202 designated by the album creation condition designation unit 901 into an analysis image having a desired number of pixels and color information (S01). In the present embodiment, as described above, the image is converted into an analysis image of short side pixels and sRGB color information.

  The image analysis unit 904 acquires an image feature amount from the analysis image converted by the image conversion unit 903 (S02). Further, as described above, the image analysis unit 904 acquires the shooting date and time from information such as Exif attached to each image data acquired from the HDD 202. Note that examples of the image feature amount include focus, and whether or not the image is in focus can be determined by calculating an average inclination of edges in the image. Here, it is known that a Sobel filter is generally used for edge detection, and the average inclination of the edge in the image is detected by the Sobel filter and the luminance difference between the start point and the end point of the edge is detected. It can be calculated by dividing by the distance between the start point and the end point. An image having a large average inclination of edges in the image can be considered to be in focus as compared with an image having a small average inclination of edges in the image. Furthermore, it is possible to determine the focus amount by setting a plurality of threshold values having different values with respect to the average inclination of the edge in the image and determining which threshold value or more. In the present embodiment, two different threshold values are set, and the focus amount is determined in three stages of ◯ Δ ×. Note that the focus inclination to be adopted for the album is ◯, the acceptable focus inclination is Δ, and the unacceptable inclination is x.

  The image analysis unit 904 executes face detection from the analysis image converted by the image conversion unit 903 as one of the image feature amounts (S03). A known method can be used for face detection, for example, Adaboost that creates a strong classifier from a plurality of weak classifiers. In the present embodiment, the face of a person (object) is detected by a strong classifier created by Adaboost and extracted as an image, and the upper left coordinate value and the lower right coordinate value of the detected face position are acquired. Thus, since the upper left coordinate value and the lower right coordinate value of the face position are acquired, the face position and the face size can be known. It should be noted that the strong classifier created by Adaboost can detect not only the face but also animals such as dogs and cats, flowers, food, buildings, figurines and the like instead of the face. That is, an object other than the face can be detected.

  The image analysis unit 904 performs personal recognition from the analysis image converted by the image conversion unit 903 (S04). Specifically, the image analysis unit 904 compares the face image extracted in step S03 with a representative face image stored for each personal ID in the face dictionary database to evaluate the similarity. Then, among the representative face images stored for each personal ID in the face dictionary database, the individual is recognized from the representative face image having the similarity that is equal to or higher than the threshold and the highest similarity. If the similarity is less than the threshold, the face image extracted in step S03 is associated with the new personal ID and registered in the face dictionary database. Further, the process related to personal recognition in step S04 is executed according to the number of faces extracted in step S03.

  The image analysis information acquired in steps S02 to S04 is stored separately for each image ID for identifying each image, as shown in FIG. Specifically, the information regarding the shooting date and time acquired in step S02, the determination result of the focus amount, the number of faces detected in step S03, and the position information of the detected face are stored. The face position information is stored separately for each personal ID acquired in step S04.

  The automatic layout processing unit 900 determines whether or not the processing of steps S01 to S04 has been completed for all the images in the image group of the HDD 202 specified by the album creation condition specifying unit 901 (S05). The automatic layout processing unit 900 returns the process to step S01 when the process from step S01 to S04 is not finished (S05 No), and returns to step S06 when the process from step S01 to S04 is finished (S05 Yes). The process is transferred to.

  In step S06, the image classification unit 905 executes scene division (S06). The image classification unit 905 divides the image group into a plurality of scenes by using the shooting date / time information acquired in step S02. For example, when there is a date that has not been shot between images, the image classification unit 905 executes the division process on the assumption that the scene has been changed between the images. In addition, when the shooting dates are consecutive, the image classification unit 905 executes the dividing process as follows, for example. If the time difference between the images is more than 16 hours, the image classification unit 905 performs the division process on the assumption that the scene has been changed between the images. When the time difference between images is less than 16 hours, if the time difference from the first shooting to the last shooting on each successive day is less than 4 hours, the images from the first shooting to the last shooting on each day are the same. Divide as a scene. When the time difference from the first shooting to the last shooting on each successive day is 4 hours or more, if the number of shots on each successive day is less than 50, the image group for each day is divided into one scene. If there are more than 50, do not divide. FIG. 8A shows the result of division by the scene division method described above.

  After executing the scene division, the image classification unit 905 next executes scene classification (S07). Hereinafter, processing relating to scene classification will be described using an example of classifying a scene (that is, a scene) into travel, daily life, and a ceremony. Note that the processing related to scene classification is executed using the scene classification table of FIG. Here, the scene classification table will be described. The creator of the photo album creation application 80 collects images determined as travel, daily life, and ceremony as scene image groups before or during the creation of the photo album creation application 80. Then, the creator of the photo album creation application 80 acquires an image feature amount for each collected scene image group. Here, the acquired image feature amount is, for example, a shooting period, the number of shots, the number of shots, and the like. The shooting period is the time difference from the shooting date and time of the first image of the scene image group to the shooting date and time of the last image, the number of shots is the number of shots of the scene image group, and the number of shots is the number of faces shot per image Defined as the number of people (ie, the number of people).

  The creator of the photo album creation application 80 further provides, for each group of scene images, an average value and standard deviation value of the shooting period, an average value and standard deviation value of the number of shots, and an average value of the number of photographers per image. And calculate the standard deviation. The scene classification table of FIG. 9 is created using the calculation result. FIG. 9 shows the average value and standard deviation value of the shooting period, the average value and standard deviation value of the number of shots, and the average value and standard deviation value of the number of shots per image in each scene. The creator of the photo album creation application 80 incorporates these calculated values (scene classification table) in the program of the photo album creation application 80 in advance.

When the processing from step S01 to S06 is executed in this state, the photo album creation application 80 uses the average value of the shooting period, the number of shots, and the number of shots as an image feature amount for each divided scene image group. calculate. The photo album creation application 80 uses the average value and the standard deviation value (that is, the pre-calculated average value and standard deviation value) of FIG. 9 regarding the shooting period, the number of shots, and the image feature values of the number of photographers of each divided scene image group. By using, scoring shown below is executed.
Score = 50− | 10 × (average value−image feature amount) / standard deviation value | (Formula 1)
Average score = (Score for the shooting period + Score for the number of shots + Score for the number of people)
/ Number of feature items (Formula 2)

  From the above (Equation 1) and (Equation 2), the average score for each trip, daily life, and ceremony is calculated for each divided scene image group. Then, with respect to the divided scene image group, a scene having the highest average score among travel, daily life, and ceremony is classified as a suitable scene. For example, in FIG. 8A, regarding the image feature amount of the divided scene image group (5), the shooting period is 36 hours, the number of shots is 300, and the number of shots is 1.7. In this case, the average travel score is 45.32, the daily average score is 18.38, the average score for the ceremony is -29.92, and the divided scene image group (5) is classified as a trip as a scene. Is done. Further, the classified scene image group is managed by a scene ID so that it can be identified. If the average score is the same, the scene is classified into a high priority scene. Here, in this embodiment, the priority is in the order of daily life> ceremony> travel, and the daily priority is the highest.

  The automatic layout processing unit 900 determines whether or not the scene classification has been completed in step S07 for all the scene image groups divided in step S06 (S08). If the automatic layout processing unit 900 determines that the scene classification has not ended (S08 No), the process returns to step S07, and if it determines that the scene classification has ended (S08 Yes), step S09. The process is transferred to.

  In step S09, the image scoring unit 907 executes processing relating to the setting of the main character (S09). The image scoring unit 907 obtains the number of appearances of each individual appearing in the image group, the number of appearances of each individual appearing in each scene, and the appearance of each individual from the result of personal recognition in step S04 and the result of scene division in step S06. Count the number of scenes to be played and set the main character. There are two types of setting for the main character: automatic setting and manual setting.

  In the automatic setting, when there are a plurality of scenes in the image group specified by the album creation condition specifying unit 901, an individual appearing in the plurality of scenes is set as the main character, and when the image group is a single scene, The individual with the most appearances in a scene is set as the main character. Further, in the manual setting, assuming that the hero setting icon is designated by the user, the image scoring unit 907 is based on the setting of the hero (that is, the personal ID) input from the hero information input unit 906. Identify the individual from the image and set it as the main character. In this case, the automatic setting is invalidated.

  When the image scoring unit 907 sets the main character (S09), the image scoring unit 907 executes processing related to scoring (S10). Note that scoring means assigning a score evaluated for each image, and the score is a value that the image selection unit 910 refers to when selecting an image, as will be described later. The image scoring unit 907 assigns a score for the main slot and a score for the subslot for each image as scoring. Here, the main slot and the slot will be briefly described with reference to FIG.

  FIG. 12 shows a group of templates used for image layout. The template group includes a plurality of templates 1601, and the template 1601 includes a main slot 1602 and sub slots 1603 and 1604. As shown in FIG. 12, the main slot 1602 is a main slot (frame for laying out an image) in the template 1601 and has a feature that the size is larger than the subslots 1603 and 1604.

  Next, specific processing related to scoring in the image scoring unit 907 will be described with reference to FIG. FIG. 10A is a table in which the characteristics of images used in the album in each scene of travel, daily life, and ceremony are divided into main slots and sub slots. The creator of the photo album creation application 80 collects in advance a plurality of images that match the image characteristics of the main slot and the image characteristics of the subslot in the travel, daily life, and ceremony scenes shown in FIG. To do. Then, the creator of the photo album creation application 80 totals image features such as the number of faces, the position of the face, and the size of the face for each of the collected images. The creator of the photo album creation application 80 further calculates the average value and standard deviation value of the image features such as the number of faces, the face position, and the face size in each slot, and stores them in the program of the album creation application. Keep it.

In this state, when the processing from step S01 to S09 is executed (that is, when the setting process of the main character in step S09 is completed), the photo album creation application 80 executes processing related to scoring by the image scoring unit 907. (S10). It is assumed that, at the time of execution of the processing related to scoring, scenes have already been classified and the main character has been set for the image to be processed. The image scoring unit 907 corresponds to the scene of the image to be scored, the average value and the standard deviation value calculated in advance, the number of faces of the main character in the image to be scored, the face position, For each feature quantity such as face size, a score and an average score are calculated using the following formulas.
Score = 50− | 10 × (average value−image feature amount) / standard deviation |
Average score = (Face number score + Face position score + Face size score)
/ Number of feature items (Formula 4)

  FIG. 10B shows, as an example, the result of calculating the scores according to (Equation 3) and (Equation 4) for each image to be scored, as shown in FIG. 10 (b). As described above, the main slot and the sub slot are scored in association with the image ID. For example, if FIG. 10B shows the calculation result of the travel scene score, the image with the image ID 2 is specified as an image suitable for the main slot. Note that the image used for the album is preferably an in-focus image. Therefore, in the image analysis information shown in FIG. 7, a score is separately provided for an image with an image ID whose focus feature is indicated by a circle. , May be added.

  The automatic layout processing unit 900 determines whether or not the processing in step S10 has been completed for all the images in the image group of the HDD 202 specified by the album creation condition specifying unit 901 (S11). The automatic layout processing unit 900 returns the process to step S10 when the process of step S10 is not completed (S11 No), and shifts the process to step S12 when the process of step S10 is completed (S11 Yes). To do.

  In step S <b> 12, the spread allocation unit 909 determines whether the number of scenes divided by the image classification unit 905 is the same as the number of spreads input from the spread number input unit 908. When the spread allocation unit 909 determines that the number of divisions and the number of spreads are the same (Yes in S12), the automatic layout processing unit 900 shifts the processing to step S16. Specifically, in FIG. 8A, the number of divided scene image groups is eight. In this case, if the number of spreads input to the spread number input unit 908 is eight, automatic layout processing is performed. The unit 900 shifts the process to step S16. In addition, when the spread allocation unit 909 determines that the number of divisions and the number of spreads are not the same (No in S12), the automatic layout processing unit 900 shifts the process to step S13.

  In step S <b> 13, the spread allocation unit 909 determines whether the number of scenes divided by the image classification unit 905 is smaller than the number of spreads input from the spread number input unit 908 (S <b> 13). When the automatic layout processing unit 900 determines that the number of divisions is smaller than the number of spreads by the spread allocation unit 909 (S13 Yes), the automatic layout processing unit 900 shifts the processing to step S14. Specifically, in FIG. 8A, the number of divided scene image groups is 8, and in this case, if the number of spreads input to the spread number input unit 908 is 10, the automatic layout processing unit In step 900, the process proceeds to step S14. If the automatic layout processing unit 900 determines that the number of divisions is larger than the number of spreads by the spread allocation unit 909 (No in S13), the automatic layout processing unit 900 shifts the processing to step S15.

  If the spread allocation unit 909 determines that the number of divisions is smaller than the number of spreads (Yes in S13), the scene image group divided by the image classification unit 905 is further divided into sub-scenes (S14). That is, the spread allocation unit 909 performs subdivision processing on the scene image group divided by the image classification unit 905 so that the number of spreads and the number of spreads are the same when the number of scene divisions <the number of spreads. Apply.

  Here, the process of dividing into sub-scenes in step S14 is illustrated using FIG. As shown in FIG. 8A, it is assumed that the number of scene divisions as the processing result of S06 is 8, and the spread number input from the spread number input unit 908 is 10. FIG. 8 (b) shows the result of dividing FIG. 8 (a) into sub-scenes, and is indicated by broken line arrows ((2) and (3), (6) and (7) in FIG. 8 (b)). By dividing into sub-scenes, the number of scene divisions is adjusted to 10.

  In the following, the division criteria when dividing into sub-scenes are shown. First, the spread allocation unit 909 searches for a divided portion where the number of images constituting the scene image group is large among the divided scene image groups shown in FIG. Here, in order to increase the number of divisions from 8 to 10 and adjust the number to be the same as the number of spreads, the spread allocation unit 909 sequentially selects two scene image groups having a large number of images. In the example shown in FIG. 8A, the spread allocation unit 909 selects the scene image group (5) and then the scene image groups (1) and (2) having the same number of images in descending order of the number of images. set to target.

  Further, when selecting the scene image group, the spread allocation unit 909, when the number of images in the scene image group is the same, from the shooting date / time of the first image constituting the scene image group to the shooting date / time of the last image. A scene image group having a larger time difference is selected as a division target. That is, in the example shown in FIG. 8A, for the scene image groups (1) and (2) having the same number of images, the last image is shot from the shooting date and time of the first image constituting the scene image group. A scene image group (2) having a larger time difference until the date and time is selected as a division target.

  Then, the spread allocation unit 909 performs subdivision processing on the selected scene image groups (5) and (2). Here, the scene image group (5) is an image group photographed over three days, and the number of images is distributed in three places as shown in FIG. 8 (a). In this case, the spread allocation unit 909 executes the division process so that the photographing date changes and the difference in the number of sheets after the division is further reduced. That is, the dividing process is executed at the location indicated by the broken-line arrow in FIG. 8B (between (6) and (7), between the second day and the third day in the scene image group (5)). In addition, since the scene image group (2) is an image group photographed over two days, the spread allocation unit 909 performs the location indicated by the broken line arrow in FIG. (Between (2) and (3)). Although FIG. 8 shows an example in which the shooting date is changed at a place where the shooting date changes, if all of the images constituting the scene image group selected as the division target are shot in one day, the scene image group is selected. Division processing is executed at a location where the time difference between shooting dates and times becomes the maximum among the images to be configured.

  If the spread allocation unit 909 determines that the number of divisions is equal to or greater than the number of spreads (No in S13), the scene merging process is performed on the scene image group that is divided into scenes by the image classification unit 905 (S15). Here, the scene integration process in step S15 will be exemplified with reference to FIG. As described above, in FIG. 8A, the number of scene divisions that is the processing result of S06 is eight, and here, the number of spreads input from the spread number input unit 908 is six. . In this case, for example, as shown in a broken line portion in FIG. 8C, two portions of the scene image group obtained by scene division are integrated.

  The following is a standard for integrating scenes. Here, in order to adjust the division number from 8 to 6, in FIG. 8A, two scene image groups divided into scenes with a small number of images are determined. In FIG. 8 (a), the scene image groups (8), (3), and (7) start from the smallest number of images in the scene image group, and the scene image groups (3) and (7) are the same. The number of images. In this case, since the scene image group (8) is the scene image group having the smallest number of images, the scene image group (8) is preferentially determined as an integration target, and the scene image group (7) is changed to the scene image group (8). Since it is adjacent, it is integrated with the scene image group (8). Therefore, the spread allocation unit 909 determines the scene image group (3) as the integration target as the second scene image group.

  Next, specific scene image group integration processing will be described in the order of the scene image group (3) and the scene image group (8). When the time difference between the scene image group (2) and the scene image group (2) before and after the scene image group (3) is compared, the time difference between the scene image group (4) and the scene image group (4) is smaller. (4) and the integration process are executed. In addition, regarding the scene image group (8), since it is the scene image group positioned last in time among the image groups, the integration processing with the scene image group (7) is executed.

  When the number of scene divisions and the number of spreads are adjusted to the same number (S12) or the same number (S14, S15), the spread allocation unit 909 executes processing related to spread allocation (S16). In addition, regarding the process related to the spread allocation, the scene image group positioned first in time is sequentially allocated to the spread.

  The image selection unit 910 selects an image to be laid out in a spread from the scene image group assigned to the spread (S17). FIG. 11 is a diagram illustrating an example in which four images to be laid out are selected from the scene image group assigned to spread. FIG. 11A is a diagram showing an image of a scene image group assigned to spread in accordance with the shooting date and time. In other words, the time difference (divided shooting period) from the first captured image to the last captured image is shown as an image of the scene image group assigned to spread.

  FIG. 11B is a diagram for explaining a method of selecting the first image to be laid out in a spread from the scene image group assigned to the spread. As illustrated in FIG. 12 described above, the template has one main slot, and the image selection unit 910 selects an image for the main slot as the first image. The image selection unit 910 selects an image having the highest score for the main slot scored in step S10 from the images in the divided shooting period shown in FIG.

  The image selection unit 910 selects an image for a subslot as the second image. Note that when selecting an image for a subslot, the divided shooting period is subdivided so as not to concentrate on one period of the divided shooting period. Hereinafter, a description will be given with reference to FIGS. First, as shown in FIG. 11C, the divided photographing period is divided into two. Next, as shown in FIG. 11D, the second image is selected from the divided shooting period (solid line in the figure) in which the first image is not selected. Note that when selecting the second image, the image having the highest score for the subslot is selected from the images corresponding to the solid line divided shooting period.

  When the second image is selected, the image selection unit 910 further divides each divided shooting period shown in FIG. 11D into two as shown in FIG. Then, as shown in FIG. 11 (f), the image selection unit 910 obtains a score for the subslot among the images corresponding to the solid line divided shooting period in which the first and second images are not selected. The highest image is selected as the third image.

  Also, as a supplement, an example of selecting an image (here, an example of selecting the fourth image) when there is no image to be selected in the segmented divided shooting period is shown in FIGS. 11 (g) to 11 (i). Will be described. According to the image selection rule described above, after the third image is selected up to FIG. 11 (f), the image in the divided shooting period in which no image is selected, as indicated by the diagonal lines in FIG. 11 (g). Should be selected. However, there may be a case in which no image exists during the divided shooting period indicated by the oblique lines in FIG. Therefore, the image selection unit 910 further divides each divided shooting period into two as shown in FIG. Next, as shown in FIG. 11 (i), the image with the highest score for the subslot is selected as the fourth image among the images corresponding to the solid divided shooting period in which the first to third images are not selected. To do.

  When an image is selected by the image selection unit 910, the image layout unit 912 determines the layout of the image (S18). FIG. 12 is a diagram for explaining layout processing in the image layout unit 912. In FIG. 12, description will be made using an example in which templates (1-1) to (4-4) are input to a certain spread according to specified template information by a template input unit 911. Further, the number of slots of the input templates (1-1) to (4-4) is three. Furthermore, in the lower part of the figure, regarding the three images selected by the image selection unit 910, whether the orientation is in the vertical direction or the horizontal direction is shown corresponding to the shooting date and time.

  In the example shown in FIG. 12, an image 1605 is an image for a main slot, and an image 1606 and an image 1607 are images for a subslot. In the present embodiment, an image having an older shooting date and time is laid out in the upper left of the template, and an image having a newer shooting date and time is arranged in the lower right. As described above, the image 1605 is an image for the main slot, and since the shooting date and time is the last image in time (that is, the newest image), (3-1) to (3) in FIG. -4) is selected as a template candidate. Regarding the image for the subslot, the image with the old shooting date and time is the image 1606 (that is, the vertical image), and the image with the new shooting date and time is the image 1607 (that is, the horizontal image). It is selected as the most suitable template for the image to be laid out.

  As described above, the image layout unit 912 determines, as information, in which slot of the template the image selected by the image selection unit 910 is to be laid out. FIG. 12 illustrates the case where the number of slots is three. However, in the case where the number of slots is four, similarly, a template set with four slots is prepared, and the compatibility between the slot and the image is determined. Thus, what image should be laid out in which template should be determined.

  When the image correction condition input from the image correction condition input unit 913 is ON, the image correction unit 914 performs image correction on the image input from the image conversion unit 903 (S19). The image correction unit 914 automatically executes, for example, dodging correction, red-eye correction, contrast correction, and the like as image correction. Note that the image correction unit 914 performs image correction on an image that has 1900 short-side pixels and is converted to an sRGB color space. The image correction unit 914 does not perform image correction on the image input from the image conversion unit 903 when the image correction condition input from the image correction condition input unit 913 is OFF.

  The layout information output unit 915 creates layout information (S20). Specifically, the photo album creation application 80 lays out the corrected image in each slot corresponding to the template determined in step S18 when the image correction is performed in step S19, and generates bitmap data. To do. Layout information including the bitmap data and the like is created in S20. The layout information output unit 915 resizes and lays out the image to be laid out in accordance with the size information of the photo slot. As another form of data included in the layout information, data indicating an association between each slot and image data may be included in the layout information instead of bitmap data. The automatic layout processing unit 900 determines whether or not the processing of step S17 to step S20 has been completed for all spreads (S21). When the automatic layout processing unit 900 determines that the processes in steps S17 to S20 have not been completed for all spreads (No in S21), it returns the process to step S17 and determines that it has been completed ( S21 Yes), the automatic layout process is terminated.

  The automatic layout processing in the photo album creation application 80 has been described above. Hereinafter, the processing for replacing a photo set in the photo slot by automatic layout with another photo will be described with reference to FIGS.

  FIG. 13 is a diagram showing an example in which a photo laid out in the photo slot by the automatic layout process is replaced with one of the photos selected in advance by the user. In the photo album creation application 80, when the user selects a photo slot A1 whose photo is to be replaced from the state shown in FIG. 13A, the state shown in FIG. That is, the photo album creation application 80 displays that the photo slot A1 is selected by displaying the frame of the photo slot A1 selected by the user thickly, changing the color, or highlighting the frame. Show. Further, as shown in FIG. 13B, the photo album creation application 80 simultaneously displays processing that can be edited in the photo slot A1. Here, as shown in FIG. 13 (b), as the processes that can be edited, the processes of “photo replacement 1301”, “photo editing 1302”, “automatic photo correction 1303”, and “red-eye correction 1304” are executed. Button is displayed near the photo slot A1.

  When the user selects “Photo exchange 1301”, the photo in the target photo slot (here, photo slot A1) can be manually exchanged. When the user selects “photo editing 1302”, it is possible to trim or rotate the photo. In addition, when the user selects “automatic photo correction 1303”, correction can be automatically performed according to the content of the photo, and when “red eye correction 1304” is selected, the red eye of the person is corrected. Can do.

  Here, a case where the user selects “Photo exchange 1301” for the purpose of exchanging pictures will be described. The photo album creation application 80 displays a recommended photo dialog 1305 as shown in FIG. 13C when the user selects “Replace photo 1301”. Note that the image a1 in the photo slot A1 in FIG. 13 is referred to as an image selected as a replacement target (or a replacement target image). The photo album creation application 80 displays a recommended photo suitable for the target photo slot A1 in the recommended photo dialog 1305. Here, R1 to R4 are displayed as recommended photos. Here, a display method of recommended photos will be described. The photo album creation application 80 excludes images that are not suitable for the album (for example, images with too low resolution) from photos selected by the user before the automatic layout process of FIG. 6 is executed. Then, the above-described processing of FIG. 6 is executed for the photos that are not excluded. After that, the photo album creation application 80 displays a recommended photo in a group of images (that is, any one of (1) to (8) in FIG. 8) associated with the spread that is the editing target. The image which is not arrange | positioned at a slot from among is identified. Then, an image with a high score among the identified images is displayed as a recommended photo. If the selected slot is a main slot, a recommended photo is selected based on the score calculated for the main slot. If the selected slot is a subslot, the score calculated for the subslot is used as a reference. Recommended photos are selected for. If the user wants to replace the photo a1 in the photo slot A1 in the recommended photo dialog 1305, the user can replace the photo a1 in the photo slot A1 by selecting the photo to be replaced. In this recommended photo dialog 1305, a photo that is selected by the user in advance of creating an album and that is suitable for the photo slot A1 is presented by the automatic layout processing unit 900. In other words, the recommended photo dialog 1305 does not present an image that is not suitable for the layout, such as a low-resolution photo.

  However, even if the recommended photo dialog 1305 is displayed, there are cases where the photo that the user wants to select is not among the photos displayed in the recommended photo dialog 1305. In this case, the user selects a “other photo” button 1306, and as shown in FIG. 13D, the photo is different from the photo displayed in the recommended photo dialog 1305 (that is, the photos L1 to L18). 18 photos) can be displayed. Note that FIG. 13D may include recommended photos R1 to R4.

  With respect to the photos shown in FIG. 13D, images that are not suitable for the album (for example, images with too low resolution, etc.) are excluded from photos selected by the user in advance of creating the album by the automatic layout processing unit 900. ing. The user can set the selected photo in the photo slot A1 by selecting one desired photo from these photos L1 to L18.

  In FIG. 13D, when the user selects the “Back” button 1307, the photo album creation application 80 cancels the process related to the photo replacement and returns to the screen of FIG. In addition, when the user selects the “Add Photo” button 1308, the photo album creation application 80 shifts the processing so that an arbitrary photo other than the photo previously selected by the user can be selected. The process of selecting an arbitrary photo will be described later with reference to FIG.

  FIG. 14 is a diagram showing an example in which the photograph L18 displayed in FIG. 13D is temporarily selected. Here, the provisional selection refers to selection at an undetermined stage until the replacement of photos is confirmed. When the photo L18 is temporarily selected by the user in FIG. 14A, the photo album creation application 80 enlarges and displays the photo L18 as shown in FIG. 14B. The user confirms the enlarged photo L18 and determines whether to replace the photo in the photo slot A1. Here, when the user selects the “Cancel” button 1401, the photo album creation application 80 cancels the process relating to the photo exchange and returns to the screen of FIG. When the user selects the “OK” button 1402, the photograph a1 in the photograph slot A1 is replaced with the photograph L18.

  In FIG. 14B, an area other than the display area of the photograph L18 is displayed so that the enlarged photograph L18 is highlighted and the photographs L1 to L18 displayed on the back of the photograph L18 are not conspicuous. , A gray mask 1403 is applied. This assists the user in specifying the photo. In FIG. 14B, when the “<” button 1404 is selected, the photograph L17 is enlarged and when the “>” button 1405 is selected, the photograph L19 is enlarged and displayed as shown in FIG. 14C. . Further, in this case, the display on the back side of the enlarged photograph L19 is also switched. In other words, the photo album creation application 80 switches the display contents for an area other than the display area of the enlarged photo in conjunction with the enlarged photo. Specifically, focusing on the back surface with the gray mask 1403 applied, the display contents are switched so that the photograph L19 is not displayed in FIG. 14B, but the photograph L19 is displayed in FIG. ing. When the user selects the “OK” button 1402 in FIG. 14C, the photo album creation application 80 replaces the photo a1 in the photo slot A1 with the photo L19 (FIG. 14D). An image arranged instead of the image selected as the replacement target is called a new image. That is, in the example of FIG. 14, the photograph a1 is an image selected as a replacement target, and the photograph L19 is a new image.

  FIG. 15 is a diagram illustrating an example in which a photo laid out in the photo slot by the automatic layout process is replaced with another photo selected from a photo other than a photo preselected by the user (other than an image). In FIG. 15A (ie, FIG. 13D described above), when the user selects the “Add Photo” button 1308 at the upper right of the display area 401, a file explorer 1501 is displayed as shown in FIG. 15B. Is displayed. The file explorer 1501 shown in FIG. 15B may be an OS standard file explorer or a file explorer created independently by the photo album creation application 80.

  In the example shown in FIG. 15, since the photo format corresponding to the photo album creation application 80 is only JPEG, only the JPEG image is displayed in the file explorer 1501 in FIG. Also, in FIG. 15B, when the photo P2 is provisionally selected by the user, the photo album creation application 80 enlarges and displays the photo P2 (FIG. 15C). Further, in FIG. 15C, when the “OK” button 1402 is selected by the user, the photo album creation application 80, as shown in FIG. 15D, displays the photo a1 in the target photo slot A1 as the photo P2. Replace with.

  FIG. 16 is a diagram showing an example in which a photo laid out in a photo slot by automatic layout processing is replaced with another photo by drag and drop. Here, an example is shown in which the photo a1 set in the photo slot A1 in FIG. 16B is replaced with another photo. FIG. 16A shows a file explorer 1501 for displaying a file, and the file explorer 1501 is generally attached to the OS. In FIG. 16A, eight photographs P1 to P8 are displayed as JPEG images on the file explorer 1501.

  Next, the user drags and drops the photo P8 displayed in the file explorer 1501 in FIG. 16A to the photo slot A1 as shown in FIG. 16B. Then, as shown in FIG. 16C, the photo a1 in the photo slot A1 is replaced with the photo P8. As shown in FIG. 16, the user can select an arbitrary photo as a replacement target from the file explorer 1501 by drag and drop. When the user performs a drag-and-drop operation, the photo album creation application 80 determines whether or not the photo that the user is going to replace is appropriate for laying out in the album. Replace. In the example shown in FIG. 16, since the photo album creation application 80 has determined that the photo P8 is suitable for layout in the album, the photo in the photo slot A1 is replaced with P8. Next, a case where it is determined that the photograph that the user is going to replace is unsuitable due to the resolution being too low when laying out the album will be described with reference to FIG.

  FIG. 17 is a diagram showing an example in which a photo laid out in a photo slot by automatic layout processing cannot be replaced with another photo by drag and drop. In FIG. 17, as shown in the state of FIG. 17B from the state of FIG. 17A, the photograph P7 is dragged and dropped to replace the photograph a1 in the photograph slot A1 with the photograph P7. However, since the photo P7 that the user is going to replace is an image with a low resolution, the photo album creation application 80 replaces the error (that is, the photo P7 in the photo slot A1) as shown in FIG. Is displayed). That is, the photo album creation application 80 displays an error dialog 1701 that the photos cannot be replaced because the size of the photo dragged and dropped by the user is too large or too small. The process of displaying the error dialog 1701 is sometimes called error display control. When the user selects the “OK” button in the displayed error dialog 1701, the error dialog 1701 is closed and the photo a1 is still set in the photo slot A1. Also, if the aspect ratio that can be set in the slot is different from the aspect ratio of the photo that has been replaced by the user, or if the image format that can be set in the slot is different from the format of the photo that has been replaced by the user, etc. A dialog is displayed.

  FIG. 18 is a flowchart showing the process of replacing photos with respect to the photos automatically laid out in the photo album. The photo album creation application 80 acquires the type of photo album selected by the user (S101). Here, the type of photo album is, for example, the size of the photo album, the number of pages, the binding method, the type of paper, and the like. Next, the photo album creation application 80 acquires the photo used by the album selected by the user (S102). The photo album creation application 80 acquires the photos stored in the picture folder selected by the user when the user stores the photos in the picture folder.

  When receiving a request for creating a photo album, the photo album creating application 80 starts creating a photo album (S103). When the creation of the photo album is started, the photo album creation application 80 analyzes the photo selected in step S102 by the automatic layout processing unit 900 and automatically lays out the photo album. That is, the processing described with reference to FIG. When the layout is completed, the photo album creation application 80 displays an editing screen (S104).

  On this editing screen, the user performs editing operations such as replacement of photos. When the user selects a photo slot to replace a photo set in a photo slot, the photo album creation application 80 displays a “Change Photo” button 1301 in response to the selection (S105). ). Furthermore, when the user selects the “Change Photo” button 1301, the photo album creation application 80 displays a recommended photo dialog 1305 (S106).

  If there is no photo in the recommended photo dialog 1305 to be replaced with the photo in the selected photo slot, the user selects an “Other Photo” button 1506. In response to the selection, the photo album creation application 80 displays a list of photos selected by the user in step S102 (S107). Note that the image displayed as S107 excludes the image data corresponding to condition 1 and the image data corresponding to condition 2 from the image data group selected in FIG. Corresponding image data. This is because the image data corresponding to the condition 2 does not include information such as Exif and is not suitable for automatic layout, but can be laid out when the user manually arranges it. On the other hand, the image data corresponding to the condition 1 is not displayed even in S107 because it cannot be laid out even if the user manually places it (that is, it is not suitable for an album). By executing the above processing, as a result, image data corresponding to condition 1 that is not suitable for an album is excluded from the options. For example, the user erroneously selects image data corresponding to condition 1 that is not suitable for an album. Can be prevented. In addition, it is possible to improve the operability of the selection operation of the image data by the user while performing efficient automatic layout processing.

  The user examines whether there is a photograph to be replaced with the photograph in the selected photograph slot in the list of photographs displayed in step S107. If there is no photograph to be replaced, the user selects the “Add Photo” button 1508. To do. The photo album creation application 80 displays the file explorer 1501 according to the selection (S108). Then, when a photo in the file explorer 1501 is selected by the user as a photo to be replaced, the photo album creation application 80 executes the processing of FIG. 19 described later, and sets the selected photo as a target slot. (S109). That is, when the photo album creation application 80 determines that setting is possible in the process of FIG. 19 described later, the photo album creation application 80 replaces the photo in the target slot with the photo selected by the user.

  FIG. 19 is a flowchart showing the flow of internal processing of the photo album creation application 80 when the photos are exchanged using the file explorer 1501. When the user selects a photo in the file explorer 1501 by the user, the photo album creation application 80 determines whether the selected photo is suitable for laying out in the photo album (in response to the selection). S201). The photo album creation application 80 displays an error dialog 1701 in the display area 401 without replacing with the selected photo, for example, when the resolution is too low or the aspect ratio does not match the photo slot (for example). S202). It is also possible to prepare a continue button in the error dialog. When this continue button is pressed, the photo album creation application 80 may replace the photo in the target photo slot with the photo determined No in S201.

  On the other hand, when the photo album creation application 80 determines that the photo selected in step S201 is suitable for layout in the photo album, the photo album creation application 80 determines whether the selected photo is stored in the external memory. (S203). Here, the external memory is, for example, a USB memory or the like. However, the external memory can be disconnected at a timing desired by the user. Therefore, if the photo album creation application 80 determines that the selected photo is stored in the external memory (Yes in S203), the photo album creation application 80 stores the photo on a PC or the like so that there is no problem even if the connection is canceled during album editing. Copy to the local environment (S204). By copying (duplicating) in this way, even if the connection of the external memory is canceled during the creation of the photo album, the creation or editing of the album is not affected. The photograph stored in the external memory is not necessarily limited to the photograph stored in the USB memory, and is different from a terminal that executes this processing, such as a photograph uploaded on a website such as SNS. Photos stored in the terminal can also be the target of this processing. After that, if the selected photo is not a photo stored in the external memory, or if it is a photo stored in the external memory, and the photo is copied to the local environment, it is stored in the target photo slot. A certain photo is replaced with the selected photo (S205).

  FIG. 20 shows a case in which a photo taken with a camera built in or connected to a PC, a smartphone, a tablet, or the like is set as a target, not a photo already existing on a PC, a memory card, or the like. This is just an example. When the user presses the “Camera” button 2001 in the display area 401, the photo album creation application 80 displays a user interface 2002 for photographing the camera. When the user presses the “shoot” button 2003, the image displayed on the user interface 2002 is taken as a photograph.

  The photo album creation application 80 presents the photographed photo to the user by enlarging it or the like, and when the user selects the photo, the photo album creation application 80 sets the photo in the photo slot. FIG. 20A shows a user interface when a “Camera” button 2001 is selected after a vertically long photo slot is selected. That is, a vertically long area is displayed as the shooting area. On the other hand, when the “Camera” button 2001 is selected after the horizontally long photo slot is selected, a horizontally long region is displayed as the shooting region. In addition, a message may be displayed so that the camera is arranged vertically or horizontally according to the shape of the slot. On the other hand, FIG. 20B shows a user interface when the “Camera” button 2001 is selected after the square photo slot is selected. That is, FIG. 20 (a) and FIG. 20 (b) have different shooting modes, FIG. 20 (a) is a normal portrait or landscape shooting mode, and FIG. 20 (b) is a square shooting mode. As described above, the photo album creation application 80 sets the shooting mode of the camera according to the shape of the target photo slot. As a specific example, the photo album creation application 80 can set the shooting mode of the camera by instructing the shooting mode to the camera application that controls the camera.

  In FIG. 21, an image to be set in a photo slot is not a photo that already exists on a PC or a memory card, but an image is read by an image reading device such as a scanner connected to a PC, a smartphone, or a tablet. The case where it does is illustrated. When the user selects the “Scan” button 2101, the photo album creation application 80 displays a user interface 2102 for reading an image in the image reading apparatus. In the user interface 2102, a “Preview” button 2103, a “Scan” button 2104, and a preview image display area 2105 are set. When the “Preview” button 2103 is selected by the user, the photo album creation application 80 displays an image obtained by causing the image reading apparatus to execute a prescan process as a preview image in the preview image display area 2105.

  When the user sets a scan target area 2106 in the preview image display area 2105 and selects a “Scan” button 2104, the photo album creation application 80 executes a scan process on the scan target area 2106. FIG. 21A shows a user interface when a “Scan” button 2101 is selected after a vertically long photo slot is selected as a scan target. FIG. 21B shows a user interface when the “Scan” button 2101 is selected after a square photo slot is selected as a scan target by the user. That is, in FIG. 21A and FIG. 21B, the rectangle of the scan target area 2106 is different. In FIG. 21A, the area to be actually scanned is a vertically long area, and in FIG. 21B, it is a square area. In this way, the photo album creation application 80 adjusts the scan target area to an optimum one according to the shape of the target photo slot.

  In FIG. 21, the reading setting is changed according to the shape of the photo slot. However, the change of the reading setting is not necessarily limited to the shape. For example, the resolution suitable for the photo slot and the effect function set for the photo slot are used. The reading setting may be changed according to the above. Therefore, for example, in the case of a large-sized photo slot, it is better to set a high-resolution photo to maintain print quality. You may make it read.

  Some photo slots have effects applied. For example, when a photo is set in a photo slot to which a gray effect is applied, a color photo is also applied to the photo slot with the gray effect applied. When setting an image for a photo slot having such an effect function, the image may be read after the effect function is set by the image reading apparatus. Therefore, for example, in a photo slot where a gray effect is applied, a photo may be read in the gray mode.

  In addition, in the present embodiment, a photograph (that is, a still image) has been described as an example of an image. However, for example, a frame image included in a moving image captured by a moving image can be used. First, the photo album creation application 80 extracts consecutive frames from the moving image data, identifies the difference in feature amount between the preceding and succeeding frames, and determines the amount of motion.

  For example, the photo album creation application 80 divides each extracted frame area into a plurality of areas, and creates a histogram relating to RGB values for the divided areas. The photo album creation application 80 calculates a histogram difference regarding the RGB values between the preceding and succeeding frames for the plurality of divided regions, and the region corresponding to the histogram for which it is determined that the difference is greater than a predetermined threshold value. Calculate the number.

  Then, the photo album creation application 80 determines that the difference in the feature amount between the previous and subsequent frames is large when the number of regions corresponding to the histogram in which the feature amount difference is determined to be greater than the predetermined threshold value is greater than or equal to the threshold value. To do. Further, the photo album creation application 80 determines that the difference in the feature amount between the previous and subsequent frames is small when the number of regions corresponding to the histogram in which the feature amount difference is determined to be greater than the predetermined threshold value is less than the threshold value. To do.

  When it is determined that the difference in feature amount between the previous and next frames is large, the frames are considered not to be similar, and the photo album creation application 80 has a frame (motion Frame with a large amount). That is, the photo album creation application 80 determines that the moving image data is obtained by shooting a shooting target with a large amount of exercise. Also, when it is determined that the difference in feature amount between the previous and subsequent frames is small, the frames are considered to be similar, and the photo album creation application 80 determines that these frames are frames with less subject movement. It is determined that the frame has a small amount of motion. That is, the photo album creation application 80 determines that the moving image data is obtained by shooting a shooting target with a small amount of exercise.

  Regarding the determination of the motion amount, a motion vector used in the moving image format can be used in addition to using the feature amount difference between frames. In the above example, the specification for calculating the difference between the RGB values as the difference between the feature amounts between the frames is used. However, for example, the specification for calculating the difference regarding the luminance value may be used. In this way, a plurality of frames extracted from one moving image data are classified and managed as a frame group into a frame group with a large amount of motion and a frame group with a small amount of motion according to the amount of motion in chronological order. .

  Next, the photo album creation application 80 extracts frames at predetermined intervals from a group of frames extracted from the moving image data and managed in chronological order, and sets the frames as analysis targets. It should be noted that the frame interval extracted as the analysis target is different between the frame group determined to have a large amount of motion and the frame group determined to have a small amount of motion. For example, frames to be analyzed are extracted at intervals of 5 frames from a frame group determined to have a large amount of motion, and frames to be analyzed are extracted at intervals of 3 frames from a group of frames determined to have a small amount of motion.

  When the photo album creation application 80 extracts frames to be analyzed at a predetermined interval, the photo album creation application 80 further executes analysis processing on the extracted frames. Specifically, the photo album creation application 80 executes processing such as object detection, size specification, smile determination, eye meditation determination, blur blur determination, and brightness determination. Here, the object detection is processing for detecting an object such as a human face, an animal, or a dish from a frame to be analyzed. The size specification is processing for specifying the size of the object detected by the object detection. The smile determination is a process for determining whether or not the detected human face is a smile when the object detected by the object detection is a human face. The eye meditation determination is a process for determining whether or not the detected person's face is meditating when the object detected by the object detection is a person's face. The blur blur determination is a process for determining the blur or blur of the analyzed frame. Brightness determination is processing for determining the degree of brightness (average luminance value) of the analyzed frame.

  Finally, when executing the analysis process, the photo album creation application 80 performs a scoring process for adding a score to the analyzed frame according to the result of the analysis process. Specifically, for example, a scoring process is performed so that a frame including objects such as a person's face, an animal, and a dish has a higher score than a frame not including these objects. Further, for example, the scoring process may be executed so that a frame including a large object is given a higher score than a frame including a small object. The scoring method is not limited to this. Therefore, for example, the scoring process may be executed so that a frame with a large amount of motion is given a higher score than a frame with a small amount of motion. The photo album creation application 80 can specify a better frame (image) from a plurality of frames constituting the moving image data by these analysis processes. And the frame (image) specified from this moving image data can also be used in this embodiment.

  In the above-described embodiment, the case where the present invention is applied to a photo album has been described. However, the present invention is not necessarily limited to this. Therefore, the present invention can be similarly applied to, for example, photo contents such as a photo book and a calendar in addition to the photo album.

  In addition, the above-described embodiment can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (CPU, MPU, etc.) of the system or apparatus reads and executes the program. It is processing. The program may be executed by one computer or may be executed in conjunction with a plurality of computers. Further, it is not necessary to implement all of the above-described processing by software, and part or all of the processing may be realized by hardware such as an ASIC. Further, the CPU is not limited to the one that performs all the processing by one CPU, and a plurality of CPUs may perform the processing while appropriately cooperating.

Claims (20)

An information processing apparatus for creating photo content,
An exclusion means for excluding images that do not fit into the image slot from a group of images pre-selected by the user;
Automatic layout means for executing automatic layout processing for automatically laying out images remaining without being excluded by the exclusion means in corresponding image slots of the photo content;
Image acquisition means for acquiring a new image to be replaced with an image selected as a replacement target from the image laid out by the automatic layout means;
When the new image is acquired from an image that has not been excluded by the exclusion unit after the automatic layout process is executed, it is determined whether or not the image slot of the image selected as the replacement target matches. If the new image is acquired from an image other than the image previously selected by the user after the automatic layout process is executed, the image slot of the image selected as the replacement target is adapted. Image determining means for determining whether or not to perform;
An information processing apparatus comprising:   2. The new image acquired by the image acquisition unit is set in an image slot of an image selected as the replacement target on condition that the image determination unit determines that the image is suitable. The information processing apparatus described in 1. A display control means for displaying on the display unit an image other than at least the image preselected by the user;
The information processing apparatus according to claim 1, wherein the user selects the new image from the displayed image. When the new image is acquired from the image stored in the external storage device connected by various interfaces by the image acquisition means,
The storage control means for controlling to copy the image acquired by the image acquisition means and store it in the internal storage means on condition that the image determination means determines that the image is suitable. The information processing apparatus according to any one of 1 to 3. When the new image is acquired from the image uploaded on the website by the image acquisition means,
The storage control means for controlling to copy the image acquired by the image acquisition means and store it in the internal storage means on condition that the image determination means determines that the image is suitable. The information processing apparatus according to any one of 1 to 4. When the new image is acquired from the image read by the image reading device connected by various interfaces by the image acquisition means,
The storage control means for controlling to copy the image acquired by the image acquisition means and store it in the internal storage means on condition that the image determination means determines that the image is suitable. The information processing apparatus according to any one of 1 to 5.   The information processing apparatus according to claim 6, further comprising a control unit that controls to adjust a reading setting in the image reading apparatus according to a shape of an image slot of the image selected as the replacement target.   The control unit according to claim 6, further comprising: a control unit configured to perform control so as to adjust a reading setting in the image reading device in accordance with an effect function set in an image slot of the image selected as the replacement target. Information processing device. When the new image is acquired from an image captured by an image capturing device connected by various interfaces by the image acquisition unit,
The storage control means for controlling to copy the image acquired by the image acquisition means and store it in the internal storage means on condition that the image determination means determines that the image is suitable. The information processing apparatus according to any one of 1 to 8.   The information processing apparatus according to claim 9, further comprising a control unit that controls to adjust a shooting mode in the imaging apparatus according to a shape of an image slot of an image selected as the replacement target.   11. The information according to claim 1, further comprising an error display control unit configured to display an error dialog on a display unit on a condition that the image determination unit determines that the image does not match. 11. Processing equipment.   The image determination unit determines that the image is not suitable when the size of the image acquired by the image acquisition unit is larger or smaller than a size that can be set in an image slot of the image selected as the replacement target. The information processing apparatus according to claim 1, wherein:   The image determination means determines that the image acquired by the image acquisition means is not suitable when the aspect ratio of the image selected as the replacement target is not an aspect ratio that can be set in the image slot. The information processing apparatus according to any one of claims 1 to 12.   The image determining unit determines that the image format of the image acquired by the image acquiring unit is not compatible when the image format is not settable in the image slot of the image selected as the replacement target. The information processing apparatus according to any one of claims 1 to 13.   The automatic layout means automatically lays out a predetermined frame extracted from the moving image in a corresponding image slot of the photo content when the image selected in advance by the user is a moving image. The information processing apparatus according to any one of claims 1 to 14, wherein the information processing apparatus is characterized in that:   The information processing apparatus according to claim 3, wherein the display control unit emphasizes and displays the temporarily selected image on the display unit in order to assist selection of the new image.   The display control means enlarges the temporarily selected image to be displayed on the display unit, masks a display area other than the temporarily selected image in the display area of the display control means, and displays it on the display unit. The information processing apparatus according to claim 16, wherein the information processing apparatus is displayed.   The information processing apparatus according to claim 17, wherein the display control unit changes display contents in conjunction with the temporarily selected image in a display area other than the temporarily selected image. An information processing method in an information processing apparatus for creating photo content,
An automatic layout step of automatically laying out an image preselected by the user in an image slot corresponding to the photo content by automatic layout means;
An image acquisition step of acquiring an image to be replaced with an image selected as a replacement target from the images laid out in the automatic layout step by the image acquisition means, wherein the image to be replaced is other than the image selected in advance by the user; An image acquisition step, which is also acquired from an image;
An image determination step for determining whether or not the image acquired in the image acquisition step matches the image slot of the image selected as the replacement target by the image determination means;
An information processing method in an information processing apparatus comprising:   The program for functioning a computer as each means of the information processing apparatus of any one of Claims 1-18.

Images