JP6742486B2 - Program, image processing method - Google Patents

Description

The present invention relates to image display technology.

Some conventional image processing apparatuses display an image related to image data on a display screen of an LCD (Liquid Cristal Display) that constitutes a user interface, and edit the image by designating the displayed image. In editing a photo album, an image slot is arranged on each page of the photo album, and a desired layout image can be obtained by assigning a photo to the image slot. Among applications for creating a photo album, there is an application in which a user first designates a photo image and then automatically assigns a photo image in accordance with a template of the photo album. As described above, in the case of automatically assigning a photographic image instead of manually, the image desired by the user may not be assigned, so that the images can be replaced. In Patent Document 1, a plurality of images specified by a user for creating a photo album are divided into groups, the groups are associated with the pages of the photo album, and when the images in the image slots are replaced, the pages are associated with the pages. Display candidate images from the group.

Patent No. 5449460

In Patent Document 1, when the number of images designated by the user is large, many candidate images are displayed. Further, since the image assigned to each page of the photo album is displayed as a candidate image, there is a problem that it is difficult to select an image because the characteristics of the image slots to be replaced are not considered. For example, even if the image slot to be replaced is a slot that is supposed to be a grayscale image, a color image will be displayed as a candidate image. Also, a horizontally long slot may be displayed with respect to a vertically long slot.

The present invention has been made in view of such a problem, and provides a technique for displaying an image suitable as a candidate for another image when replacing an arbitrary image with another image.

According to one embodiment of the present invention, a computer
Acquisition means for acquiring an image data group including a plurality of image data,
Dividing means for dividing the plurality of image data into a plurality of groups,
A layout display control means for displaying a layout image in which an image based on image data included in a predetermined group of the plurality of groups is displayed in a slot of a page corresponding to the predetermined group ,
First display control means for displaying a first candidate screen including, as a candidate image , an image including a shooting date and time within a predetermined time range including a shooting date and time of the first image arranged in the layout image ,
A second candidate screen including, as a candidate image, an image including a shooting date and time in at least one of a time range before the predetermined time range and a time range after the predetermined time range. Second display control means for displaying
Changing means for changing from the first candidate screen to the second candidate screen,
Arrangement means for arranging a candidate image selected by the user from the candidate images
It is a program to function as
The predetermined time range and at least one of the time ranges are included in the shooting period of the predetermined group.

According to the configuration of the present invention, when an arbitrary image is replaced with another image, an image suitable as a candidate for the other image can be displayed.

The block diagram which shows the structural example of a system. The block diagram which shows the hardware structural example of PC1. The figure which shows the structural example of a software module group. The figure which shows the example of a display of the window 401. The flowchart of the process which PC1 performs. The flowchart of the process for displaying a list of image candidates. The figure which shows an example of the image of a two-page spread, and the window containing a candidate image. The figure which shows the example of a transition of the window containing a candidate image. FIG. 11 is a block diagram of a functional unit that executes the process of FIG. 10 in the application 80. 9 is a flowchart of processing performed by executing the automatic layout processing unit 916. The figure which shows an example of image analysis information. The figure which shows an example of the grouping of a captured image. The figure which shows an example of scene classification reference data. The figure for demonstrating the process of step S1010. The figure explaining the process of step S1017. The figure which shows a template group. The figure explaining the selection method of a candidate picked-up image. The figure explaining the selection method of a candidate picked-up image.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. In addition, the embodiment described below shows an example of the case where the present invention is concretely implemented, and is one of the concrete examples of the configurations described in the claims.

[First Embodiment]
In the present embodiment, an example of the image processing device will be described. This image processing apparatus acquires an image data group including a plurality of image data, and displays a layout image in which an image based on the image data selected from the acquired image data group is arranged in a slot on a display screen (first Display control). A candidate for replacing the first image based on the shooting date and time of the first image arranged in the selected slot to be replaced in the layout image displayed on the display screen. The image is displayed on the display screen (second display control).

First, a configuration example of the system according to the present embodiment will be described with reference to the block diagram of FIG. As shown in FIG. 1, the system according to the present embodiment includes a PC (personal computer) 1 operated by a user who creates a photo album, and a PC 2 which is a server of ABC Company that provides a photo album ordering site. doing. The PC1 and the PC2 are connected to each other via the Internet so as to be able to perform data communication with each other.

First, the PC 1 will be described. The application 80 and the Web browser 143 are installed in the PC 1.

The application 80 is an application for providing various functions for creating a photo album and uploading it to the PC 2.

The web browser 143 is a browser used when using WWW (World Wide Web). The Web browser 143 provides various functions for accessing the PC2, uploading the photo album created on the PC1 side to the PC2, and enabling the ordering of the photo album via the order site provided by the PC2. It is an application.

The PC 1 is connected to a local area network 4 such as Ethernet (registered trademark) (for example, a home network for general homes built in the user's home), and the local area network 4 is connected to the Internet. ing.

Next, the PC 2 will be described. A Web server 9 which is an application of a server function is installed in the PC 2, and the Web server 9 further includes an application of a shopping site 11 which is a photo album ordering site.

The PC 2 is connected to a local area network 8 such as Ethernet (registered trademark) (for example, an office network built in the company of ABC Company), and the local area network 8 is connected to the Internet.

An OS (operating system) is installed in both PC1 and PC2, and the application operates on this OS in both PC1 and PC2.

Next, an example of the hardware configuration of the PC 1 will be described with reference to the block diagram of FIG. The hardware configuration of the PC 2 may be the same as the hardware configuration shown in FIG. 2, or a hardware configuration different from the hardware configuration shown in FIG. 2 may be adopted.

The RAM 201 has an area for storing computer programs and data loaded from an HDD (hard disk drive) 202 and computer programs and data received from the outside (for example, the PC 2) via the communication control unit 207. Further, the RAM 201 has a work area used when the CPU 204 executes various processes. In this way, the RAM 201 can provide various areas as needed.

The HDD 202 is an example of a large-capacity information storage device, and the HDD 202 stores an OS and computer programs and data for causing the CPU 204 to execute each process described below as being performed by the PC 1. The computer program also includes the application 80 and the Web browser 143 described above. Further, this data also includes what is described as known information in the processing described later. The computer programs and data stored in the HDD 202 are appropriately loaded into the RAM 201 under the control of the CPU 204, and are processed by the CPU 204.

The operation unit 203 is configured by an interface such as a keyboard and a mouse, and can be operated by a user to input various operation inputs to the CPU 204.

The CPU 204 executes a process using a computer program or data stored in the RAM 201, thereby controlling the operation of the entire PC 1 and executing or controlling each process described below as being executed by the PC 1.

The communication control unit 207 is for performing data communication with the PC 2 via the local area network 4 and the Internet.

The display unit 205 includes a CRT, a liquid crystal screen, and the like, and can display the processing result by the CPU 204 as an image or a character. The display unit 205 and the operation unit 203 may be integrated into a touch panel screen.

The RAM 201, the HDD 202, the operation unit 203, the CPU 204, the communication control unit 207, and the display unit 205 are all connected to the bus 206.

Next, a configuration example of the software module group stored in the HDD 202 of the PC 1 will be described with reference to FIG. Note that the software modules shown in FIG. 3 represent a part of the software module group stored in the HDD 202, but not all. The software module group stored in the HDD 202 is appropriately changed according to the OS installed in the PC 1, the usage environment of the PC 1, and the like.

The module 92 is an Ethernet control stack that controls Ethernet. The module 91 is an IP Network control stack that controls the IP Network. The module 90 is a WSD control stack that controls a WSD (Web Service on Devices) that provides a device search mechanism on the network. Module 88 is a PnP-X control stack that controls the plug and play of the network. PnP-X is an abbreviation for Plug and Play Extensions, which is a function that is standardly installed in Windows 8 (registered trademark) as a series of extended functions of Plug and Play that provides support for network-connected devices. The module 85 is a device driver group, and includes a standard driver group 87 that is standardly packaged with the OS, and an IHV-made driver group 86 that is provided by the Independent Hardware Vendor (IHV).

The module 84 is an application/DDI interface and is configured to include an Application Programming Interface (API) and a Device Driver Interface (DDI). The application group 82 is an application group including the application 80 and the Web browser 143 described above.

Next, a process of activating the application 80 on the PC 1 to arrange a desired captured image in a prescribed area on each page of the photo album to create a desired photo album will be described. When the application 80 is started, the window 401 illustrated in FIG. 4 is displayed on the display screen of the display unit 205. Hereinafter, unless otherwise specified, display control of various GUIs including the window 401, processing according to an operation input performed by the user operating the operation unit 203 using the GUI, etc. Both are assumed to be the CPU 204.

In the display area 402, a thumbnail image of a double-page spread in the photo album is displayed. In the display area 402, from the left, thumbnail images of double-page spreads on the first and second pages, thumbnail images of double-page spreads on the third and fourth pages,..., Double-page spreads on the ninth and tenth pages. A thumbnail image of the page is displayed. Further, in FIG. 4, the user operates the operation unit 203 to specify the thumbnail images of the double-page spreads of the third page and the fourth page. In this case, the thumbnail image is in the selected state (the frame 404 is superimposed on the thumbnail image and displayed), and the display area 403 displays “third page and fourth page” corresponding to the thumbnail image. The "page spread image" is displayed. Areas (photograph slots) 405a to 405d in which the captured images can be arranged are provided in the "images of double-page spreads by the third page and the fourth page", and these areas 405a to 405d perform processing described later. The captured image is resized to the size of the area and arranged by.

Here, when the user operates the operation unit 203 to specify any one of the regions 405a to 405d in a state where the captured image is arranged in each of the regions 405a to 405d, the image is displayed in the one region. Instead of the captured image, a list of captured image candidates that can be arranged in the one area is displayed. Therefore, the user operates the operation unit 203 to designate a desired candidate image among the candidate images displayed in the list, thereby arranging the designated candidate image in the one area (replacement of images). You can

When the user operates the operation unit 203 to specify the button 406, the CPU 204 controls the communication control unit 207 and uploads the created photo album data to the PC 2.

Next, regarding the processing for displaying a list of image candidates that can be arranged in the area designated by the user among the areas 405a to 405d in the state where the captured images are arranged in the areas 405a to 405d, respectively, of FIG. It will be described according to the flowchart.

<Step S101>
The user operates the operation unit 203 to specify any one of the areas 405a to 405d (replacement target slot) to display a list of image candidates that can be arranged in the area on the display unit 205. You can Therefore, the CPU 204 receives a selection operation of any one of the areas 405a to 405d.

<Step S102>
The CPU 204 sets the captured image being displayed in the replacement target slot as a starting point (a candidate starting image) for searching the HDD 202 for a candidate of a captured image that can be arranged in the replacement target slot instead of the captured image.

<Step S103>
The CPU 204 searches the HDD 202 for a candidate (candidate image) of a captured image that can be arranged in the replacement target slot instead of the candidate origin image, based on the candidate origin image (lists). Details of the processing in this step will be described later.

<Step S104>
The CPU 204 causes the display unit 205 to display a window including the candidate images listed in step S103. This window will be described with reference to FIG. In FIG. 7A, "images of double-page spreads by the first and second pages" are shown, and 500 to 502 are photo slots. FIG. 7B shows “images of double-page spreads on the third and fourth pages”, and 503 to 505 are photo slots. In FIG. 7B, the photo slots 504 and 505 are overlapped on the photo slot 503, but there may be a case where the photo slots are further arranged on the photo slot in this way. In FIG. 7C, "images of double-page spread pages by the fifth and sixth pages" are shown, and 506 to 507 are photo slots.

Here, in the state where the “image of the double-page spread by the third and fourth pages” in FIG. 7B is displayed in the display area 403, the captured image displayed in the photo slot 503 is captured by another image. It is assumed that the user operates the operation unit 203 to specify the photo slot 503 in order to display the candidate image so as to replace it with an image. At this time, in step S104, the window illustrated in FIG. 7D is displayed on the display unit 205.

In this window, a list of candidate picked-up images 641 to 644 (picked-up images listed based on the picked-up image displayed in the photo slot 503 in step S103) that are candidates for picked-up images that can be arranged in the photo slot 503 are listed. It is displayed. The user can specify a desired one of the candidate captured images 641 to 644 to be arranged in the photo slot 503 by operating the operation unit 203. Here, it is assumed that the user operates the operation unit 203 to specify a desired one of the candidate captured images 641 to 644 to be placed in the photo slot 503. At this time, the CPU 204 resizes the designated candidate picked-up image to the size of the photo slot 503 instead of the picked-up image being displayed in the photo slot 503, and then arranges (displays) it in the photo slot 503.

Further, the previous button 510 is for displaying a candidate image based on the captured image arranged in the photo slot one before the currently selected photo slot in the photo slot order ordered through all the pages of the photo album. It is a button. In the case of FIG. 7D, when the user operates the operation unit 203 to specify the front button 510, the CPU 204 causes the CPU 204 to select a candidate image based on the captured images arranged in the photo slot one order before the photo slot 503. Is displayed.

Further, the rear button 511 is for displaying a candidate image based on the imaged image placed in the photo slot that is one behind the currently selected photo slot in the photo slot order that is ordered through all the pages of the photo album. It is a button. In the case of FIG. 7D, when the user operates the operation unit 203 to specify the back button 511, the CPU 204 causes the CPU 204 to select a candidate image based on the picked-up images arranged in the photo slot one order after the photo slot 503. Is displayed.

When the front button 510 or the rear button 511 is designated, the CPU 204 sets the captured image being displayed in the new photo slot as a candidate starting point image, and lists new candidate captured images based on the candidate starting point image. Then, the CPU 204 displays a list of the listed picked-up candidate images in the window. FIG. 8 shows a transition example of the candidate captured images in the window when the front button 510 and the rear button 511 are specified.

The window of FIG. 8A is displayed when the photo slot 500 is selected in “Image of spread page by first page and second page” (FIG. 7A) (referred to as image 1-2). This is the window that will be opened. In this window, candidate picked-up images 600 to 603 listed based on picked-up images being displayed in the photo slot 500 are displayed in a list. When the back button 511 is specified in this window, as shown in FIG. 8B, a window displaying a list of candidate captured images 610 to 613 listed based on the captured images displayed in the photo slot 501 is displayed. It If the front button 510 is specified in this window, the window changes to the window shown in FIG. On the other hand, if the rear button 511 is specified in the window of FIG. 8B, candidate captured images 620 to 623 listed based on the captured images displayed in the photo slot 502 are listed as shown in FIG. 8C. The displayed window is displayed. If the front button 510 is specified in this window, the window changes to the window shown in FIG. On the other hand, assume that the back button 511 is specified in the window of FIG. In this case, as shown in FIG. 8D, the list based on the imaged image displayed in the photo slot 503 of “Image of double-page spread by page 3 and page 4” (referred to as image 3-4). A window including the uploaded candidate captured images 630 to 633 is displayed. If the front button 510 is specified in this window, the window changes to the window shown in FIG. On the other hand, if the rear button 511 is specified in the window of FIG. 8D, as shown in FIG. 8E, the candidate image pickups listed based on the imaged image displayed in the photo slot 504 in the image 3-4 are displayed. A window listing the images 640 to 643 is displayed. If the front button 510 is specified in this window, the window changes to that shown in FIG. On the other hand, if the rear button 511 is specified in the window of FIG. 8E, as shown in FIG. 8F, the candidate image pickups listed based on the imaged image displayed in the photo slot 505 in the image 3-4 are displayed. A window listing the images 650-653 is displayed. If the front button 510 is specified in this window, the window changes to the window shown in FIG. On the other hand, assume that the back button 511 is specified in the window of FIG. In this case, as shown in FIG. 8G, the list is based on the imaged images displayed in the photo slot 506 of “Images of double-page spreads on page 5 and page 6” (referred to as images 5-6). A window including the uploaded candidate captured images 660 to 663 is displayed. If the front button 510 is specified in this window, the window changes to the window shown in FIG. On the other hand, if the rear button 511 is specified in the window of FIG. 8G, the candidate imaged listed based on the imaged image displayed in the photo slot 507 in the image 5-6 is displayed as shown in FIG. 8H. A window listing the images 670-673 is displayed. If the front button 510 is specified in this window, the window changes to the window shown in FIG.

As described above, in the present embodiment, the user operates the operation unit 203 to specify the front button 510 and the rear button 511, thereby displaying a list of candidate captured images corresponding to the photo slots across the images of the double-page spread. Can be made.

Further, by designating the front button 510 and the rear button 511, as a candidate image to be replaced with the captured image being displayed in the photo slot selected in the display area 403, the image for the photo slot outside the page to which the photo slot belongs is selected. Candidate captured images can also be displayed. This makes it possible to arrange a captured image that is completely different from the captured image that can be arranged in the target photo slot, in the target photo slot.

In the above example, the order of photo slots ordered through all pages of the photo album is the photo slot at the top left of the page, the photo slot at the top right, the photo slot at the bottom left, and the photo slot at the bottom right. However, it is not limited to this order. Further, although the photo slots assigned to the entire page, like the photo slot 503, are in the first order of the page, this is also an example, and the present invention is not limited to this. For example, another order such as upper left, lower left, upper right, lower right may be adopted. Further, in the above example, the number of candidate captured images displayed in the window is four, but the number is not limited to this number.

Further, in the above example, the candidate image based on the captured image placed in the previous or next photo slot by the front button 510 or the rear button 511 is displayed, but it is displayed in another photo slot on the window. There are various methods for forming a candidate image based on the arranged captured image, and the method is not limited to a specific method. For example, it is possible to provide an input field for designating how much before or after the photo slot is selected and input a numerical value there (for example, when selecting the photo slot two before, "-2 "Enter "2" to select the next photo slot."

Returning to FIG. 7, the button 520 is a button for displaying a captured image other than the candidate captured image displayed in the window. When the user operates the operation unit 203 to specify this button 520, the CPU 204 causes the display unit 205 to display a list of images initially specified by the user when creating a photo album.

<Step S105>
The CPU 204 determines whether or not the previous button 510 (previous candidate button) is designated. If the result of this determination is that it has been designated, the processing proceeds to step S106, and if it has not been designated, the processing proceeds to step S107.

<Step S106>
The CPU 204, in the “order of photo slots ordered through all pages of the photo album”, sets the captured image displayed in the previous photo slot of the photo slot that is the current candidate start image to the new candidate start image. Set to. Then, the process returns to step S103.

<Step S107>
The CPU 204 determines whether or not the rear button 511 (rear candidate button) is designated. If the result of this determination is that it has been designated, the processing proceeds to step S108, and if it has not been designated, the processing proceeds to step S109.

<Step S108>
The CPU 204, in the “order of photo slots ordered through all pages of the photo album”, sets the captured image displayed in the photo slot that is one after the photo slot that is currently the candidate start image to the new candidate start image. Set to. Then, the process returns to step S103.

<Step S109>
The CPU 204 determines whether or not the user has operated the operation unit 203 to select one of the candidate captured images in the currently displayed window. If the result of this determination is that it has been selected, the process proceeds to step S110, and if it has not been selected, the process returns to step S104.

<Step S110>
The CPU 204 replaces the captured image being displayed in the replacement target slot with the candidate captured image selected by the user operating the operation unit 203 and displays the candidate captured image.

Next, a process will be described in which the CPU 204 executes the application 80 to generate the window 401 illustrated in FIG. 4 and display the window 401 on the display unit 205. The process of FIG. 5, that is, the process of displaying a list of picked-up image candidates to be displayed instead of the picked-up image currently displayed in the photo slot, and displaying the candidate selected by the user in the photo slot is described below. This is a process performed on the window generated by the process of 10.

First, in the application 80, a functional unit that executes processing according to the flowchart of FIG. 10 will be described with reference to the block diagram of FIG. 9. As shown in FIG. 9, the processing according to the flowchart of FIG. 10 is executed by the designation unit 901 and the automatic layout processing unit 916. Further, as shown in FIG. 9, the automatic layout processing unit 916 is composed of a plurality of functional units, and the functions of the automatic layout processing unit 916 are realized by the plurality of functional units. Note that in FIG. 9, the designating unit 901 is shown as a functional unit separate from the automatic layout processing unit 916, but the designating unit 901 may be included in the automatic layout processing unit 916.

Further, for the sake of explanation, the functional unit shown in FIG. 9 may be described as the main body of processing, but in reality, the function of the functional unit is realized by the CPU 204 executing a computer program corresponding to the functional unit. To be done.

Further, the functions of the respective functional units described later may be partly or wholly left to other functional units, or may be realized by performing a cooperative operation with other functional units. That is, as long as the function of the automatic layout processing unit 916 is realized by a plurality of function units, the operation of each function unit is not limited to a specific form.

The designation unit 901 acquires “various information constituting the photo album creation condition” and supplies the acquired information to the automatic layout processing unit 916. “Various information constituting the photo album creation condition” is, for example, information set by a user operating the operation unit 203 on a GUI (graphical user interface) displayed on the display unit 205. When the user inputs a completion instruction after setting these pieces of information on the GUI, the designation unit 901 acquires “various pieces of information constituting the photo album creation condition” set on the GUI. Of course, the designation unit 901 may acquire from the HDD 202 the “various information forming the photo album creation condition” that is created in advance on the GUI and stored in the HDD 202. The conditions to be set include, for example, designation of the target image or the hero ID, the number of double-page spreads of the album, template information, ON/OFF conditions for image correction, and the like. The image may be specified by accompanying information or attribute information of individual image data such as shooting date and time, or may be specified based on the structure of the file system including the image data, such as device or directory specification. Good. In the present embodiment, the “spread” corresponds to, for example, one display window in display, and in printing, corresponds to a pair of adjacent pages printed on different sheets.

The image acquisition unit 902 reads the captured image data group stored in the HDD 202.

The image conversion unit 903, for each captured image data acquired by the image acquisition unit 902, reduces the image data of the captured image data (for example, the short side has a size of 490 pixels, and each pixel has color information of sRGB). Have) is generated as an analysis image.

The image analysis unit 904 performs analysis processing such as feature amount extraction, face recognition, individual recognition, and facial expression recognition on each analysis image generated by the image conversion unit 903, and the captured image from which the analysis image is generated. The image capturing date and time of the captured image is acquired from the supplementary information (eg, Exif information) of

The image classification unit 905 divides the captured image group acquired by the image acquisition unit 902 into a plurality of scenes based on the result of the analysis processing by the image analysis unit 904 and the imaging date and time, and classifies the scenes. A scene is a shooting scene such as a trip, a daily life, or a wedding. It can be said that the scene is, for example, a collection of images taken at one shooting opportunity for one shooting target.

The input unit 906 acquires, from the information acquired by the designating unit 901, information identifying the hero (for example, identification information of the hero).

The image score unit 907 obtains a score for each captured image data acquired by the image acquisition unit 902.

The input unit 908 acquires the number of spreads of the photo album from the information acquired by the designating unit 901.

The spread allocation unit 909 allocates each group classified by the image classification unit 905 or each group rearranged based on each group to each spread of the photo album.

The image selection unit 910 arranges, for each spread, a photo slot of each of the captured images in the spread based on the score obtained by the image score unit 907 for each captured image in the group assigned to the spread. The captured image to be selected is selected.

The input unit 911 acquires, from the information acquired by the designating unit 901, a template group (a template having a different layout) representing a layout of two-page spreads (a layout of each photo slot in the two-page spread). The template here is an image of a two-page spread.

The image layout unit 912 selects one of the templates input by the input unit 911 based on the captured image selected by the image selection unit 910.

The input unit 913 acquires, from the information acquired by the designating unit 901, a flag value indicating whether to perform image correction on the captured image.

When the flag value acquired by the input unit 913 is a value indicating that image correction is to be performed, the image correction unit 914 performs dodge correction and red-eye correction on the captured image selected by the image selection unit 910. , Correction processing such as contrast correction is performed. Of course, the image correction unit 914 does not execute the correction process when the flag value acquired by the input unit 913 is a value indicating that the image correction is not performed.

The output unit 915 arranges, in each photo slot on the template selected by the image layout unit 912, the captured image assigned to the photo slot by resizing the image to the size of the photo slot and arranging the “page spread image”. To generate. Then, the output unit 915 outputs the generated “image of the two-page spread” as layout information.

In the above description, the image to be image-corrected or placed in the photo slot is the captured image, but it may be an analysis image generated from the captured image. Of course, it may be another image generated from the captured image.

Next, the processing performed by the CPU 204 executing the automatic layout processing unit 916 will be described with reference to the flowchart of FIG. 10.

<Step S1001>
The image conversion unit 903 sequentially focuses on the images included in the designated image group, and generates the above analysis image from the focused image. The analysis image is generated based on the “generation parameters such as the number of vertical and horizontal pixels of the analysis image and the color space of pixel values” included in the information acquired by the designation unit 901.

<Step S1002>
The image analysis unit 904 extracts the feature amount from the analysis image generated in step S1001 and acquires the image capture date and time of the captured image from the incidental information of the captured image from which the analysis image was generated. Although various features can be used as the feature amount to be extracted, the focus amount is extracted as the feature amount as an example below.

A known Sobel filter can be used as a method for detecting edges in an image. An edge (luminance gradient) can be calculated by detecting an edge from an image using a Sobel filter and dividing the brightness difference between the detected start point and end point of the detected edge by the distance between the start point and the end point. Then, the average value of the inclinations of the edges in the image is calculated. An image with a large average tilt value can be considered to be more in focus than an image with a small average tilt value. Therefore, by setting a plurality of different threshold values for each of the inclinations in advance and determining which threshold value is the average value of the inclinations obtained from the image or more, the focus amount in the image can be calculated. In the present embodiment, two different thresholds θ1 and θ2 (θ1<θ2) are set, and if the calculated average value A of the inclination is A<θ1, the focus amount=×, and if θ1≦A<θ2, the focus amount. =Δ, A≧θ2, it is determined that the focus amount=◯. The focus amount=○ indicates the inclination of the focus to be adopted in the photo album. The focus amount=Δ represents an allowable focus tilt. The focus amount=× represents an unacceptable inclination.

<Step S1003>
The image analysis unit 904 detects a human face from the analysis image generated in step S1001. A technique for detecting a face from an image is a known technique, and examples thereof include Adaboost that creates a strong discriminator from a plurality of prepared weak discriminators. In the present embodiment, the face of a person is detected from the image by the strong classifier created by Adaboost. By this face detection processing, the coordinate position and size of the human face area (face area (rectangular area)) on the image are acquired. Instead of the size, the coordinate position of the upper left corner and the coordinate position of the lower right corner of the face area may be acquired. The strong discriminator can be created by Adaboost not only for faces but also for animals such as dogs and cats, objects such as flowers, food, buildings, and figurines as detection targets instead of faces. Thereby, an object other than the face can be detected.

<Step S1004>
The image analysis unit 904 performs, for each face area detected in step S1003, individual recognition processing that is processing for recognizing who the face in the face area is. For example, regarding the images of the faces of various people registered in the HDD 202 in advance, the degree of similarity with the images in the face area detected in step S1003 is obtained. Then, among the images of the faces of various people that are registered in the HDD 202 in advance, an image for which the maximum degree of similarity is obtained that is greater than or equal to the threshold value is specified, and the identification information (( The identification information (personal ID) of an individual having a face in the image is acquired. If all the calculated similarities are less than the threshold value, new identification information is issued, and the image in the face area detected in step S1003 is registered in the HDD 202 in association with the issued identification information. ..

In this step, any personal recognition process may be adopted as long as the identification information of the person having the face in the face region detected in step S1003 can be acquired.

Then, the image analysis unit 904 registers the information collected in steps S1002 to S1004 in the HDD 202 as image analysis information. FIG. 11 shows an example of the image analysis information. The image analysis information includes, for each captured image, an ID (image ID) unique to the captured image, an image capture date and time of the captured image, a focus amount, the number of human faces detected from the captured image, and the captured image. The coordinate positions of the upper left corner and the lower right corner of the detected face area of each person are included. For example, the following information is registered for the captured image with the image ID=1.

Imaging date and time = "July 1, 2015 10:11:12"
Focus amount = "○"
Number of detected human faces = "6"
The coordinate position of the upper left corner and the coordinate position of the lower right corner of the face area of the person with personal ID=1 The coordinate position of the upper left corner and the coordinate position of the lower right corner of the face area of the person with personal ID=2 Person with the personal ID=3 Coordinate position of the upper left corner and the lower right corner of the face area of the person's face area The upper left corner of the person's face area of the individual ID=4 and the coordinate position of the lower right corner of the person's face area of the individual ID=5 The coordinate position of the corner and the coordinate position of the lower right corner The coordinate position of the upper left corner and the coordinate position of the lower right corner of the face area of the person with the individual ID=6 <Step S1005>
When all of the captured image groups acquired by the image acquisition unit 902 are selected in step S1001, the process proceeds to step S1006 via step S1005. On the other hand, if an unselected captured image remains in the captured image group acquired by the image acquisition unit 902, the process returns to step S1001.

<Step S1006>
The image classification unit 905 acquires the imaging date and time of each captured image acquired by the image acquisition unit 902 from the image analysis information registered in the HDD 202. Then, the image classification unit 905 divides the captured image group (captured image data group) acquired by the image acquisition unit 902 into a plurality of groups based on the captured date and time of each acquired captured image.

For example, when the captured images (captured image data) acquired by the image acquisition unit 902 are viewed in ascending order of captured date and time, the captured date and time of the Nth (N is an integer of 1 or more) captured image and (N+1) It is assumed that the difference between the image pickup date and time of the th imaged image is one day or more. At this time, a group boundary is set between the Nth captured image and the (N+1)th captured image. If the difference between the image pickup date and time of the Nth image pickup image and the image pickup date and time of the (N+1)th image pickup image in the order of image pickup date and time is 16 hours or more, the Nth image pickup image and the (N+1)th image pickup image are taken. Set the boundary of the group with the image. In addition, it is assumed that the difference in image pickup date and time between adjacent image pickup images in the order of the image pickup date and time is less than 16 hours regardless of which image is picked up. In this case, if the difference between the date and time of the first captured image and the date and time of the last captured image in the captured image group captured in one day is less than 4 hours, the captured image group of the one day is selected. There will be one group. On the other hand, if the difference between the image pickup date and time of the first image pickup image and the image pickup date and time of the last image pickup image in the image pickup image group taken in the one day is 4 hours or more, the image pickup in the one day is performed. If the number of images is less than 50 (the number of captured image data captured in one day is less than 50), the captured image group of the one day is set as one group. An example of such grouping of captured images is shown in FIG. In FIGS. 12A to 12C, the horizontal axis represents the image capturing date/time (older toward the left, newer toward the right), and the vertical axis represents the number of captured images. In FIG. 12A, the captured image group is divided into eight groups 1201 to 1208. In FIG. 12A, the arrows indicate the boundaries of groups.

<Step S1007>
The image classifying unit 905 selects one of the groups divided in step S1006 as the target group, and determines which scene is classified into the target group. In the present embodiment, it is determined whether the group is classified into a travel scene, a daily life scene, and a ceremony scene. Of course, the types of scenes are not limited to these types.

For this step, data (scene classification reference data) serving as a reference for determining to which scene the group is classified is registered in the HDD 202 in advance. An example of how to create the scene classification reference data is shown below.

For example, an image group that has been determined in advance as a travel image (travel image group), an image group that has been determined in advance as a daily image (daily image group), and an image group that has been determined in advance as a ceremony image (ceremony image) Groups), and a plurality of image groups for each group are prepared in advance. Then, for each of the travel image group, the daily image group, and the ceremony image group, the average and standard deviation of the image capturing period, the average and standard deviation of the number of captured images, and the average and standard deviation of the number of image capturing persons are obtained. For example, a case will be described in which the average and standard deviation of the imaging period, the average and standard deviation of the number of captured images, and the average and standard deviation of the number of imaging persons are obtained for the travel image group.

The image capturing period of an image group is the difference between the image capturing date and time of the image captured in the earliest and the image capturing date and time of the most recently captured image in the image group. Therefore, for each travel image group, the difference between the image capture date and time of the image captured in the earliest in the travel image group and the image capture date and time of the most recently captured image in the travel image group is obtained as the image capture period of the travel image group, and the travel image is captured. The average and standard deviation of the imaging period obtained for the image group are obtained.

Further, the number of captured images of the image group is the number of images forming the image group. Therefore, for each travel image group, the number of captured images forming the travel image group is obtained, and the average and standard deviation of the number of captured images obtained for each travel image group are obtained.

Further, the number of captured images of the image group is the number of faces shown in each image forming the image group. Therefore, for all the travel image groups, the number of faces shown in each of the captured images forming the travel image group is obtained by a well-known technique such as face recognition, and the average and standard deviation of the obtained number of faces are obtained. Ask for.

Similarly, for the daily image group and the ceremony image group, the average and standard deviation of the image capturing period, the average and standard deviation of the number of captured images, and the average and standard deviation of the number of image capturing persons are obtained. Then, the average and standard deviation of the imaging period, the average and standard deviation of the number of captured images, and the average and standard deviation of the number of captured images obtained for each of the travel image group, the daily image group, and the ceremony image group in this way It is registered in the HDD 202 in advance as classification reference data. An example of the scene classification reference data is shown in FIG.

The image classification unit 905 obtains the average of the image capturing period, the number of captured images, and the number of captured images for the target group in the same manner as above. Then, the image classification unit 905 refers to the scene classification reference data, and for the target group, scores for travel (scene classification 1), daily (scene classification 2), and ceremony (scene classification 3). Calculate as follows.

Score of imaging period of scene i=50−|10×(average of imaging period of scene i−imaging period of selected group)/standard deviation of imaging period of scene i|
Score of number of captured images of scene i=50−|10×(average number of captured images of scene i−number of captured images of selected group)/standard deviation of number of captured images of scene i|
Score of the number of imagers in the scene i=50−|10×(average number of imagers in the scene i−average number of imagers in the selected group)/standard deviation of the number of imagers in the scene i|
Score of scene i = (score of image capturing period of scene i + score of number of images captured in scene i + score of number of persons capturing image of scene i)/3
However, i=1, 2, 3
Then, the image classification unit 905 determines (classifies) the scene with the highest score among the scene classification 1 score, the scene classification 2 score, and the scene classification 3 score obtained for the target group as the scene of the target group. .. When there are a plurality of scene classifications with the highest score, the highest scene classification in the preset priority order is adopted. For example, the order of daily life>ceremony> travel is set in advance. For example, it is assumed that the group 1205 in FIG. 12A has an imaging period of 36 hours, the number of images to be captured is 300, and the average number of persons to be imaged is 1.7. In this case, using the scene classification reference data of FIG. 13, the travel score is 45.32, the daily score is 18.38, the ceremony score is −29.92, and the scenes in group 1205 are classified as travel. .. It should be noted that which scene each group is classified into is managed by the HDD 202 in association with identification information unique to the classification destination scene for each group, for example.

<Step S1008>
If all of the groups divided in step S1006 are selected in step S1007, the process proceeds to step S1009. On the other hand, if an unselected group remains among the groups divided in step S1006, the process returns to step S1007.

<Step S1009>
The input unit 906 acquires, from the information acquired by the designating unit 901, information that identifies the main character (here, the main character ID). The hero ID is an ID unique to the person who is the hero (personal ID in FIG. 11) of the persons who appear in the captured image group acquired by the image acquisition unit 902. This hero ID may be specified by the user using the GUI described above, or may be determined by the following processing without a user operation.

For example, by the processing in step S1004 described above, it is possible to acquire information regarding which person ID has appeared in each captured image. Therefore, the number of appearances of each person in the captured image group acquired by the image acquisition unit 902, the number of appearances of each person in each scene, the number of scenes in which each person appears, and the like can be obtained. From such information, it is possible to determine whether each person should be the main character. For example, when the captured image group acquired by the image acquisition unit 902 is composed of a plurality of scenes, the personal ID of the person who appears in the most scenes is the main character ID. When the captured image group acquired by the image acquisition unit 902 is composed of a single scene, the personal ID of the person who appears most frequently in the single scene is set as the main character ID.

<Step S1010>
The image scoring unit 907 obtains a score for the main photo slot (main slot) and a score for the sub photo slot (sub slot) for one focused image in the captured image group acquired by the image acquisition unit 902. .. Here, the main photo slot and the sub photo slot will be described with reference to FIG. 16(A) to (4-4) are a group of templates input by the input unit 911. In the template 1601 of (1-1), 1602-1604 are photo slots, but the photo slot 1602 is larger in size than the other photo slots 1603 and 1604. In such a case, the photo slot 1602 is the main photo slot and the photo slots 1603 and 1604 are the sub photo slots. Therefore, reference data used to obtain the score for the main photo slot and the score for the sub photo slot for the selected image are created in advance as follows, for example.

What is preferable as the image to be placed in the main photo slot and what is preferable as the image to be placed in the sub photo slot depends on the scene of the image. FIG. 14A shows an example of the relationship between the characteristics of images that are preferable as images to be arranged in the main photo slot, the characteristics of images that are preferable as images to be arranged in the sub-photo slots, and the scenes of the images.

Therefore, the travel image group, the daily image group, the ceremony image group, and the sub-photo slot that satisfy the conditions defined in FIG. 14(A) for the main photo slot are defined as shown in FIG. 14(A). Prepare a set of travel images to satisfy, daily images, and images of ceremonies. Then, for the image group of each scene prepared for the main photo slot, the average and standard deviation of the number of faces appearing in the image group, the face appearance position, and the face size are obtained. Similarly, for the image group of each scene prepared for the sub-picture slot, the average and standard deviation of the number of faces appearing in the image group, the face appearance position, and the face size are obtained. The average and standard deviation of the number of faces, the appearance positions of the faces, and the size of the faces obtained for the main photo slot and the sub photo slot in this way are registered in the HDD 202 in advance as the reference data.

Then, the image score unit 907 obtains a score 1 for the main photo slot and a score 2 for the sub photo slot for the image of interest I as follows.

<Score for main photo slot>
Face number score 1=50-|10x (average number of faces obtained for the main photo slot for the scene of the target image I-number of faces of the protagonist appearing in the target image I)/mainly for the scene of the target image I Standard deviation of the number of faces obtained for photo slots｜
Score of face appearance position 1=50−|10×(average of face appearance positions obtained for main photo slot for scene of target image I−appearance position of main character's face appearing in target image I)/target image I About the scene Standard deviation of face appearance position obtained for the main photo slot｜
Face size score 1=50−|10×(average face size obtained for main photo slot for scene of target image I−face size of hero appearing in target image I)/mainly for scene of target image I Standard deviation of face size obtained for photo slots｜
Score 1 = (number of faces score 1 + face appearance position score 1 + face size score 1)/3

<Score for sub-picture slot>
Face number score 2=50−|10×(average number of faces obtained for the sub-photo slot for the scene of the target image I−the number of faces of the protagonist appearing in the target image I)/sub for the scene of the target image I Standard deviation of the number of faces obtained for photo slots｜
Score of face appearance position 2=50−|10×(average of face appearance positions obtained for the sub-photo slot for the scene of the image of interest I−appearance position of the main character's face appearing in the image of interest I)/of the image of interest I About the scene Standard deviation of the face appearance position obtained for the sub photo slot｜
Face size score 2=50-|10x (average face size obtained for the sub-photo slot for the scene of the target image I-the size of the hero's face appearing in the target image I)/sub for the scene of the target image I Standard deviation of face size obtained for photo slots｜
Score 2=(number of faces score 2+face appearance position score 2+face size score 2)/3

Regarding the number of “faces of the hero appearing in the image of interest I”, in the line of the image ID corresponding to the image of interest I in the image analysis information of FIG. 11, the position of the upper left corner and the right of the personal ID corresponding to the hero ID If the position of the lower corner is registered, it is "1", and if it is not registered, it is "0". In addition, regarding the appearance position and size of the “face of the hero appearing in the image of interest I”, in the line of the image ID corresponding to the image of interest I in the image analysis information of FIG. 11, the personal ID corresponding to the hero ID is registered. It can be specified by referring to the upper left corner position and the lower right corner position.

Further, since it is preferable that the image used for the photo album is in focus, the score may be further added to the captured image with the focus=“◯” shown in FIG. 11. FIG. 14B shows an example of the score 1 for the main photo slot and the score 2 for the sub photo slot, which are obtained for the picked-up images corresponding to the respective image IDs.

<Step S1011>
When all of the captured image groups acquired by the image acquisition unit 902 have been processed in step S1010, the process proceeds to step S1012. On the other hand, if an unprocessed captured image remains in the captured image group acquired by the image acquisition unit 902, the process returns to step S1010.

<Step S1012>
The input unit 908 acquires the number of spreads of the photo album from the information acquired by the designating unit 901. Then, the double-page spread allocation unit 909 determines whether the number of groups divided in step S1006 and the number of double-page spreads acquired by the input unit 908 are the same. As a result of this determination, if they are the same, the process proceeds to step S1016, and if they are not the same, the process proceeds to step S1013. For example, in the case of FIG. 12A, since the number of divided groups=8, if the number of spreads acquired by the input unit 908=8, the process proceeds to step S1016.

<Step S1013>
The spread allocation unit 909 determines whether or not the number X of groups divided in step S1006<the number Y of spreads acquired by the input unit 908. As a result of this determination, if X<Y, the process proceeds to step S1014, and if X≧Y, the process proceeds to step S1015.

<Step S1014>
The two-page spread allocation unit 909 performs group re-division (sub-scene division) of the captured image group acquired by the image acquisition unit 902 so that X=Y. In order to explain the process of step S1014 using a specific example, a case where Y=10 in the state where X=8 as shown in FIG. 12A is taken as an example. In this case, as shown in FIG. 12B, the groups 1202 and 1205 of FIG. 12A are divided by the portion indicated by the dotted line to generate ten groups 1201 to 1210. The position of this dotted line (division reference line) is obtained as follows, for example.

First, of the groups 1201 to 1208 in FIG. 12A, a group having a large number of captured images is specified. In the case of X=8 and Y=10, it is necessary to provide the division reference lines at two places, and therefore, of the groups 1201 to 1208, two groups, the group having the largest number of captured images and the group having the second largest number of captured images, are provided. Identify the group. In the case of FIG. 12A, the group having the largest number of captured images is the group 1205, and the groups having the second largest number are the groups 1201 and 1202. Here, although the number of captured images is the same in the groups 1201 and 1202, the imaging period of the group 1202 is longer than that of the group 1201. Therefore, in this case, the group 1202 is selected. Therefore, as a result, each of the group 1205 and the group 1202 is divided into two groups.

First, a method of dividing the group 1202 will be described. In FIG. 12A, there are two peaks of the number of captured images in the group 1202, and the position between them is the position of the division reference line. Next, a method of dividing the group 1205 will be described. In FIG. 12A, the group 1205 has three peaks in the number of captured images, but the difference in the number of captured images between the two groups after division is made smaller. Therefore, here, the position on the right side of the positions between the three peaks is the position of the division reference line. In addition, here, although division is performed at different locations on the shooting date, if the location where the number of captured images is large is a single day, division is performed at the location with the largest time difference in the single day. Various group dividing methods can be used as the group dividing method in this step, and the method is not limited to a specific group dividing method.

<Step S1015>
The two-page spread allocation unit 909 performs group re-division (scene integration) of the captured image group acquired by the image acquisition unit 902 so that X=Y. In the following, in order to describe the process of step S1015 using a specific example, a case where Y=6 is set in the state where X=8 as shown in FIG. 12A. In this case, several groups are integrated into one group to reduce the number of groups and achieve X=Y. As shown in FIG. 12C, by combining the groups 1203 and 1204 in FIG. 12A and the groups 1207 and 1208 in FIG. ~ 1206 is generated. The position of this dotted line (integrated reference line) is obtained as follows, for example.

First, of the groups 1201 to 1208 in FIG. 12A, a group having a small number of captured images is specified. In the case of X=8 and Y=6, since it is necessary to provide integrated reference lines at two locations, two groups, the group having the smallest number of captured images and the group having the second smallest number of captured images, among the groups 1201 to 1208 are provided. Identify the group. In the case of FIG. 12A, the group having the smallest number of captured images is the group 1208, and the groups having the second smallest number are the groups 1203 and 1207. Here, the number of captured images is the same in the group 1203 and the group 1207, but the group 1207 is adjacent to the group 1208 to be integrated. In this case, therefore, the group 1203 is selected. Therefore, as a result, the groups 1203 and 1208 are integrated.

First, a method of integrating the groups 1203 will be described. Among the groups 1202 and 1204 adjacent to the group 1203, the group 1204 has a shorter imaging period. Therefore, the group 1203 is integrated with the group 1204 to generate the group 1203 in FIG.

Since only the group 1207 is adjacent to the group 1208, the group 1208 is integrated with the group 1207 to generate the group 1206 in FIG. 12C.

Various group integration methods can be used as the group integration method in this step, and the method is not limited to a specific group integration method.

<Step S1016>
The spread allocating unit 909 allocates each of the same number of groups as the number of spreads acquired by the input unit 908 to one of the spreads. Although various forms can be used for allocating which group to which spread, for example, a group whose imaging date and time (imaging date and time of an arbitrary captured image in a group) is assigned to a more front spread is assigned. ..

<Step S1017>
The image selection unit 910 first selects one unselected spread among the spreads, and selects a captured image to be displayed in the spread from the captured image group belonging to the group assigned to the selected spread. To do. In the case of FIG. 4, four captured images are selected. A process of selecting four captured images to be displayed in the spread from the captured image group belonging to the group assigned to the selected spread will be described with reference to FIG. 15 as an example.

In FIG. 15, Start represents the image pickup date and time of the imaged image captured in the earliest in the imaged image group belonging to the group (focused group) assigned to the selected spread. End represents the image capturing date and time of the captured image most recently captured in the captured image group belonging to the target group. Therefore, (End-Start) represents the imaging period of the target group.

First, a picked-up image to be placed in the main photo slot is selected from the picked-up image group (FIG. 15A) belonging to the focused group. As the picked-up image to be placed in the main photo slot, the picked-up image having the highest score for the main photo slot is selected from the picked-up images belonging to the target group. The captured image selected as the captured image to be placed in the main photo slot is shown as (1) in FIG. 15(B).

Next, the picked-up images (three) to be arranged in the sub-photo slot are selected from the picked-up image group belonging to the target group. In order not to concentrate the image selection on a part of the image pickup period, a device for subdividing the image pickup period is devised as described below.

First, the first captured image to be placed in the sub-photo slot is selected. Therefore, first, as shown in FIG. 15C, the imaging period is divided into two divided periods. Then, as shown in FIG. 15D, of the two divided periods, for the sub-photograph slot, among the imaged image groups belonging to the remaining divided period (solid line) excluding the divided period (dotted line) to which the captured image (1) belongs The picked-up image with the highest score is selected as the picked-up image to be placed in the sub-photo slot. The captured image selected as the first captured image to be placed in the sub-photo slot is shown as (2) in FIG. 15D.

Next, the second captured image to be placed in the sub-photo slot is selected. Therefore, first, as shown in FIG. 15E, the imaging period is divided into four divided periods by dividing each divided period into two. As shown in FIG. 15(F), a sub-photograph of the captured image group belonging to the remaining divided period (solid line) excluding the divided period (dotted line) to which the captured images (1) and (2) belong among the four divided periods. The picked-up image with the highest score for the slot is selected as the picked-up image to be placed in the sub-photo slot. The captured image selected as the second captured image to be placed in the sub-photo slot is shown as (3) in FIG. 15(F).

Next, the third captured image to be placed in the sub photo slot is selected. The picked-up image is selected from the divided period G to which the picked-up images (1) to (3) do not belong, among the divided periods obtained by dividing the picked-up image to be arranged in the sub-photograph slot into four, as shown in FIG. As indicated by the diagonal lines in (G), it is assumed that no captured image belongs to the divided period G. In such a case, as shown in FIG. 15(H), the imaging period is divided into eight divided periods. As shown in FIG. 15(I), each divided period is divided into two to make eight divided periods. Among the eight divided periods, the score for the sub-photograph slot is included in the captured image group belonging to the divided period (dotted line) to which the captured images (1) to (3) belong and the remaining divided period (solid line) excluding the divided period G. The highest captured image is selected as the captured image to place in the sub-photo slot. The picked-up image selected as the third picked-up image to be placed in the sub-photo slot is shown as (4) in FIG. 15(I).

<Step S1018>
The image layout unit 912 selects a template based on the image selected in step S107 to determine the image layout. From the template group input by the input unit 911, a template to be applied to the focused spread is selected based on the captured image selected for the focused spread. In the following, in order to describe the process in step S1018 using a specific example, it is assumed that the input unit 911 inputs 16 templates (1-1) to (4-4) in FIG. 16A. In step S1017, it is assumed that one captured image for the main photo slot and two captured images for the sub photo slot are selected. In this case, as shown in FIG. 16A, a template group having three photo slots is set as a candidate. Here, when the three picked-up images selected in step S1017 are arranged from the left in order from the oldest picked-up date and time, as shown in FIG. 16B, a vertically long (vertical) image 1606 and a horizontally long (horizontal) image are displayed. It is assumed that the image 1607 is a vertically long (vertical) image 1605. Images 1606 and 1607 are captured images arranged in the sub-photo slot, and image 1605 is a captured image arranged in the main photo slot.

Next, a template is selected from the candidates in which the position of the main slot in the template corresponds to the time-series position of the main slot image in the selected image group of the scene of interest. In this embodiment, it is assumed that an image with an older shooting date and time is laid out in the upper left of the template and an image with a newer shooting date and time is laid out in the lower right. As shown in FIG. 16B, since the vertical image 1605 to be placed in the main photo slot has the latest image pickup date and time, the layout for arranging the image for the vertically long main photo slot with the latest image pickup date and time in the lower right is as follows. Any of (3-1) to (3-4) remains as a candidate.

Further, the image with the oldest imaging date and time is the image 1606 for the vertically long sub photo slot, and the layout for arranging the image for the vertically long sub photo slot with the oldest imaging date and time is (3-1), (3 -2) remains as a candidate.

Also, the image with the second oldest imaging date and time is the image 1607 for the horizontally long sub photo slot, and the layout for arranging the image for the horizontally long sub photo slot with the second oldest imaging date and time is other than the upper left and lower right corners. Only (3-2).

Therefore, in this case, the layout of (3-2) is selected as the template to be applied to the spread of interest. The method of selecting the template applied to the spread of interest is not limited to the above method, and another method may be adopted.

<Step S1019>
When the flag value acquired by the input unit 913 is a value indicating that image correction is to be performed, the image correction unit 914 corrects the dodging of the captured image selected by the image selection unit 910 in step S1017. , Correction processing such as red-eye correction and contrast correction is performed. Of course, the image correction unit 914 does not execute the correction process when the flag value acquired by the input unit 913 is a value indicating that the image correction is not performed.

<Step S1020>
The output unit 915 resizes the captured image selected in step S1017 for the photo slot in each photo slot on the template selected in step S1018 to the size of the photo slot and arranges the “page spread”. Image of ". Then, the output unit 915 outputs the generated “image of the two-page spread” as layout information. The output destination may be the display unit 205 or the HDD 202.

<Step S1021>
If all the spreads are selected in step S1017, the process according to the flowchart of FIG. 10 ends. On the other hand, when there is an unselected spread, the process returns to step S1017.

By performing the processing according to the flowchart of FIG. 10 described above, it is possible to create a photo album in which captured images are arranged in the photo slots of each spread. Next, when the spread image in which the four images selected by the image selection method shown in FIG. 15 are laid out is displayed in the display area 403 of the window 401, the user operates the operation unit 203. Then, the case where an arbitrary photo slot is designated will be described.

In FIG. 17A, the captured images (1) to (4) are selected for the main photo slot and the sub photo slot as shown in FIG. 15I, and in such a state, the captured image (3 ) Shows the case where the user operates the operation unit 203 to specify the photo slot. That is, in FIG. 17A, the captured image (3) is the candidate starting point image. When the picked-up image (3) is the candidate starting point image, the picked-up image belonging to the range 1701 near the picked-up date and time of the candidate starting point image is selected as the candidate picked-up image. The neighborhood range of the captured image (3) here corresponds to a divided period including the captured image (3) of FIG. 15(I). That is, in the present embodiment, the divided period including the starting image among the divided periods when the last image for the spread is selected is set as the “neighborhood range”. In FIG. 17A, captured images (5) and (6) among the captured images belonging to the neighborhood range 1701 are selected as candidate captured images. Note that various methods can be used as the method of selecting the captured images (5) and (6), and the selection method is not limited to a specific method. For example, two captured images may be randomly selected from the captured images (other than the captured image (3)) belonging to the neighboring range 1701, or the two captured images farthest from the captured date and time of the captured image (3) within the neighboring range 1701. May be selected. Further, even if a picked-up image of a person different from the picked-up image (3) or a picked-up image having a composition different from that of the picked-up image (3) is selected as a candidate picked-up image from the picked-up images belonging to the neighborhood range 1701. I do not care.

Although an example in which two captured images are selected as the candidate captured images is shown here, the number of captured images selected as the candidate captured images is not limited to 2 and may be any number. Here, a case where four candidate captured images are displayed as shown in FIG. 7D will be described. In the present embodiment, the upper two candidate picked-up images have a high score by the first evaluation method, and the lower two candidate picked-up images have a high score by the second evaluation method. As a result, in the present embodiment, images having different scores and high scores can be displayed as candidate images.

In this way, no matter which selection method is adopted, the candidate captured image is selected from the same divided period (a group of captured images whose image capturing dates and times are relatively close) as the candidate starting image, so that the time series in the page is destroyed. Without this, the user can easily replace the image.

Next, a case will be described where the candidate captured images are switched by instructing the front button or the rear button. It is assumed that currently captured images (5) and (6) are displayed as candidate captured images. At this time, when the user operates the operation unit 203 to instruct the front button, as shown in FIG. 17B, the captured image (2) which is the captured image before the captured image (3) in the above order is displayed. It becomes a candidate starting point image. Then, the captured image belonging to the neighborhood range 1702 (divided period 1702) of the captured date and time of the captured image (2) is selected as the candidate captured image. In FIG. 17B, the captured images (7) and (8) among the captured images belonging to the neighborhood range 1702 are selected as the candidate captured images. In addition, it is assumed that the user operates the operation unit 203 to instruct the rear button while the captured images (5) and (6) are currently displayed as the candidate captured images. At this time, as shown in FIG. 17B, the captured image (1) which is the captured image after the captured image (3) in the above order becomes the candidate starting point image. Then, the captured image belonging to the neighborhood range 1703 (divided period 1703) of the captured date and time of the captured image (1) is selected as the candidate captured image. In FIG. 17B, the captured images (9) and (10) among the captured images belonging to the neighborhood range 1703 are selected as candidate captured images.
By the method described above, when replacing an arbitrary image with another image, an image suitable as a candidate for the other image can be displayed.
In the present embodiment, an image near the image currently assigned in the image slot is displayed as a candidate image. This allows the user to easily select an image suitable for the slot when laying out the image. In the present embodiment, an image near the replacement target slot is displayed as a candidate image. Specifically, candidate images are selected and displayed from the same shooting period as the replacement target slot. This allows the user to easily replace the images without breaking the time series in the page.
Furthermore, when selecting and displaying the next candidate of the replacement candidate image, the user can easily break the time series in the page by selecting the candidate image from the neighborhood range of the slot that is previous in time. You can swap images without.

[Second Embodiment]
Hereinafter, the difference from the first embodiment will be mainly described, and unless otherwise stated below, it is assumed to be the same as the first embodiment. In the first embodiment, as shown in FIG. 17B, when the front button is instructed while the captured images (5) and (6) are displayed as candidate captured images, the captured image (2) becomes the candidate. The captured image that becomes the starting point image and belongs to the near range 1702 (corresponding to the capturing period 1702) is selected as a candidate captured image. In the present embodiment, as shown in FIG. 18, when the front button is instructed in the state where captured images (5) and (6) are displayed as candidate captured images, the area between the near range 1701 and the near range 1702 is displayed. A captured image belonging to the range 1801 is selected as a candidate captured image. In FIG. 18, among the captured images belonging to the range 1801, captured images (7′) and (8′) are selected as candidate captured images. On the other hand, when the subsequent candidate button is pressed, the selection range of the candidate image is set to the shooting period 1703 as in the first embodiment. This is because there is no image in the shooting period 1802. If there is an image in the shooting period (c), the image is selected from the shooting period (c). That is, in the present embodiment, the candidate image is selected and displayed from the shooting period that is one behind the shooting period to which the image arranged in the replacement target slot belongs. The method of selecting the candidate image from the shooting period is the same as that in the first embodiment, and thus the description is omitted.

Next, with respect to the processing for displaying a list of image candidates that can be arranged in the area designated by the user among the areas 405a to 405d in the state where the captured images are arranged in the areas 405a to 405d, respectively, of FIG. It will be described according to the flowchart. That is, the present embodiment differs from the first embodiment in that the process according to the flowchart of FIG. 6 is performed instead of performing the process according to the flowchart of FIG.

<Step S1901>
The CPU 204 receives a selection operation of any one of the areas 405a to 405d.

<Step S1902>
The CPU 204 sets, as the extraction time range, the neighborhood range of the imaging date and time of the captured image being displayed in the replacement target slot.

<Step S1903>
The CPU 204 searches for (lists up) a picked-up image candidate (candidate image) that can be arranged in the replacement target slot instead of the candidate start-point image, from the picked-up images belonging to the extraction time range.

<Step S1904>
The CPU 204 causes the display unit 205 to display a window including the candidate images listed in step S1903.

<Step S1905>
The CPU 204 determines whether or not the previous button 510 (previous candidate button) is designated. If the result of this determination is that it has been designated, the processing proceeds to step S1906, and if it has not been designated, the processing proceeds to step S1907.

<Step S1906>
The CPU 204 sets a time range adjacent to the extraction time range and having a date and time in the past as a new extraction time range. Then, the process returns to step S1903.

<Step S1907>
The CPU 204 determines whether or not the rear button 511 (rear candidate button) is designated. If the result of this determination is that it has been designated, the processing proceeds to step S1908, and if it has not been designated, the processing proceeds to step S1909.

<Step S1908>
The CPU 204 sets a time range adjacent to the extraction time range and having a new date and time as a new extraction time range. Then, the process returns to step S1903.

<Step S1909>
The CPU 204 determines whether or not the user operates the operation unit 203 to select one of the candidate captured images in the displayed window. If the result of this determination is that it has been selected, processing advances to step S1910, and if it has not been selected, processing returns to step S1904.

<Step S1910>
The CPU 204 displays (places) the candidate captured image selected by the user operating the operation unit 203 in the replacement target slot, instead of the captured image being displayed in the replacement target slot.
In the present embodiment, an image near the image currently assigned in the image slot is displayed as a candidate image. This allows the user to easily select an image suitable for the slot when laying out the image. In the present embodiment, an image near the replacement target slot is displayed as a candidate image. Specifically, the candidate image is selected and displayed from the same shooting section as the replacement target slot. This allows the user to easily replace the images without breaking the time series in the page. Further, even when the next candidate of the replacement candidate image is selected and displayed, the user can easily break the time series in the page by selecting the candidate image from the previous shooting period temporally adjacent to each other. You can swap the images without.

[Third Embodiment]
As described above, in the first and second embodiments, the processing according to the flowcharts of FIGS. 5 and 6 is based on the premise that the photo album generated by the processing according to the flowchart of FIG. 10 is used. However, the captured image group may be divided into the same number of spreads as the number of spreads based on the shooting date and time, and the photo album may be assigned to each spread, and the photo album generated by other processing may be used. It doesn't matter.

Further, the processing according to the flowcharts of FIGS. 5 and 6 described in the first and second embodiments replaces the captured images listed in the photo album with another captured image. However, even in the case where the captured image group is divided into a plurality of groups based on the capturing date and time and the captured images belonging to one of the groups are displayed, the processing according to the flowcharts of FIGS. If you can, you can. That is, in this case, a replacement candidate for the displayed captured image is selected from the group to which the displayed captured image belongs and presented to the user.

That is, the processing according to the flowcharts of FIGS. 5 and 6 is not limited to the purpose of creating a photo album and uploading it to the PC 2, but may be used in various situations and for various purposes. It is a thing.

[Fourth Embodiment]
Instead of PC1 and PC2, any device may be used as long as it can realize the same processing. For example, instead of the PC 1, a device such as a tablet terminal device or a smartphone may be used.

[Fifth Embodiment]
Each of the areas 405a to 405d is not limited to the one in which only an image is arranged, but may be one in which text is arranged. Further, in the above-described embodiment, the areas (areas 405a to 405d) for arranging images and characters are objects. However, a subject such as a person displayed on the image may be an object, and any object that can be displayed on the screen may be an object.
[Sixth Embodiment]
In some cases, such as when the picked-up image (4) in FIG. 17 or the picked-up image (4) in FIG. 18 is selected as an image to be replaced, the next shooting period does not exist. In this case, the user's designation may be suppressed by not displaying the UI for designating the next shooting period (not displaying the rear button 511). Further, the user's designation may be suppressed, and the candidate image may be displayed from the Start time zone when the user's designation is made. Here, the case where the image in the last shooting period is the replacement target has been described as an example, but the same applies to the case where the image in the first shooting period is the replacement target.
[Seventh Embodiment]
Only one of the front button 510 and the rear button 511 may be displayed. Even if the next candidate image can be displayed only in the previous shooting period or the subsequent shooting period, the convenience for the user is enhanced.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

201: RAM 202: HDD 203: Operation part 204: CPU 205: Display part

Claims (12)

Computer,
Acquisition means for acquiring an image data group including a plurality of image data,
Dividing means for dividing the plurality of image data into a plurality of groups,
A layout display control means for displaying a layout image in which an image based on image data included in a predetermined group of the plurality of groups is displayed in a slot of a page corresponding to the predetermined group ,
First display control means for displaying a first candidate screen including, as a candidate image , an image including a shooting date and time within a predetermined time range including a shooting date and time of the first image arranged in the layout image ,
A second candidate screen including, as a candidate image, an image including a shooting date and time in at least one of a time range before the predetermined time range and a time range after the predetermined time range. Second display control means for displaying
Changing means for changing from the first candidate screen to the second candidate screen,
Arrangement means for arranging a candidate image selected by the user from the candidate images
It is a program to function as
The program, wherein the predetermined time range and at least one of the time ranges are included in a shooting period of the predetermined group. The at least any one of the time ranges includes a time range including a date and time when a second image of the candidate images included in the second candidate screen is different from the first image , or the predetermined time range and the second image. The program according to claim 1, wherein the program is a time range including the shooting date and time . The layout display control means,
An image based on image data satisfying a predetermined criterion of the image data group included in the predetermined group is arranged in the first slot ,
The image data other than the image data satisfying the specified standard among the image data group included in the predetermined group is arranged in the second slot smaller in size than the first slot. The program according to Item 1 or 2 . The program according to claim 1, wherein the page corresponding to the predetermined group is a spread page. 4. The program according to claim 1, further causing the computer to function as a detection unit that detects an object included in the image data in the image data group. The program according to claim 5, wherein the detection means can detect animals other than humans. An acquisition step of acquiring an image data group including a plurality of image data,
A dividing step of dividing the plurality of image data into a plurality of groups,
A layout display control step of displaying a layout image in which an image based on image data included in a predetermined group of the plurality of groups is displayed in a slot of a page corresponding to the predetermined group ;
A first display control step of displaying a first candidate screen including, as a candidate image , an image including a shooting date and time within a predetermined time range including a shooting date and time of the first image arranged in the layout image ,
A second candidate screen including, as a candidate image, an image including a shooting date and time in at least one of a time range before the predetermined time range and a time range after the predetermined time range. A second display control step for displaying
A changing step of changing from the first candidate screen to the second candidate screen;
An arranging step of arranging a candidate image selected by the user from the candidate images ,
It said predetermined time range, and, wherein at least one of time range, the image processing method comprising Rukoto included in the shooting period of the predetermined group. The at least any one of the time ranges includes a time range including a date and time when a second image of the candidate images included in the second candidate screen is different from the first image, or the predetermined time range and the second image. The image processing method according to claim 7, wherein the time range includes the shooting date and time of. In the layout display control step,
An image based on image data satisfying a predetermined criterion of the image data group included in the predetermined group is arranged in the first slot,
The image data other than the image data satisfying the specified standard among the image data group included in the predetermined group is arranged in the second slot smaller in size than the first slot. Item 9. The image processing method according to Item 7 or 8. The image processing method according to claim 7, wherein the page corresponding to the predetermined group is a spread page. The image processing method according to claim 7, further comprising a detection step of detecting an object included in the image data in the image data group. The image processing method according to claim 11, wherein an animal other than a person can be detected in the detecting step.

Images