JP6572054B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image display technique.

  Some conventional image processing apparatuses display an image related to image data on a display screen of an LCD (Liquid Cristal Display) constituting a user interface, and edit the image by designating the displayed image. In photo album editing, an image slot is arranged on each page of the photo album, and a desired layout image can be obtained by assigning a photo to the image slot. In some applications for creating a photo album, a user first designates a photo image and automatically assigns a photo image according to a photo album template. As described above, when photographic images are automatically assigned instead of manually, an image desired by the user may not be assigned, so that the images can be replaced. In Patent Literature 1, when a plurality of images designated by a user for creating a photo album are divided into groups, the pages of the photo album are associated with the groups, and the images in the image slots are exchanged, this corresponds to the page. Display candidate images from the group.

Japanese Patent No. 5449460

  In Patent Document 1, when the number of images specified by the user is large, many candidate images are displayed. In addition, since the image assigned to each page of the photo album is displayed as a candidate image, there is a problem that it is difficult to select an image without considering the characteristics of the image slot to be replaced. For example, even if the image slot to be replaced is a slot that is assumed to arrange a grayscale image, a color image is also displayed as a candidate image. In addition, a horizontally long slot may be displayed with respect to a vertically long slot.

  The present invention has been made in view of such problems, and provides a technique for displaying an image suitable as a candidate for another image when an arbitrary image is replaced with another image.

One aspect of the present invention is an acquisition means for acquiring an image data group including a plurality of image data;
Dividing means for dividing the plurality of image data into a plurality of groups;
Layout for displaying a layout image of the image based on predetermined image data among the image data group included in the predetermined group, was placed in a slot of the page corresponding to the given group of the plurality of groups on the display screen Display control means;
In a layout image of a page corresponding to the predetermined group displayed on the display screen, designation means for designating a replacement target slot for replacing an image;
Based on the shooting date and time of the first image arranged in the slot to be replaced specified by the specifying means, the image of the shooting date and time included in a predetermined time range including the shooting date and time of the first image is displayed. a first display control means for displaying the first candidate screen including a candidate image to replace the one image,
Candidate images to be replaced with the first image are images with shooting dates and times included in at least one of the time range before the predetermined time range and the time range after the predetermined time range. Second display control means for displaying a second candidate screen including:
Changing means for changing from the first candidate screen to the second candidate screen based on a user instruction;
Arrangement means for arranging a selection candidate image selected by the user from among the candidate images in the slot to be replaced ;
The predetermined time range and the at least one time range are included in the shooting period of the predetermined group,
The arrangement unit selects the selected candidate image when the candidate image included in the first candidate screen is selected by the user or when the candidate image included in the second candidate screen is selected by the user. characterized that you place the replacement target slot of the page corresponding to the given group.

  According to the configuration of the present invention, when replacing an arbitrary image with another image, an image suitable as a candidate for the other image can be displayed.

The block diagram which shows the structural example of a system. The block diagram which shows the hardware structural example of PC1. The figure which shows the structural example of a software module group. The figure which shows the example of a display of the window 401. FIG. The flowchart of the process which PC1 performs. 10 is a flowchart of processing for displaying a list of image candidates. The figure which shows an example of the image of a spread page, and the window containing a candidate image. The figure which shows the example of a transition of the window containing a candidate image. FIG. 11 is a block diagram of a functional unit that executes the process of FIG. 10 is a flowchart of processing performed by executing an automatic layout processing unit 916. The figure which shows an example of image analysis information. The figure which shows an example of grouping of a captured image. The figure which shows an example of scene classification reference | standard data. The figure for demonstrating the process of step S1010. The figure explaining the process of step S1017. The figure which shows a template group. The figure explaining the selection method of a candidate picked-up image. The figure explaining the selection method of a candidate picked-up image.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
In this embodiment, an example of an image processing apparatus will be described. The image processing apparatus acquires an image data group including a plurality of image data, and displays a layout image in which an image based on image data selected from the acquired image data group is arranged in a slot on a display screen (first image data). Display control). Candidates for selecting a replacement target slot to be replaced in the layout image displayed on the display screen and replacing the first image based on the shooting date and time of the first image arranged in the selected replacement target slot An image is displayed on the display screen (second display control).

  First, a configuration example of a system according to the present embodiment will be described with reference to the block diagram of FIG. As shown in FIG. 1, the system according to the present embodiment includes a PC (personal computer) 1 operated by a user who creates a photo album, and a PC 2 which is an ABC company server that provides a photo album ordering site. doing. PC1 and PC2 are connected to each other via the Internet so that data communication is possible.

  First, the PC 1 will be described. An application 80 and a Web browser 143 are installed on the PC 1.

  The application 80 is an application for providing various functions for creating a photo album and uploading it to the PC 2.

  The Web browser 143 is a browser used when using WWW (World Wide Web). The Web browser 143 provides various functions for accessing the PC 2 and uploading the photo album created on the PC 1 side to the PC 2 or enabling the order of the photo album via the order site provided by the PC 2. Is an application.

  The PC 1 is connected to a local area network 4 such as Ethernet (registered trademark) (for example, a home network for general homes built in a user's home), and the local area network 4 is connected to the Internet. ing.

  Next, the PC 2 will be described. The PC 2 is installed with a Web server 9 that is an application of a server function, and the Web server 9 further includes an application of a shopping site 11 that is a photo album ordering site.

  The PC 2 is connected to a local area network 8 (for example, an office network built in the company of ABC) such as Ethernet (registered trademark), and the local area network 8 is connected to the Internet.

  Note that an OS (operating system) is installed in both PC1 and PC2, and the application runs on this OS in both PC1 and PC2.

  Next, a hardware configuration example of the PC 1 will be described with reference to the block diagram of FIG. The hardware configuration of the PC 2 may be the same as the hardware configuration shown in FIG. 2, or a hardware configuration different from the hardware configuration shown in FIG. 2 may be adopted.

  The RAM 201 has an area for storing computer programs and data loaded from the HDD (hard disk drive) 202 and computer programs and data received from the outside (for example, the PC 2) via the communication control unit 207. Further, the RAM 201 has a work area used when the CPU 204 executes various processes. Thus, the RAM 201 can provide various areas as appropriate.

  The HDD 202 is an example of a large-capacity information storage device, and the HDD 202 stores a computer program and data for causing the CPU 204 to execute each process described later as performed by the OS and the PC 1. This computer program includes the above-described application 80 and Web browser 143. In addition, this data includes what is described as known information in the processing described later. Computer programs and data stored in the HDD 202 are appropriately loaded into the RAM 201 under the control of the CPU 204 and are processed by the CPU 204.

  The operation unit 203 includes an interface such as a keyboard and a mouse, and various operation inputs can be input to the CPU 204 by a user operation.

  The CPU 204 executes processing using the computer program and data stored in the RAM 201, thereby controlling the operation of the entire PC 1 and executing or controlling each processing described later as what the PC 1 performs.

  The communication control unit 207 performs data communication with the PC 2 via the local area network 4 and the Internet.

  The display unit 205 is configured by a CRT, a liquid crystal screen, or the like, and can display the processing result by the CPU 204 with images and characters. The display unit 205 and the operation unit 203 may be integrated to form a touch panel screen.

  The RAM 201, the HDD 202, the operation unit 203, the CPU 204, the communication control unit 207, and the display unit 205 are all connected to the bus 206.

  Next, a configuration example of the software module group stored in the HDD 202 of the PC 1 will be described with reference to FIG. Note that the software modules shown in FIG. 3 show a part of the software module group stored in the HDD 202, but not all. In addition, the software module group stored in the HDD 202 is appropriately changed according to the OS installed in the PC 1, the usage environment of the PC 1, and the like.

  The module 92 is an Ethernet control stack that controls the Ethernet. The module 91 is an IP network control stack that controls the IP network. The module 90 is a WSD control stack for controlling WSD (Web Service on Devices) that provides a device search mechanism on the network. The module 88 is a PnP-X control stack that controls plug and play of the network. Note that PnP-X is an abbreviation for Plug and Play Extensions, which is a standard function installed in Windows 8 (registered trademark) as a series of plug-and-play extended functions that provide support for network-connected devices. The module 85 is a device driver group, and includes a standard driver group 87 included in the OS as a standard and an IHV driver group 86 provided by Independent Hardware Vendor (IHV).

  The module 84 is an application / DDI interface, and includes an Application Programming Interface (API) and a Device Driver Interface (DDI). The application group 82 is an application group including the application 80 and the web browser 143 described above.

  Next, a process for creating a desired photo album by activating the application 80 in the PC 1 and arranging a desired captured image in a prescribed area on each page of the photo album will be described. When the application 80 is activated, a window 401 illustrated in FIG. 4 is displayed on the display screen of the display unit 205. Thereafter, unless otherwise specified, the subject of the subsequent processing is the display control of various GUIs including the window 401 and the processing according to the operation input performed by the user operating the operation unit 203 using the GUI. Both are assumed to be the CPU 204.

  In the display area 402, thumbnail images of the spread pages in the photo album are displayed. In the display area 402, from the left, thumbnail images of the spread pages by the first and second pages, thumbnail images of the spread pages by the third and fourth pages,..., Spread pages by the ninth and tenth pages. A thumbnail image of the page is displayed. Also, in FIG. 4, the user operates the operation unit 203 to designate the thumbnail images of the spread pages for the third and fourth pages. In this case, the thumbnail image is in a selected state (the frame 404 is superimposed on the thumbnail image), and the display area 403 displays “third page and fourth page corresponding to the thumbnail image. "A spread page image" is displayed. The “images of the spread pages by the third page and the fourth page” are provided with areas (photo slots) 405a to 405d in which captured images can be arranged, and in these areas 405a to 405d, processing to be described later is performed. Thus, the captured image is resized to the size of the area and arranged.

  Here, when the user operates the operation unit 203 and specifies any one of the areas 405a to 405d in a state where the captured images are arranged in each of the areas 405a to 405d, the image is displayed in the one area. A list of captured image candidates that can be arranged in the one area is displayed instead of the captured image. However, the user designates a desired candidate image among the candidate images displayed in the list by operating the operation unit 203 to place the designated candidate image in the one region (image replacement). Can do.

  When the user operates the operation unit 203 and designates the button 406, the CPU 204 controls the communication control unit 207 to upload the created photo album data to the PC 2.

  Next, a process for displaying a list of image candidates that can be arranged in an area designated by the user among the areas 405a to 405d in a state where captured images are arranged in each of the areas 405a to 405d will be described with reference to FIG. It demonstrates according to a flowchart.

<Step S101>
The user operates the operation unit 203 to designate any one of the regions 405a to 405d (replacement target slot), thereby causing the display unit 205 to display a list of image candidates that can be arranged in the region. Can do. However, the CPU 204 accepts a selection operation of any one of the areas 405a to 405d.

<Step S102>
The CPU 204 sets the captured image being displayed in the replacement target slot as a starting point (candidate starting point image) for searching the HDD 202 for a captured image candidate that can be placed in the replacement target slot instead of the captured image.

<Step S103>
The CPU 204 searches for (lists up) from the HDD 202 a candidate for a captured image (candidate image) that can be placed in the replacement target slot instead of the candidate origin image, based on the candidate origin image. Details of the processing in this step will be described later.

<Step S104>
The CPU 204 causes the display unit 205 to display a window including the candidate images listed in step S103. This window will be described with reference to FIG. FIG. 7A shows “images of two-page spread pages based on the first page and the second page”, and reference numerals 500 to 502 denote photo slots. FIG. 7B shows “a spread page image by the third and fourth pages”, and reference numerals 503 to 505 denote photo slots. In FIG. 7B, the photo slots 504 and 505 overlap the photo slot 503, but there may be a case where a photo slot is further arranged on the photo slot in this way. FIG. 7C shows an image of a spread page by the fifth and sixth pages, and reference numerals 506 to 507 denote photo slots.

  Here, in the state where the “page image of the second and fourth pages” in FIG. 7B is displayed in the display area 403, the captured image displayed in the photo slot 503 is captured by another image. It is assumed that the user operates the operation unit 203 and designates a photo slot 503 in order to display the candidate image in order to replace the image. At this time, in step S <b> 104, the window illustrated in FIG. 7D is displayed on the display unit 205.

  In this window, candidate captured images 641 to 644 (captured images listed based on the captured images displayed in the photo slot 503 in step S103), which are candidates of captured images that can be arranged in the photo slot 503, are listed. It is displayed. The user can specify a desired one of the candidate captured images 641 to 644 to be placed in the photo slot 503 by operating the operation unit 203. Here, it is assumed that the user designates a desired one of the candidate captured images 641 to 644 to be placed in the photo slot 503 by operating the operation unit 203. At this time, the CPU 204 resizes the designated candidate captured image to the size of the photo slot 503 instead of the captured image currently displayed in the photo slot 503, and arranges (displays) the image in the photo slot 503.

  The previous button 510 is used to display a candidate image based on the captured image arranged in the photo slot immediately before the currently selected photo slot in the photo slot order ordered through all pages of the photo album. Button. In the case of FIG. 7D, when the user operates the operation unit 203 and designates the front button 510, the CPU 204 causes the candidate image based on the captured image arranged in the photo slot in the previous order to the photo slot 503. Is displayed.

  The back button 511 is used to display a candidate image based on the captured image arranged in the photo slot immediately after the currently selected photo slot in the photo slot order ordered through all pages of the photo album. Button. In the case of FIG. 7D, when the user operates the operation unit 203 and designates the back button 511, the CPU 204 selects a candidate image based on the captured image arranged in the photo slot in the next order after the photo slot 503. Is displayed.

  When the front button 510 or the rear button 511 is designated, the CPU 204 sets the captured image being displayed in the new photo slot as a candidate starting image, and lists a new candidate captured image based on the candidate starting image. Then, the CPU 204 displays the listed candidate captured images as a list in the window. FIG. 8 shows an example of transition of the candidate captured image in the window when the front button 510 and the rear button 511 are designated.

  The window shown in FIG. 8A is displayed when the photo slot 500 is selected in “the image of the spread page by the first page and the second page” (FIG. 7A) (referred to as image 1-2). The window to be In this window, a list of candidate captured images 600 to 603 listed based on the captured images being displayed in the photo slot 500 is displayed. When the back button 511 is designated in this window, as shown in FIG. 8B, a window displaying a list of candidate captured images 610 to 613 listed based on the captured images being displayed in the photo slot 501 is displayed. The If the previous button 510 is designated in this window, the window transitions to the window of FIG. On the other hand, when the back button 511 is designated in the window of FIG. 8B, a list of candidate captured images 620 to 623 listed based on the captured images being displayed in the photo slot 502 is displayed as shown in FIG. 8C. The displayed window is displayed. If the previous button 510 is designated in this window, the window changes to the window shown in FIG. On the other hand, assume that the back button 511 is designated in the window of FIG. In this case, as shown in FIG. 8D, the list is based on the picked-up images being displayed in the photo slot 503 of “images of spread pages by the third and fourth pages” (referred to as images 3-4). A window including the uploaded candidate captured images 630 to 633 is displayed. If the previous button 510 is designated in this window, the window changes to the window of FIG. On the other hand, when the rear button 511 is designated in the window of FIG. 8D, candidate imaging listed based on the captured image displayed in the photo slot 504 in the image 3-4 as shown in FIG. 8E. A window displaying a list of images 640 to 643 is displayed. If the previous button 510 is designated in this window, the window transitions to the window of FIG. On the other hand, when the back button 511 is designated in the window of FIG. 8E, candidate imaging listed based on the captured image displayed in the photo slot 505 in the image 3-4 as shown in FIG. 8F. A window displaying a list of images 650 to 653 is displayed. If the previous button 510 is designated in this window, the window changes to the window shown in FIG. On the other hand, assume that the back button 511 is designated in the window of FIG. In this case, as shown in FIG. 8 (g), the list is based on the picked-up images being displayed in the photo slot 506 of “images of spread pages by the fifth and sixth pages” (referred to as images 5-6). A window including the uploaded candidate captured images 660 to 663 is displayed. If the previous button 510 is designated in this window, the window changes to the window shown in FIG. On the other hand, when the rear button 511 is designated in the window of FIG. 8G, candidate imaging listed based on the captured image displayed in the photo slot 507 in the image 5-6 as shown in FIG. 8H. A window displaying a list of images 670 to 673 is displayed. When the previous button 510 is designated in this window, the window changes to the window shown in FIG.

  As described above, in this embodiment, when the user operates the operation unit 203 and designates the front button 510 or the rear button 511, a list of candidate captured images corresponding to the photo slots straddling the spread page image is displayed. Can be made.

  Further, by designating the front button 510 and the rear button 511, as an image which becomes a candidate to be replaced with the captured image being displayed in the photo slot selected in the display area 403, for the photo slot outside the page to which the photo slot belongs. Candidate captured images can also be displayed. As a result, a captured image that is completely different from the captured image that can be arranged in the target photo slot can be arranged in the target photo slot.

  In the above example, “the photo slot order ordered through all pages of the photo album” is the upper left photo slot, the upper right photo slot, the lower left photo slot, and the lower right photo slot. However, it is not limited to this order. Moreover, although the photo slot allocated to the whole page like the photo slot 503 is the first order of the page, this is also an example and the present invention is not limited to this. For example, other orders such as upper left, lower left, upper right, and lower right may be adopted. In the above example, the number of candidate captured images displayed in the window is four. However, the number is not limited to this number.

  In the above example, the candidate images based on the captured images arranged in the previous or next photo slot by the front button 510 or the rear button 511 are displayed, but the other photo slots are displayed on the window. There are various methods for generating candidate images based on the arranged captured images, and the method is not limited to a specific method. For example, an input field for designating how much previous or subsequent photo slot is to be selected may be provided, and a numerical value may be input there (for example, “-2 when selecting the second previous photo slot”). "If you want to select the second photo slot, enter" 2 ").

  Returning to FIG. 7, the button 520 is a button for displaying a captured image other than the candidate captured images displayed in the window. When the user designates this button 520 by operating the operation unit 203, the CPU 204 causes the display unit 205 to display a list of images that the user designates first when creating a photo album.

<Step S105>
The CPU 204 determines whether or not the previous button 510 (previous candidate button) has been designated. As a result of this determination, if it is designated, the process proceeds to step S106, and if it is not designated, the process proceeds to step S107.

<Step S106>
In the “photo slot order ordered through all pages of the photo album”, the CPU 204 sets the captured image displayed in the photo slot immediately before the photo slot that is the current candidate start image as a new candidate start image. Set to. Then, the process returns to step S103.

<Step S107>
The CPU 204 determines whether or not a back button 511 (back candidate button) has been designated. As a result of this determination, if it is designated, the process proceeds to step S108, and if it is not designated, the process proceeds to step S109.

<Step S108>
In the “photo slot order ordered through all pages of the photo album”, the CPU 204 sets the captured image displayed in the photo slot immediately after the photo slot that is the current candidate start image as a new candidate start image. Set to. Then, the process returns to step S103.

<Step S109>
The CPU 204 determines whether or not the user has selected any one of the candidate captured images in the displayed window by operating the operation unit 203. As a result of the determination, if selected, the process proceeds to step S110, and if not selected, returns to step S104.

<Step S110>
The CPU 204 replaces the captured image being displayed in the replacement target slot with the candidate captured image selected by the user by operating the operation unit 203 and displays it.

  Next, a process for generating the window 401 as illustrated in FIG. 4 and causing the display unit 205 to display it by executing the application 80 by the CPU 204 will be described. The processing of FIG. 5, that is, the processing for displaying a list of captured image candidates to be displayed instead of the captured image being displayed in the photo slot and displaying the candidate selected by the user in the photo slot, is described below. This is a process performed on the window generated by the 10 processes.

  First, a functional unit that executes processing according to the flowchart of FIG. 10 in the application 80 will be described with reference to the block diagram of FIG. As shown in FIG. 9, the processing according to the flowchart of FIG. 10 is executed by the designation unit 901 and the automatic layout processing unit 916. As shown in FIG. 9, the automatic layout processing unit 916 includes a plurality of functional units, and the functions of the automatic layout processing unit 916 are realized by the plurality of functional units. In FIG. 9, the designation unit 901 is illustrated as a functional unit separate from the automatic layout processing unit 916, but the designation unit 901 may be included in the automatic layout processing unit 916.

  For the sake of explanation, the functional unit shown in FIG. 9 may be described as the subject of processing. In practice, the function of the functional unit is realized by the CPU 204 executing a computer program corresponding to the functional unit. Is done.

  In addition, some or all of the functions of the functional units described later may be left to other functional units, or may be realized by performing cooperative operations with other functional units. That is, as long as the function of the automatic layout processing unit 916 is realized by a plurality of functional units, the operation of each functional unit is not limited to a specific form.

  The designation unit 901 acquires “various information constituting the photo album creation conditions” and supplies the acquired information to the automatic layout processing unit 916. “Various information constituting photo album creation conditions” is information set on a GUI (graphical user interface) displayed on the display unit 205 when the user operates the operation unit 203, for example. When the user inputs a completion instruction after setting these pieces of information on the GUI, the designation unit 901 acquires “various information constituting the photo album creation conditions” set on the GUI. Of course, the designation unit 901 may acquire from the HDD 202 “various information constituting the photo album creation conditions” created in advance on the GUI and stored in the HDD 202. The conditions to be set include, for example, designation of the target image and the main character ID, the number of album spreads, template information, ON / OFF conditions for image correction, and the like. The designation of the image may be, for example, designation by accompanying information or attribute information of individual image data such as the shooting date and time, or designation based on the structure of the file system including the image data such as designation of a device or directory. Good. In the present embodiment, the “page spread” corresponds to, for example, one display window in display, and corresponds to a pair of adjacent pages printed on different sheets in printing.

  The image acquisition unit 902 reads a captured image data group stored in the HDD 202.

  For each captured image data acquired by the image acquisition unit 902, the image conversion unit 903 has reduced image data of the captured image data (for example, the short side has a size of 490 pixels and each pixel has sRGB color information). Is generated as an analysis image.

  The image analysis unit 904 performs analysis processing such as feature amount extraction, face recognition, personal recognition, and facial expression recognition on each analysis image generated by the image conversion unit 903, and a captured image that is a generation source of the analysis image Acquisition date and time of the captured image is acquired from the incidental information (for example, Exif information).

  The image classification unit 905 divides the captured image group acquired by the image acquisition unit 902 into a plurality of scenes based on the result of the analysis processing by the image analysis unit 904 and the imaging date and time, and classifies the scenes. A scene is a shooting scene such as a trip, daily life, wedding or the like. A scene can also be said to be a collection of images taken at one shooting opportunity for one shooting target, for example.

  The input unit 906 acquires information (for example, identification information of the hero) that identifies the hero among the information acquired by the specifying unit 901.

  The image score unit 907 obtains a score for each captured image data acquired by the image acquisition unit 902.

  The input unit 908 acquires the number of photo album spreads from the information acquired by the specifying unit 901.

  The spread assignment unit 909 assigns each group classified by the image classification unit 905 or each group rearranged based on each group to each spread of the photo album.

  The image selection unit 910 arranges each spread in the photo slot in the spread of each captured image based on the score obtained by the image scoring unit 907 for each captured image in the group assigned to the spread. The captured image to be selected is selected.

  The input unit 911 acquires, from the information acquired by the specifying unit 901, a template group (a template having a different layout) representing a spread layout (a layout of each photo slot in the spread). The template here is an image of a spread page.

  The image layout unit 912 selects one template based on the captured image selected by the image selection unit 910 from among the templates input by the input unit 911.

  The input unit 913 acquires a flag value indicating whether or not to perform image correction on the captured image among the information acquired by the specifying unit 901.

  When the flag value acquired by the input unit 913 is a value indicating that image correction is to be performed, the image correction unit 914 performs dodging correction and red-eye correction on the captured image selected by the image selection unit 910. Then, correction processing such as contrast correction is performed. Of course, the image correction unit 914 does not perform the correction process when the flag value acquired by the input unit 913 is a value indicating that image correction is not performed.

  The output unit 915 “spread page image” in which the captured image assigned to the photo slot is resized to the size of the photo slot in each photo slot on the template selected by the image layout unit 912. Is generated. Then, the output unit 915 outputs the generated “page spread image” as layout information.

  In the above description, the image to be placed in the image correction or the photo slot is a captured image, but may be an analysis image generated from the captured image. Of course, another image generated from the captured image may be used.

  Next, processing performed by the CPU 204 executing the automatic layout processing unit 916 will be described with reference to the flowchart of FIG.

<Step S1001>
The image conversion unit 903 sequentially focuses on images included in the designated image group, and generates the analysis image from the target image. The analysis image is generated based on “generation parameters such as the number of vertical and horizontal pixels of the analysis image and the color space of the pixel value” included in the information acquired by the specifying unit 901.

<Step S1002>
The image analysis unit 904 extracts the feature amount from the analysis image generated in step S1001, and acquires the imaging date and time of the captured image from the incidental information of the captured image that is the generation source of the analysis image. Various features can be used as the feature amount to be extracted. In the following, the focus amount is extracted as the feature amount as an example.

  A known Sobel filter can be used as a method for detecting an edge in an image. An edge slope (luminance gradient) can be calculated by detecting an edge from an image using a Sobel filter and dividing the luminance difference between the start point and the end point of the detected edge by the distance between the start point and the end point. Then, the average value of the inclination of the edge in the image is calculated. An image having a large average value of inclination can be considered to be in focus as compared with an image having a small average value of inclination. Therefore, by setting a plurality of different threshold values for the inclination in advance and determining which threshold value the average value of the inclination obtained from the image is greater than, it is possible to calculate the focus amount in the image. In the present embodiment, two different threshold values θ1 and θ2 (θ1 <θ2) are set, and if the average value A of the obtained slopes is A <θ1, the focus amount = ×, and if θ1 ≦ A <θ2, the focus amount. If Δ = Δ and A ≧ θ2, it is determined that the focus amount = ◯. “Amount of focus = ◯” represents the inclination of the focus to be adopted for the photo album. The focus amount = Δ represents an allowable focus inclination. The amount of focus = x represents an unacceptable inclination.

<Step S1003>
The image analysis unit 904 detects a human face from the analysis image generated in step S1001. A technique for detecting a face from an image is a known technique, and for example, Adaboost that creates a strong classifier from a plurality of weak classifiers prepared. In this embodiment, a human face is detected from an image by a strong classifier created by Adaboost. By this face detection process, the coordinate position and size of the human face area (face area (rectangular area)) on the image are acquired. Instead of the size, the coordinate position of the upper left corner and the coordinate position of the lower right corner of the face area may be acquired. Note that a strong classifier can be created by Adaboost using objects such as dogs and cats, animals such as dogs and cats, flowers, food, buildings, and figurines as detection targets instead of faces. Thereby, objects other than the face can be detected.

<Step S1004>
The image analysis unit 904 performs a personal recognition process, which is a process of recognizing who the face in the face area is for each face area detected in step S1003. For example, the similarity between the images of various human faces registered in advance in the HDD 202 and the images in the face area detected in step S1003 is obtained. Then, among various facial images registered in advance in the HDD 202, an image that is equal to or greater than the threshold and for which the maximum similarity is obtained is identified, and identification information (in advance) associated with the identified image ( Identification information (personal ID) of an individual having a face in the image is acquired. If all of the obtained similarities are less than the threshold, new identification information is issued, and the image in the face area detected in step S1003 is registered in the HDD 202 in association with the issued identification information. .

  In this step, any personal recognition process may be adopted as long as identification information of a person having a face in the face area detected in step S1003 can be acquired.

  The image analysis unit 904 registers the information collected in steps S1002 to S1004 in the HDD 202 as image analysis information. An example of the image analysis information is shown in FIG. The image analysis information includes, for each captured image, an ID (image ID) unique to the captured image, the imaging date and time of the captured image, the focus amount, the number of human faces detected from the captured image, and the captured image. The coordinate positions of the upper left corner and the lower right corner of each detected human face area are included. For example, the following information is registered for the captured image with the image ID = 1.

Date and time of imaging = “10:11:12 on July 1, 2015”
Focus amount = "○"
Number of detected human faces = “6”
Upper left corner coordinate position and lower right corner coordinate position of person ID = 1 person face area Upper left corner coordinate position and lower right corner coordinate position of person ID = 2 person face area Person ID = 3 person Upper left corner coordinate position and lower right corner coordinate position of human face area Upper left corner coordinate position and lower right corner coordinate position of human face area with personal ID = 4 Upper left corner of human face area with personal ID = 5 The coordinate position of the corner and the coordinate position of the lower right corner The coordinate position of the upper left corner and the coordinate position of the lower right corner of the face area of the person with personal ID = 6 <Step S1005>
When all the captured image groups acquired by the image acquisition unit 902 are selected in step S1001, the process proceeds to step S1006 via step S1005. On the other hand, when an unselected captured image remains in the captured image group acquired by the image acquisition unit 902, the process returns to step S1001.

<Step S1006>
The image classification unit 905 acquires the imaging date and time of each captured image acquired by the image acquisition unit 902 from the image analysis information registered in the HDD 202. Then, the image classification unit 905 divides the captured image group (captured image data group) acquired by the image acquisition unit 902 into a plurality of groups based on the acquired captured date and time of each captured image.

  For example, when the captured images (captured image data) acquired by the image acquisition unit 902 are viewed in order from the oldest captured date and time, the captured date and time of the Nth captured image (N is an integer of 1 or more) and (N + 1) It is assumed that the difference between the captured date and time of the first captured image is one day or more. At this time, a group boundary is set between the Nth captured image and the (N + 1) th captured image. In addition, if the difference between the imaging date / time of the N-th captured image and the imaging date / time of the (N + 1) -th captured image is 16 hours or more in order from the oldest imaging date / time, the N-th captured image and the (N + 1) -th captured image Set group boundaries between images. In addition, it is assumed that the difference in imaging date / time between adjacent captured images in the order of imaging date / time is less than 16 hours between any captured images. In this case, if the difference between the imaging date / time of the first captured image and the imaging date / time of the last captured image in the captured image group captured in one day is less than 4 hours, One group. On the other hand, if the difference between the imaging date and time of the first captured image and the imaging date and time of the last captured image in the group of captured images captured on the first day is 4 hours or more, the imaging captured on the first day If the number of images is less than 50 (the number of captured image data captured in one day is less than 50), the group of captured images for one day is defined as one group. An example of such grouping of captured images is shown in FIG. 12A to 12C, the horizontal axis indicates the imaging date and time (older toward the left side and newer toward the right side), and the vertical axis indicates the number of captured images. In FIG. 12A, the captured image group is divided into eight groups 1201 to 1208. In FIG. 12A, arrows indicate group boundaries.

<Step S1007>
The image classification unit 905 selects one of the groups divided in step S1006 as a target group, and determines which scene the target group is classified into. In the present embodiment, it is determined whether the group is classified into a travel scene, a daily life scene, or a ceremony scene. Of course, the types of scenes are not limited to these types.

  For this step, data (scene classification reference data) serving as a reference for determining in which scene the group is classified is registered in the HDD 202 in advance. An example of a method for creating scene classification reference data is shown below.

  For example, an image group (travel image group) previously determined as a travel image, an image group (daily image group) previously determined as a daily image, and an image group (ceremony image) previously determined as a ceremony image A plurality of image groups are prepared in advance. Then, for each of the travel image group, the daily image group, and the ceremony image group, the average and standard deviation of the imaging period, the average and standard deviation of the number of captured images, and the average and standard deviation of the number of captured people are obtained. For example, the case where the average and standard deviation of the imaging period, the average and standard deviation of the number of captured images, and the average and standard deviation of the number of photographed people are obtained for the travel image group will be described.

  The imaging period of the image group is a difference between the imaging date / time of the image captured most recently in the image group and the imaging date / time of the most recently captured image. However, for each travel image group, the difference between the imaging date / time of the image captured most recently in the travel image group and the imaging date / time of the most recently captured image is obtained as the imaging period of the travel image group, and the respective travel images The average and standard deviation of the imaging period obtained for the image group are obtained.

  Further, the number of captured images of the image group is the number of images constituting the image group. However, for each travel image group, the number of captured images constituting the travel image group is determined, and the average and standard deviation of the number of captured images determined for each travel image group are determined.

  In addition, the number of people who take an image group is the number of faces that appear in each image constituting the image group. However, for all the travel image groups, the number of faces appearing in each captured image constituting the travel image group is obtained by a known technique such as face recognition, and the average and standard deviation of the obtained face numbers are obtained. Ask for.

  Similarly, for the daily image group and the ceremony image group, the average and standard deviation of the imaging period, the average and standard deviation of the number of images taken, and the average and standard deviation of the number of people taken are obtained. Then, the average and standard deviation of the imaging period, the average and standard deviation of the number of images taken, the average and standard deviation of the number of people taken for each of the travel image group, the daily image group, and the ceremony image group are obtained in the scene. It is registered in advance in the HDD 202 as classification reference data. An example of the scene classification reference data is shown in FIG.

  The image classification unit 905 obtains the average of the imaging period, the number of captured images, and the number of captured people for the group of interest in the same manner as described above. Then, the image classification unit 905 refers to the scene classification standard data, and for the group of interest, scores for travel (scene classification 1), scores for daily life (scene classification 2), and scores for ceremony (scene classification 3). Obtained as follows.

Score of imaging period of scene i = 50− | 10 × (average of imaging period of scene i−imaging period of selected group) / standard deviation of imaging period of scene i |
Score of number of shots of scene i = 50− | 10 × (average of number of shots of scene i−number of shots of selected group) / standard deviation of number of shots of scene i |
Score of the number of photographers of scene i = 50− | 10 × (average of number of photographers of scene i−average number of photographers of selected group) / standard deviation of number of photographers of scene i |
Score of scene i = (score of imaging period of scene i + score of number of images of scene i + score of number of people of scene i) / 3
However, i = 1, 2, 3
Then, the image classification unit 905 determines (classifies) the scene with the highest score among the score of scene classification 1, the score of scene classification 2, and the score of scene classification 3 obtained for the target group as the scene of the target group. . When there are a plurality of scene classifications having the highest score, the highest scene classification is adopted in a preset priority order. For example, the order of daily life>ceremony> travel is set in advance. For example, it is assumed that the imaging period of the group 1205 in FIG. 12A is 36 hours, the number of images to be captured is 300, and the average number of people to be captured is 1.7. In this case, using the scene classification reference data in FIG. 13, the travel score is 45.32, the daily score is 18.38, the ceremony score is −29.92, and the scene of the group 1205 is classified as travel. . Note that the scene into which each group is classified is managed by the HDD 202 in association with identification information unique to the scene to be classified for each group, for example.

<Step S1008>
If all the groups divided in step S1006 have been selected in step S1007, the process proceeds to step S1009. On the other hand, if an unselected group remains among the groups divided in step S1006, the process returns to step S1007.

<Step S1009>
The input unit 906 acquires information (here, referred to as a hero ID) that identifies the hero among the information acquired by the specifying unit 901. The protagonist ID is an ID (personal ID in FIG. 11) unique to the person as the main character among the people appearing in the captured image group acquired by the image acquisition unit 902. The main character ID may be specified by the user using the above-described GUI, or may be determined without the user operation by the following processing.

  For example, the information of which personal ID has appeared in each captured image can be acquired by the processing in step S1004. Therefore, the number of appearances of each person in the captured image group acquired by the image acquisition unit 902, the number of appearances of each person in each scene, the number of scenes in which each person appears, and the like can be obtained. From such information, it is possible to determine whether each person is the main character. For example, when the group of captured images acquired by the image acquisition unit 902 includes a plurality of scenes, the personal ID of the person appearing in the most scenes is set as the main character ID. When the captured image group acquired by the image acquisition unit 902 is composed of a single scene, the personal ID of the person who appears most frequently in the single scene is set as the main character ID.

<Step S1010>
The image scoring unit 907 obtains a score for the main photo slot (main slot) and a score for the sub photo slot (subslot) for one image of interest in the captured image group acquired by the image acquisition unit 902. . Here, the main photo slot and the sub photo slot will be described with reference to FIG. (1-1) to (4-4) in FIG. 16A are template groups input by the input unit 911. In the template 1601 of (1-1), 1602 to 1604 are photo slots, but the photo slot 1602 is larger in size than the other photo slots 1603 and 1604. In such a case, the photo slot 1602 is a main photo slot, and the photo slots 1603 and 1604 are sub-photo slots. Therefore, the reference data used for obtaining the score for the main photo slot and the score for the sub photo slot for the selected image is created in advance as follows, for example.

  What is preferable as an image to be arranged in the main photo slot and what is preferable as an image to be arranged in the sub photo slot differ depending on the scene of the image. FIG. 14A shows an example of the relationship between the characteristics of an image preferable as an image to be arranged in the main photo slot, the characteristics of an image preferable as an image to be arranged in the sub photo slot, and an image scene.

  However, the conditions defined in FIG. 14A for the travel image group, the daily image group, the ceremony image group, and the sub-photo slot satisfying the conditions defined in FIG. A travel image group, a daily image group, and a ceremony image group are prepared in advance. Then, for the image group of each scene prepared for the main photo slot, the average and standard deviation of the number of faces appearing in the image group, the appearance position of the face, and the face size are obtained. Similarly, for the image group of each scene prepared for the sub-photo slot, the average number and standard deviation of the number of faces appearing in the image group, the appearance position of the face, and the face size are obtained. The average and standard deviation of the number of faces, the appearance position of the face, and the face size obtained for the main photo slot and the sub photo slot in this way are registered in advance in the HDD 202 as the reference data.

  Then, the image scoring unit 907 obtains the score 1 for the main photo slot and the score 2 for the sub photo slot for the image I of interest as follows.

<Score for main photo slot>
Score of face number 1 = 50− | 10 × (the average number of faces obtained for the main photo slot for the scene of the target image I−the number of faces of the main character appearing in the target image I) / the main scene of the scene of the target image I Standard deviation of the number of faces obtained for a photo slot |
Score of face appearance position 1 = 50− | 10 × (average of face appearance positions obtained for main photo slot for scene of target image I−appearance position of face of main character appearing in target image I) / target image I Standard deviation of the face appearance position obtained for the main photo slot for the scene |
Face size score 1 = 50− | 10 × (the average of the face sizes obtained for the main photo slot for the scene of the target image I−the size of the face of the main character appearing in the target image I) / the main of the scene of the target image I Standard deviation of face size obtained for photo slot |
Score 1 = (Face number score 1 + Face appearance position score 1 + Face size score 1) / 3

<Score for sub-photo slot>
Number of faces score 2 = 50− | 10 × (average number of faces determined for sub-photo slot for scene of target image I−number of faces of main character appearing in target image I) / sub for scene of target image I Standard deviation of the number of faces obtained for a photo slot |
Score of face appearance position 2 = 50− | 10 × (average of face appearance positions obtained for sub-photo slot for scene of target image I−appearance position of face of main character appearing in target image I) / of target image I Standard deviation of the face appearance position obtained for the sub photo slot for the scene |
Score of face size 2 = 50− | 10 × (average of face sizes obtained for sub-photo slot for scene of target image I−size of face of main character appearing in target image I) / sub for scene of target image I Standard deviation of face size obtained for photo slot |
Score 2 = (Face number score 2 + Face appearance position score 2 + Face size score 2) / 3

Regarding the number of “the face of the main character appearing in the target image I”, the position of the upper left corner and the right side of the personal ID corresponding to the main character ID in the row of the image ID corresponding to the main image I in the image analysis information of FIG. It is “1” if the position of the lower corner is registered, and “0” if it is not registered. Also, the appearance position and size of “the face of the main character appearing in the target image I” are registered in the personal ID corresponding to the main character ID in the row of the image ID corresponding to the target image I in the image analysis information of FIG. It can be specified by referring to the position of the upper left corner and the position of the lower right corner.

Further, since it is preferable that the image used for the photo album is in focus, a score may be further added to the captured image with the focus = “◯” shown in FIG. FIG. 14B shows an example of score 1 for the main photo slot and score 2 for the sub photo slot obtained for the captured images corresponding to the respective image IDs.

<Step S1011>
When all the captured image groups acquired by the image acquisition unit 902 are processed in step S1010, the process proceeds to step S1012. On the other hand, when an unprocessed captured image remains in the captured image group acquired by the image acquisition unit 902, the process returns to step S1010.

<Step S1012>
The input unit 908 acquires the number of photo album spreads from the information acquired by the specifying unit 901. Then, the spread allocation unit 909 determines whether the number of groups divided in step S1006 is the same as the number of spreads acquired by the input unit 908. If the result of this determination is the same, the process proceeds to step S1016. If not, the process proceeds to step S1013. For example, in the case of FIG. 12A, since the number of group divisions is 8, if the number of spreads acquired by the input unit 908 is 8, the process proceeds to step S1016.

<Step S1013>
The spread allocation unit 909 determines whether or not the number X of groups divided in step S1006 <the number Y of spreads acquired by the input unit 908. If X <Y as a result of the determination, the process proceeds to step S1014. If X ≧ Y, the process proceeds to step S1015.

<Step S1014>
The spread allocation unit 909 performs group re-division (sub-scene division) of the captured image group acquired by the image acquisition unit 902 so that X = Y. In the following, in order to describe the processing in step S1014 with a specific example, a case where Y = 10 in a state where X = 8 as shown in FIG. In this case, as shown in FIG. 12B, ten groups 1201 to 1210 are generated by dividing the groups 1202 and 1205 in FIG. The position of the dotted line (division reference line) is obtained as follows, for example.

  First, among the groups 1201 to 1208 in FIG. 12A, a group having a large number of captured images is specified. In the case of X = 8 and Y = 10, it is necessary to provide division reference lines at two locations. Therefore, among the groups 1201 to 1208, two of the group with the largest number of captured images and the group with the second largest number are displayed. Identify the group. In the case of FIG. 12A, the group with the largest number of captured images is the group 1205, and the groups with the second largest number are the groups 1201 and 1202. Here, the group 1201 and the group 1202 have the same number of captured images, but the imaging period of the group 1202 is longer than that of the group 1201. In this case, however, the group 1202 is selected. However, as a result, each of the group 1205 and the group 1202 is divided into two groups.

  First, a method for dividing the group 1202 will be described. In FIG. 12A, since there are two peaks of the number of captured images in the group 1202, the position between them is defined as the position of the division reference line. Next, a method for dividing the group 1205 will be described. In FIG. 12A, the group 1205 has three peaks in the number of captured images, but the difference in the number of captured images between the two groups after division is made smaller. Therefore, here, the right position among the positions between the three peaks is set as the position of the division reference line. Here, the image is divided at different locations on the shooting date. However, if the location where the number of captured images is large is a single day, the image is divided at the location where the time difference is the largest within the single date. Note that various group division methods can be used as the group division method in this step, and the group division method is not limited to a specific group division method.

<Step S1015>
The spread allocation unit 909 performs group re-division (scene integration) of the captured image group acquired by the image acquisition unit 902 as X = Y. In the following, in order to explain the processing in step S1015 with a specific example, a case where Y = 6 in the state where X = 8 as shown in FIG. In this case, X = Y is achieved by reducing the number of groups by integrating several groups into one group. As shown in FIG. 12C, the groups 1203 and 1204 in FIG. 12A are integrated at the portion indicated by the dotted line, and the groups 1207 and 1208 in FIG. Generate ~ 1206. The position of the dotted line (integrated reference line) is obtained as follows, for example.

  First, among the groups 1201 to 1208 in FIG. 12A, a group with a small number of captured images is specified. In the case of X = 8 and Y = 6, it is necessary to provide integrated reference lines at two locations. Therefore, two of the groups 1201 to 1208, the group with the smallest number of captured images and the group with the second smallest number, are included. Identify the group. In the case of FIG. 12A, the group with the smallest number of captured images is the group 1208, and the groups with the second smallest number are the groups 1203 and 1207. Here, the group 1203 and the group 1207 have the same number of captured images. However, since the group 1207 is adjacent to the group 1208 to be integrated, the group 1203 is selected in this case. However, as a result, each of the group 1203 and the group 1208 is to be integrated.

  First, a method for integrating the group 1203 will be described. Among the groups 1202 and 1204 adjacent to the group 1203, the group 1204 has a shorter imaging period. Therefore, the group 1203 is integrated with the group 1204 to generate the group 1203 in FIG.

  Since only the group 1207 is adjacent to the group 1208, the group 1208 is integrated with the group 1207, thereby generating the group 1206 in FIG.

  Various group integration methods can be used as the group integration method in this step, and the method is not limited to a specific group integration method.

<Step S1016>
The spread allocation unit 909 allocates each of the same number of groups as the number of spreads acquired by the input unit 908 to any spread. Which group is assigned to which spread can be in various forms. For example, a group having a past imaging date and time (an imaging date and time of an arbitrary captured image in the group) is assigned to the first spread. .

<Step S1017>
First, the image selection unit 910 selects one unselected spread from each spread, and selects a captured image to be displayed in the spread from among a group of captured images belonging to a group assigned to the selected spread. To do. In the case of FIG. 4, four captured images are selected. A process of selecting four captured images to be displayed in the spread among the captured image group belonging to the group assigned to the selected spread will be described with reference to FIG.

  In FIG. 15, “Start” represents the imaging date and time of the captured image most recently captured in the captured image group belonging to the group (group of interest) assigned to the selected spread. End represents the imaging date and time of the captured image recently captured in the captured image group belonging to the group of interest. However, (End-Start) represents the imaging period of the group of interest.

  First, a captured image to be arranged in the main photo slot is selected from the captured image group (FIG. 15A) belonging to the group of interest. As the captured image to be arranged in the main photo slot, the captured image having the highest score for the main photo slot is selected from the captured images belonging to the group of interest. A captured image selected as a captured image to be arranged in the main photo slot is represented as (1) in FIG.

  Next, picked-up images (three) to be arranged in the sub-photo slot are selected from picked-up image groups belonging to the group of interest. As described below, the image selection period is subdivided so that the image selection does not concentrate on a part of the imaging period.

  First, the first captured image to be arranged in the sub photo slot is selected. For this purpose, first, as shown in FIG. 15C, the imaging period is divided into two divided periods. Then, as shown in FIG. 15D, for the sub-photo slot in the group of captured images belonging to the remaining divided period (solid line) from which the divided period (dotted line) to which the captured image (1) belongs is omitted. The captured image with the highest score is selected as the captured image to be placed in the sub-photo slot. The picked-up image selected as the first picked-up image to be arranged in the sub-photo slot is shown as (2) in FIG.

  Next, the second captured image to be arranged in the sub photo slot is selected. For this purpose, first, as shown in FIG. 15E, the imaging period is divided into four divided periods by dividing each divided period into two. As shown in FIG. 15F, sub-photographs of a group of captured images belonging to the remaining divided period (solid line) from which the divided period (dotted line) to which the captured images (1) and (2) belong are excluded from the four divided periods. The captured image with the highest score for the slot is selected as the captured image to be placed in the sub-photo slot. A captured image selected as the second captured image to be arranged in the sub-photo slot is shown as (3) in FIG.

  Next, a third captured image to be arranged in the sub photo slot is selected. The captured image is selected from the divided period G to which the captured images (1) to (3) do not belong, among the divided periods obtained by dividing the captured image to be arranged in the sub-photo slot into the above four divided periods. Assume that no captured image belongs to the divided period G, as indicated by hatching in (G). In such a case, as shown in FIG. 15H, the imaging period is divided into eight divided periods. As shown in FIG. 15I, each divided period is divided into two to obtain eight divided periods. Of the eight divided periods, the divided period (dotted line) to which the captured images (1) to (3) belong and the score for the sub photograph slot out of the captured image group belonging to the remaining divided period (solid line) excluding the divided period G. The highest captured image is selected as the captured image to be placed in the sub-photo slot. The picked-up image selected as the third picked-up image arranged in the sub-photo slot is shown as (4) in FIG.

<Step S1018>
The image layout unit 912 selects a template based on the image selected in step S107 in order to determine the image layout. From the template group input by the input unit 911, a template to be applied to the target spread is selected based on the captured image selected for the target spread. In the following, in order to describe the processing in step S1018 with a specific example, it is assumed that the input unit 911 has input 16 templates (1-1) to (4-4) in FIG. In step S1017, it is assumed that one captured image is selected for the main photo slot and two captured images are selected for the sub photo slot. In this case, as shown in FIG. 16A, a template group having three photo slots is set as a candidate. Here, when the three picked-up images selected in step S1017 are arranged in order from the oldest picked-up date and time, the image is long (vertically oriented) 1606, horizontally long (landscape) as shown in FIG. Assume that the image 1607 is a vertically long (vertical) image 1605. Note that images 1606 and 1607 are captured images arranged in the sub photograph slot, and an image 1605 is a captured image disposed in the main photograph slot.

  Next, from the candidates, a template in which the position of the main slot in the template corresponds to the time-series position of the main slot image in the selected image group of the scene of interest is selected. In the present embodiment, it is assumed that an image with an older shooting date and time is laid out in the upper left of the template, and an image with a newer shooting date and time in the lower right. As shown in FIG. 16B, since the shooting date and time of the vertically long image 1605 arranged in the main photo slot is recent, the layout for arranging the image for the vertically long main photo slot having the latest shooting date and time is shown in the lower right. Any one of (3-1) to (3-4) remains as a candidate.

  The image with the oldest shooting date / time is the image 1606 for the vertically long sub-photo slot, and the layout in which the image for the vertically long sub-photo slot with the oldest shooting date / time is arranged in the upper left is (3-1), (3 -2) remains as a candidate.

  Also, the image with the second oldest shooting date / time is the image 1607 for the horizontally long sub photo slot, and the layout for arranging the image with the second oldest shooting date / time for the sub photo slot other than the upper left and lower right is (3-2) only.

  In this case, however, the layout of (3-2) is selected as a template to be applied to the target spread. Note that the method for selecting a template to be applied to the target spread is not limited to the above method, and other methods may be adopted.

<Step S1019>
If the flag value acquired by the input unit 913 is a value indicating that image correction is to be performed, the image correction unit 914 performs dodging correction on the captured image selected by the image selection unit 910 in step S1017. Correction processing such as red-eye correction and contrast correction is performed. Of course, the image correction unit 914 does not perform the correction process when the flag value acquired by the input unit 913 is a value indicating that image correction is not performed.

<Step S1020>
The output unit 915 arranges, in each photo slot on the template selected in step S1018, the picked-up image selected in step S1017 for the photo slot, which is resized to the size of the photo slot. Image ". Then, the output unit 915 outputs the generated “page spread image” as layout information. The output destination may be the display unit 205 or the HDD 202.

<Step S1021>
If all spreads are selected in step S1017, the process according to the flowchart of FIG. 10 ends. On the other hand, if an unselected spread remains, the process returns to step S1017.

  By performing the processing according to the flowchart of FIG. 10 described above, a photo album in which captured images are arranged in each spread photo slot can be created. Next, when a spread image in which four images selected by the image selection method shown in FIG. 15 are laid out is displayed in the display area 403 of the window 401, the user operates the operation unit 203. A case where an arbitrary photo slot is designated will be described.

  In FIG. 17A, as shown in FIG. 15I, the captured images (1) to (4) are selected for the main photo slot and the sub photo slot, and in this state, the captured image (3 ) Shows a case where the user designates by operating the operation unit 203. That is, in FIG. 17A, the captured image (3) is a candidate starting point image. When the captured image (3) becomes a candidate starting image, a captured image belonging to the neighborhood 1701 of the imaging date and time of the candidate starting image is selected as a candidate captured image. The vicinity range of the captured image (3) here corresponds to a divided period including the captured image (3) in FIG. That is, in the present embodiment, the division period including the starting image among the division periods when the last image for the spread is selected is referred to as a “neighboring range”. In FIG. 17A, the captured images (5) and (6) are selected as candidate captured images among the captured images belonging to the vicinity range 1701. Various methods can be used for selecting the captured images (5) and (6), and the method is not limited to a specific method. For example, two of the captured images (other than the captured image (3)) belonging to the vicinity range 1701 may be selected at random, or the two captured images farthest from the imaging date and time of the captured image (3) within the vicinity range 1701. You may choose. Further, even if a captured image in which a person different from the captured image (3) is captured or a captured image having a composition different from that of the captured image (3) is selected as a candidate captured image from the captured images belonging to the neighborhood range 1701. I do not care.

  Although an example in which two captured images are selected as candidate captured images is shown here, the number of captured images to be selected as candidate captured images is not limited to two and may be any number. Here, a case where four candidate captured images are displayed as shown in FIG. In the present embodiment, the upper two candidate captured images have a high score by the first evaluation method, and the lower two candidate captured images have a high score by the second evaluation method. Thereby, in this embodiment, an image with a high score for different purposes can be displayed as a candidate image.

  As described above, regardless of which selection method is adopted, the candidate captured image is selected from the same division period (captured image group having a relatively close imaging date and time) as the candidate starting image, so that the time series in the page is destroyed. Thus, the user can easily replace the images.

Next, a case will be described in which candidate captured images are switched by instructing the above-described front button and rear button. Assume that the captured images (5) and (6) are currently displayed as candidate captured images. At this time, when the user operates the operation unit 203 to instruct the front button, as shown in FIG. 17B, the captured image (2) that is the captured image before the captured image (3) in the above order is displayed. It becomes a candidate origin image. Then, a captured image belonging to the range 1702 (division period 1702) of the captured date and time of the captured image (2) is selected as a candidate captured image. In FIG. 17B, the captured images (7) and (8) are selected as candidate captured images among the captured images belonging to the neighborhood range 1702. Further, it is assumed that the user operates the operation unit 203 and designates a back button in a state where the captured images (5) and (6) are currently displayed as candidate captured images. At this time, as shown in FIG. 17B, the captured image (1), which is the captured image after the captured image (3) in the above order, becomes the candidate starting image. Then, a captured image belonging to the range 1703 (division period 1703) of the captured date and time of the captured image (1) is selected as a candidate captured image. In FIG. 17B, the captured images (9) and (10) are selected as candidate captured images from the captured images belonging to the neighborhood range 1703.
When an arbitrary image is replaced with another image by the above-described method, an image suitable as a candidate for the other image can be displayed.
In this embodiment, an image in the vicinity of the image currently assigned in the image slot is displayed as a candidate image. Accordingly, the user can easily select an image suitable for the slot when laying out the image. In the present embodiment, an image near the replacement target slot is displayed as a candidate image. Specifically, candidate images are selected and displayed from the same shooting period as the replacement target slot. As a result, the user can easily replace the images without breaking the time series in the page.
Furthermore, when selecting and displaying the next candidate for the replacement candidate image, the user can easily significantly collapse the time series in the page by selecting the candidate image from the vicinity of the previous slot in terms of time. The images can be replaced without any change.

[Second Embodiment]
In the following, differences from the first embodiment will be described mainly, and unless otherwise noted, the same as the first embodiment. In the first embodiment, as shown in FIG. 17B, when the front button is instructed when the captured images (5) and (6) are displayed as candidate captured images, the captured image (2) is selected as a candidate. A captured image that becomes a starting image and belongs to the vicinity range 1702 (corresponding to the imaging period 1702) is selected as a candidate captured image. In the present embodiment, as shown in FIG. 18, when the front button is designated in the state where the captured images (5) and (6) are displayed as candidate captured images, the area between the vicinity range 1701 and the vicinity range 1702 is displayed. A captured image belonging to the range 1801 is selected as a candidate captured image. In FIG. 18, among the captured images belonging to the range 1801, captured images (7 ′) and (8 ′) are selected as candidate captured images. On the other hand, when the back candidate button is pressed, the selection range of candidate images is set as the imaging period 1703 as in the first embodiment. This is because there is no image in the shooting period 1802, and when there is an image in the shooting period (c), an image is selected from the shooting period (c). That is, in the present embodiment, candidate images are selected and displayed from a shooting period that is one after the shooting period to which the image arranged in the replacement target slot belongs. Since a method for selecting candidate images from the shooting period is the same as that in the first embodiment, description thereof is omitted.

  Next, a process for displaying a list of image candidates that can be arranged in an area designated by the user among the areas 405a to 405d in a state where captured images are arranged in each of the areas 405a to 405d will be described with reference to FIG. It demonstrates according to a flowchart. That is, this embodiment is different from the first embodiment in that instead of performing the process according to the flowchart of FIG. 5, the process according to the flowchart of FIG. 6 is performed.

<Step S1901>
The CPU 204 accepts any one selection operation from the areas 405a to 405d.

<Step S1902>
The CPU 204 sets a range near the imaging date and time of the captured image being displayed in the replacement target slot as the extraction time range.

<Step S1903>
The CPU 204 searches for (lists up) the captured image candidates (candidate images) that can be placed in the replacement target slot from the captured images that belong to the extraction time range instead of the candidate starting point image.

<Step S1904>
The CPU 204 causes the display unit 205 to display a window including the candidate images listed in step S1903.

<Step S1905>
The CPU 204 determines whether or not the previous button 510 (previous candidate button) has been designated. As a result of this determination, if it is designated, the process proceeds to step S1906, and if it is not designated, the process proceeds to step S1907.

<Step S1906>
The CPU 204 sets a time range adjacent to the extraction time range and having a date and time in the past as a new extraction time range. Then, the process returns to step S1903.

<Step S1907>
The CPU 204 determines whether or not a back button 511 (back candidate button) has been designated. As a result of this determination, if it is designated, the process proceeds to step S1908, and if it is not designated, the process proceeds to step S1909.

<Step S1908>
The CPU 204 sets a time range adjacent to the extraction time range and having a new date and time as a new extraction time range. Then, the process returns to step S1903.

<Step S1909>
The CPU 204 determines whether or not the user has selected any one of the candidate captured images in the displayed window by operating the operation unit 203. As a result of the determination, if selected, the process proceeds to step S1910. If not selected, the process returns to step S1904.

<Step S1910>
The CPU 204 displays (arranges) candidate captured images selected by the user by operating the operation unit 203 in the replacement target slot, instead of the captured image currently displayed in the replacement target slot.
In this embodiment, an image in the vicinity of the image currently assigned in the image slot is displayed as a candidate image. Accordingly, the user can easily select an image suitable for the slot when laying out the image. In the present embodiment, an image near the replacement target slot is displayed as a candidate image. Specifically, candidate images are selected from the same shooting section as the replacement target slot and displayed. As a result, the user can easily replace the images without breaking the time series in the page. Furthermore, when selecting and displaying the next candidate for the replacement candidate image, the user can easily greatly collapse the time series in the page by selecting the candidate image from the previous shooting period that is temporally adjacent. The images can be replaced without any change.

[Third Embodiment]
In the first and second embodiments, as described above, the processing according to the flowcharts of FIGS. 5 and 6 is based on the premise that the photo album generated by the processing according to the flowchart of FIG. 10 is used. However, it is sufficient to divide the captured image group into the same number of spreads as the number of spreads based on the shooting date and time, and any photo album in which the group is assigned to each spread, and a photo album generated by other processing is used. It doesn't matter.

  Further, the processing according to the flowcharts of FIGS. 5 and 6 described in the first and second embodiments is to replace the captured images displayed as a list in the photo album with other captured images. However, even in the case where the captured image group is divided into a plurality of groups based on the captured date and time and the captured images belonging to one group are displayed, the processing according to the flowcharts of FIGS. Can respond. That is, in this case, a candidate for replacement of the displayed captured image is selected from the group to which the displayed captured image belongs and presented to the user.

  That is, the processing according to the flowcharts of FIGS. 5 and 6 is not limited to the purpose of creating a photo album and uploading it to the PC 2, but can be used for various situations and various purposes. Is.

[Fourth Embodiment]
Any device can be used as long as it is a device capable of realizing equivalent processing instead of PC1 and PC2. For example, a device such as a tablet terminal device or a smartphone may be used instead of the PC 1.

[Fifth Embodiment]
Each of the areas 405a to 405d is not limited to the one in which only the image is arranged, and may be one in which the text is arranged. In the above-described embodiment, the areas (areas 405a to 405d) for arranging images and characters are objects. However, a subject such as a person displayed on the image may be simply used as the object, and any object may be used as long as it can be displayed on the screen.
[Sixth Embodiment]
There may be a case where the next shooting period does not exist, as in the case where the captured image (4) in FIG. 17 or the captured image (4) in FIG. 18 is selected as an image to be replaced. In this case, the user designation may be suppressed, for example, the UI display for designating the next shooting period is not performed (the display of the back button 511 is not performed). In addition, if the user designation is not suppressed without suppressing the user designation, the candidate image may be displayed from the time zone on the Start side. Here, the case where the image of the last shooting period is the replacement target has been described as an example, but conversely, the same applies to the case where the image of the first shooting period is the replacement target.
[Seventh Embodiment]
Only one of the front button 510 and the rear button 511 may be displayed. Even when the next candidate image can be displayed only in the previous shooting period or the subsequent shooting period, the convenience of the user is enhanced.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  201: RAM 202: HDD 203: Operation unit 204: CPU 205: Display unit

Claims (7)

Acquisition means for acquiring an image data group including a plurality of image data;
Dividing means for dividing the plurality of image data into a plurality of groups;
Layout for displaying a layout image of the image based on predetermined image data among the image data group included in the predetermined group, was placed in a slot of the page corresponding to the given group of the plurality of groups on the display screen Display control means;
In a layout image of a page corresponding to the predetermined group displayed on the display screen, designation means for designating a replacement target slot for replacing an image;
Based on the shooting date and time of the first image arranged in the slot to be replaced specified by the specifying means, the image of the shooting date and time included in a predetermined time range including the shooting date and time of the first image is displayed. a first display control means for displaying the first candidate screen including a candidate image to replace the one image,
Candidate images to be replaced with the first image are images with shooting dates and times included in at least one of the time range before the predetermined time range and the time range after the predetermined time range. Second display control means for displaying a second candidate screen including:
Changing means for changing from the first candidate screen to the second candidate screen based on a user instruction;
Arrangement means for arranging a selection candidate image selected by the user from among the candidate images in the slot to be replaced ;
The predetermined time range and the at least one time range are included in the shooting period of the predetermined group,
The arrangement unit selects the selected candidate image when the candidate image included in the first candidate screen is selected by the user or when the candidate image included in the second candidate screen is selected by the user. an image processing apparatus characterized that you place the replacement target slot of the page corresponding to the given group. The at least one time range is a time period including a shooting date and time of a second image different from the first image among candidate images included in the second candidate screen , or the predetermined time range and the second image The image processing apparatus according to claim 1, wherein the time period is between a time period including a shooting date and time . The layout display control means includes
An image based on image data satisfying a predetermined standard among image data groups included in the predetermined group is arranged in a first slot in the layout image ,
Arranging image data other than the image data satisfying the specified standard in the image data group included in the predetermined group in a second slot having a size smaller than the first slot in the layout image. the image processing apparatus according to claim 1 or 2, characterized. An acquisition step of acquiring an image data group including a plurality of image data;
A dividing step of dividing the plurality of image data into a plurality of groups;
Layout for displaying a layout image of the image based on predetermined image data among the image data group included in the predetermined group, was placed in a slot of the page corresponding to the given group of the plurality of groups on the display screen A display control process;
In the layout image of the page corresponding to the predetermined group displayed on the display screen, a designation step for designating a replacement target slot for replacing the image;
Based on the shooting date and time of the first image arranged in the slot to be replaced specified in the specifying step, the image of the shooting date and time included in a predetermined time range including the shooting date and time of the first image is selected . a first display control step of displaying the first candidate screen including a candidate image to replace the one image,
Candidate images to be replaced with the first image are images with shooting dates and times included in at least one of the time range before the predetermined time range and the time range after the predetermined time range. A second display control step of displaying a second candidate screen including:
Based on a user instruction, a changing process for changing from the first candidate screen to the second candidate screen;
An arrangement step of arranging a selection candidate image selected by the user from among the candidate images in the slot to be replaced ,
The predetermined time range and the at least one time range are included in the shooting period of the predetermined group,
In the arrangement step, the selected candidate image is selected regardless of whether the candidate image included in the first candidate screen is selected by the user or the candidate image included in the second candidate screen is selected by the user. an image processing method characterized that you place the replacement target slot of the page corresponding to the given group. The at least one time range is a time period including a shooting date and time of a second image different from the first image among candidate images included in the second candidate screen, or the predetermined time range and the second image The image processing method according to claim 4, wherein the time period is between a time period including the shooting date and time. In the layout display control step,
An image based on image data satisfying a predetermined standard among image data groups included in the predetermined group is arranged in a first slot in the layout image,
Arranging image data other than the image data satisfying the specified standard in the image data group included in the predetermined group in a second slot having a size smaller than the first slot in the layout image. 6. The image processing method according to claim 4 or 5, characterized in that: The computer program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 3 .

Images