JP7286393B2 - Image processing device, image processing method, and program - Google Patents

Description

The technology of the present disclosure relates to an image processing device, an image processing method, and a program for creating an album.

In addition to the spread of digital cameras, the spread of smart devices and the improvement in camera performance of smart devices in recent years have led to a rapid increase in the number of photographs taken by users. As a method of collecting photographs, a photobook has been proposed in which photographs are digitally processed and arranged. Since it is processed digitally, it can be edited on a computer.

Patent Document 1 proposes a method of grouping images based on incidental information of the images, analyzing only the groups with high importance to evaluate the images, and creating an electronic album using the images with evaluation values above a certain level. .

JP 2015-114920 A

However, in Patent Document 1, if images are divided into groups, only important groups are evaluated, and high-scoring images are used, existing pages other than the added pages will be adjusted if the user manually adds pages later. It may not be possible. As a result, the layout result desired by the user may not be obtained.

The technology of the present disclosure includes display control means for displaying album data including an existing page in which a plurality of images are arranged, addition means for adding an additional page to the album data, and selection by the user for the additional page. layout means for laying out a plurality of image data including the image data added to the existing page; and a layout result of the plurality of image data on the additional page and a layout result of the plurality of image data on the existing page. The image processing apparatus includes a changing means for changing the arrangement of the image data.

According to the technique of the present disclosure, it is possible to obtain a layout result desired by the user.

1 is a block diagram of hardware capable of executing an application according to the present invention; FIG. 1 is a software block diagram of an application according to the present invention; FIG. FIG. 4 is a diagram of a display screen displayed by an application according to the present invention; FIG. 4 is a processing flow diagram of automatic layout according to the present invention; It is a figure explaining the image-analysis information which concerns on this invention. FIG. 4 is a diagram for explaining image group division according to the present invention; FIG. 4 is a diagram for explaining scene classification according to the present invention; It is a figure explaining the image score which concerns on this invention. FIG. 4 is a diagram for explaining image selection according to the present invention; It is a figure explaining the image layout which concerns on this invention. FIG. 5 is a diagram for explaining detailed image selection according to the present invention; FIG. 5 is a diagram illustrating a detailed flow of image selection according to the present invention; FIG. 5 is a diagram for explaining detailed image selection according to the present invention; FIG. 5 is a diagram illustrating a detailed flow of image selection according to the present invention; FIG. 4 is a diagram illustrating automatic layout of additional pages and existing pages according to the present invention; FIG. 4 is a diagram illustrating the flow of automatic layout of additional pages and existing pages according to the present invention; FIG. 4 is a diagram illustrating automatic layout of additional pages and existing pages according to the present invention; FIG. 4 is a diagram illustrating the flow of automatic layout of additional pages and existing pages according to the present invention; FIG. 4 is a diagram illustrating automatic layout of additional pages and existing pages according to the present invention; FIG. 4 is a diagram illustrating the flow of automatic layout of additional pages and existing pages according to the present invention; It is a figure explaining the specification method of the additional page which concerns on this invention. FIG. 10 is a diagram illustrating a method of specifying an image to be used for an additional page according to the present invention; It is a figure explaining the layout result based on this invention. It is a figure explaining the layout result based on this invention.

(Embodiment 1)
BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

In the present embodiment, an example of a method for automatically generating a layout by operating an application for creating an album (hereinafter also referred to as an "application") in an image processing apparatus will be described.

FIG. 1 is a block diagram for explaining the hardware configuration of an image processing apparatus according to the present invention. Examples of the image processing device include a PC, a smart phone, etc. In this embodiment, a PC is used.

1, image processing apparatus 100 includes CPU 101, ROM 102, RAM 103, storage device 104, display 105, keyboard 106, mouse 107, and data communication device . These are interconnected by a data bus 109 .

A CPU (central processing unit/processor) 101 is a system control unit and controls the entire image processing apparatus 100 . Further, the CPU 101 executes the image processing method described in this embodiment according to a program. In addition, although the number of CPUs in the figure is one, it is not limited to this, and a plurality of CPUs may be provided.

The ROM 102 stores programs executed by the CPU 101 . The RAM 103 provides memory for temporarily storing various information when the CPU 101 executes the program. A storage device (SSD, HDD, etc.) 104 is a storage medium for storing a database that holds image files and processing results such as image analysis. In this embodiment, the storage device 104 stores an application program for creating an album, the details of which will be described later.

The display 105 is a device that presents the UI (user interface) and image layout results of the present embodiment to the user. The display 105 may have a touch sensor function. The keyboard 106 is one of input devices, and is used to input arbitrary information on the UI displayed on the display 105, for example. In this embodiment, the number of double-page spreads of the album is input via the keyboard 106 . A mouse 107 is one of input devices, and is used, for example, to select or click buttons on the UI displayed on the display 105 .

A data communication device 108 is a device for communicating with an external device such as a printer or a server. For example, after the album data is generated by the process described later, the user can use the album creation application to instruct the printing of the album. As a result, the album data generated by the album creation application is transmitted to the printer or server connected to the PC via the data communication device 108 . A printer connected to the image processing apparatus 100 is used to print an album based on the album data. Also, the album data sent to the server may be printed using a printer connected to the server.

A data bus 109 connects each unit (102 to 108) and the CPU 101 described above.

FIG. 2 is a block diagram for explaining the software configuration of the album creation application according to the present invention. In this embodiment, the album creation application stored in the storage device 104 is activated when the user double-clicks the application icon displayed on the display 105 using the mouse 107 . The album creation application has various functions, but in this embodiment, the automatic layout function provided by the automatic layout processing section 219 will be described.

As shown in FIG. 2, the application includes an album creation condition setting section 201 , an automatic layout processing section 219 and an album display section 220 . The automatic layout function is a function for classifying or selecting photographed photographs based on their contents and attributes, laying out the photographs, and generating album data to be displayed on the display 105 .

The album creation condition setting unit 201 designates album creation conditions to the automatic layout processing unit 219 according to the user's UI operation using the mouse 107 or the like. In this embodiment, the following items can be set as album creation conditions. The album creation conditions that can be set include the main character, the number of spreads, the type of template, the album creation mode, whether or not to perform image correction, the number of spread photos to adjust the number of images placed per spread of the album, and the creation of the album. It is a product specification that specifies a product. The image/moving image can be specified by attribute information or associated information of individual image data such as the date and time of shooting, or based on the structure of the file system containing the image/moving image data, such as device or directory designation. may be A double-page spread corresponds to, for example, one display window in the case of display, and a pair of adjacent pages printed on different sheets in the case of printing.

The moving image acquisition unit 202 acquires the moving image group (moving image data group) specified by the album creation condition setting unit 201 from the storage device 104 . The moving image group here refers to a moving image data group saved in a folder designated by 303 in FIG. 3, which will be described later. The acquired moving image data is decompressed from a compressed state using a moving image decoder or the like that is standardly installed in the OS so that it can be analyzed.

The moving image analysis unit 203 analyzes the moving image data acquired by the moving image acquisition unit 202 . The moving image analysis unit 203 cuts out consecutive frames from the moving image, and calculates the amount of difference between the frame of interest and the frames before and after it. If the amount of difference is large, the frame of interest is not similar to the frames before and after it, so it is determined that the frame has a large amount of motion. If the amount of difference is small, it is determined that the series of frames are similar and the amount of motion is small. As for the amount of motion, a motion vector used in the moving image format may be used in addition to the difference between frames described above.

Then, the moving image analysis unit 203 selects frames to be analyzed from the determined amount of motion. Based on the amount of movement, it can be divided into a moving section and a stationary section. The interval between frames to be analyzed is shortened in a section determined to have a large amount of motion, and the interval of frames to be analyzed is lengthened in a section determined to have a small amount of motion. For example, a section with a large amount of motion is analyzed at intervals of 5 frames, and a section with a small amount of motion is analyzed with the first, last, and intermediate frames of the section.

Furthermore, the moving image analysis unit 203 performs analysis processing on the frame that is the analysis target. Items to be analyzed include detection of objects used in layouts such as faces and animals, determination of object sizes, determination of smiles, closed eyes, blurred blur, and brightness of the entire image. Then, the score of the frame is calculated according to the result of each determination item as follows, and scored.

Here, object detection performed by the moving image analysis unit 203 will be described. For example, if an object corresponding to the main character set in the album creation condition (that is, the face of the main character) is included in the frame, the moving image analysis unit 203 gives a higher score than an image that does not include the object corresponding to the main character. set. Next, size determination performed by the moving image analysis unit 203 will be described. The moving image analysis unit 203 gives a high score if the size of the object detected in the object size determination is large. In addition, the moving image analysis unit 203 sets a high score to a frame having a large number of faces determined to be smiling in smile determination and a large number of faces determined not to close their eyes in eyes closed determination. In addition, the moving image analysis unit 203 calculates the edge amount of the image using a Laplacian filter in blurring/bokeh determination, and sets a higher score for a frame with a larger edge amount. In addition, the moving image analysis unit 203 sets a high score to a frame in which the average brightness of the image is equal to or higher than a predetermined value with respect to the brightness of the entire image.

The moving image analysis unit 203 finally calculates a total score, which is the sum of the scores for each item. In addition, the region for determining blurring may be changed according to the determined amount of motion. For example, when a moving subject is tracked during video shooting, the subject is clearly captured, but the background is blurred, resulting in a blurry image as a whole. Therefore, when there is a large amount of motion, a method of determining the amount of blur using only a subject area such as a face is also conceivable.

In this way, the moving image analysis unit 203 determines a frame having a total score equal to or higher than a predetermined value calculated from each divided moving/still section as the best shot.

The frame acquisition unit 204 cuts out a frame determined to be the best shot from the moving image based on the analysis result of the moving image analysis unit 203, and stores it in the storage device 104 as image data. At that time, the motion amount, camera work, and score determined by the moving image analysis unit are stored in association with the clipped image data. The frame acquisition unit 204 may store the image data in association with the use, such as an image to be used in the album based on the determined score, or a candidate image for the user to use as a replacement after the album is created.

The image acquisition unit 205 acquires the image group (image data group) specified by the album creation condition setting unit 201 from the storage device 104 . The group of images here refers to a group of candidate images laid out when creating one album. For example, if Jan. 1, XX to Dec. 31, XX are designated as shooting dates in the album creation condition setting unit 201, images shot from Jan. 1, XX to Dec. 31, XX All correspond to the designated image group. The image acquisition unit 205 also acquires width and height information of each image in the designated image group.

The images stored in the storage device 104 include still images and clipped images clipped from moving images. A still image or a clipped image is obtained from an imaging device such as a digital camera or a smart device. The imaging device may be included in the image processing apparatus 100 or may be included in an external device. Note that if the imaging device is an external device, the image is acquired via the data communication device 108 . Still images and cutout images may be images obtained from a server, storage, or the like on a network via the data communication device 108 . Images obtained from servers on a network include images provided via social networking services (hereinafter referred to as “SNS images”). A program executed by the CPU 101 analyzes data attached to each image to determine the storage source. The SNS image may manage the acquisition destination within the application so that the image can be acquired from the SNS via the application. Note that the images stored in the storage device 104 are not limited to the images described above, and may be images of other types.

An image conversion unit 206 converts the number of pixels and color information of image data stored in the storage device 104 . In this embodiment, the image data is converted into image data having 420 pixels on the short side and color information of sRGB.

The image analysis unit 207 analyzes image data. In this embodiment, the image conversion unit 206 analyzes the converted image data. Specifically, a feature amount is acquired from the image data, and object determination, face detection, expression recognition of the detected face, and individual recognition of the detected face are performed in the image data. Further, the shooting date and time is acquired from data (eg, Exif information) attached to the image acquired from the storage device 104 . Analysis information composed of the obtained information may be saved as a binary file for each image data. Analysis time can be shortened by using the binary file of the analysis result when performing layout processing.

The image classification unit 208 divides (image group division) the obtained image group, that is, the image group designated by the album creation condition setting unit 201 into sub-image groups (sub-image data groups). Image group division is performed using image number information, shooting date/time information included in analysis information acquired from the image analysis unit 207, and detected face information. A scene is a predetermined shooting scene such as travel, daily life, wedding ceremony, or the like. A scene can also be said to be, for example, a collection of images of a single subject taken at one shooting opportunity for a certain period of time.

The main character information input unit 209 inputs the personal ID (identification information) registered in the face dictionary database of the main character specified by the album creation condition setting unit 201 to the image score unit 210 .

The image scorer 210 scores each image. In this embodiment, each image is scored so that an image suitable for the layout has a high score. Image data analysis information, image classification results, main character information, image information, and album creation conditions are used when scoring images so that images suitable for layout receive high scores. Also, other information may be used additionally or alternatively. The calculated image score result may be saved as a binary file for each image. Note that the image scoring unit 210 scores the image data cut out from the moving image, taking into consideration the analysis result of the moving image analyzing unit 203, as will be described later.

The page spread number input unit 211 inputs the number of page spreads of the album specified by the album creation condition setting unit 201 to the page spread assignment unit 212 .

The double-page spread allocation unit 212 divides the image group and allocates it to each double-page spread. The two-page spread referred to here is, for example, a two-page spread when an album is created. A double-page spread is used here, but a single page may also be used. The double-page spread allocation unit 212 divides the image group according to the input number of double-page spreads, and allocates part of the image group to each double-page spread. For example, when the number of spreads is 5, the acquired image group is divided into 5 sub-image groups, and one sub-image group is assigned to each spread.

The image selection unit 213 sequentially selects the images with the highest score from the group of sub-images assigned to each spread by the spread assignment unit 212 based on the scores given by the image scoring unit 210 .

The template setting unit 214 reads a plurality of templates from the storage device 104 according to the template information specified by the album creation condition setting unit 201 and inputs them to the image layout unit 215 .

The image layout unit 215 determines the layout of an image spread or page, and outputs layout information. Specifically, image layout section 215 selects a template suitable for the image selected by image selection section 213 from a plurality of templates input by template setting section 214 . Then, the image layout unit 215 creates layout information (also called album data) for each page or spread by determining the layout positions of the images in the selected template for each page or spread.

The layout information output unit 218 outputs the layout information of the entire album by arranging the page or spread layout information created by the image layout unit 215 in a predetermined order. The layout information is, for example, bitmap data in which the selected image is laid out in the selected template. Alternatively, the information may be information indicating which image is arranged at which coordinate of the template.

An image correction unit 217 performs dodging correction, red-eye correction, and contrast correction. The image correction condition input unit 216 inputs to the image correction unit 217 ON/OFF conditions for image correction specified by the album creation condition setting unit 201 . ON or OFF of image correction may be designated for each type of correction, or may be collectively designated for all types. When the image correction condition is ON, the image correction unit 217 performs the ON correction on the image. When the image correction condition is OFF, the OFF correction is not performed. The number of pixels of the image input from the image conversion unit 206 to the image correction unit 217 can be changed according to the layout image size determined by the image layout unit 215 . In the present embodiment, image correction is performed on each image after generating a layout image. However, the present invention is not limited to this, and correction of each image may be performed before layout (arrangement) on a double-page spread or page. may

The album display section 220 performs display control using the created album data based on the layout information output by the layout information output section 218 . As a result, an image of album data is displayed on display 105 .

An album creation application according to the present embodiment is installed in the image processing apparatus 100 , and when activated by a user operation, the album creation application program stored in the storage device 104 is loaded into the RAM 103 . A program in RAM 103 is executed by CPU 101 to start an album creation application.

FIG. 3 shows a display screen 301 provided by the activated album creation application, which is displayed on the display 105. As shown in FIG. The user can set album creation conditions via the display screen 301 . The user can set album creation conditions, which will be described later, via the UI screen on the display screen 301 .

The display screen 301 has a pass box 302 and a folder selection button 303 as input image/moving image group designation units. The user inputs in the path box 302 the storage location (path) of the folder containing the group of images and moving images that are layout candidates for the album to be created. The user can also select the folder from the tree structure displayed by clicking the folder selection button 303 with the mouse 107 . When the user selects a folder containing an image/moving image group from the folder selection button 303 , the selected folder path may be displayed in the path box 302 .

A plurality of face images of persons are displayed as icons in the hero specifying portion 304, and the user can specify the hero by selecting an icon. A face image is displayed for each person in the main character designation portion 304 . In other words, one face image is displayed for each person in the main character designation section 304, and each face image is associated with an individual ID.

The main character specifying section 304 is used to specify the main character, who is the main character, from the person appearing in the analysis target image, that is, the photograph. A face image is detected in advance from an image, for example, and registered in a face dictionary database in association with a personal ID. The icons displayed in the hero designation section 304 are, for example, the face images of persons selected by the user from among the face images of persons registered in the face dictionary database, or the face images of persons determined by a method described later. A plurality of icons of face images of different persons are arranged in the main character designation portion 304 and can be selected by clicking with the mouse 107 . The hero can be automatically set by the procedure shown in FIG. The person whose icon is selected by the user from among the icons displayed in the hero designation portion 304 is designated as the hero.

A double-page spread box 305 is a box for designating the number of double-page spreads in the album. The user can use the keyboard 106 to directly enter a number into the spread number box 305 or use the mouse 107 to enter a number from a list into the spread number box 305 .

A template designating section 306 displays a plurality of illustration images as icons showing the taste of the template (pop style, chic style, etc.), and can designate a template. In this embodiment, a template has an image layout frame (slot) for arranging an image. In the template designation section 306, icons of a plurality of templates are arranged, and a template can be selected by clicking with the mouse 107. FIG.

A mode designation portion 307 displays a plurality of illustration images as icons showing objects to be considered when creating an album, and can designate objects. In the present embodiment, three objects, ie, "people", "animals", and "cooking" will be described as object examples. Other objects may be selected from three or more objects. Also, it is possible to designate a plurality of objects at the same time, but here, for the sake of simplicity of explanation, the case where one object is selected will be explained. An object to be considered can be specified by clicking an icon provided for each object in the mode specifying section 307 with the mouse 107 .

A check box 308 can specify ON/OFF of image correction. When a check is put in with the mouse 107, the image correction described above is turned on, and when a check is not put in, the image correction is turned off. In the present embodiment, all image corrections are turned ON/OFF by one button, but the present invention is not limited to this, and a check box may be provided for each type of image correction.

A double-page spread photo count designation bar 309 allows the user to set the number of images to be arranged in each double-page spread of the album to be created using a slider bar. When the user sets the slider bar to the "many" side, the number of images arranged in each spread is increased. Also, when the user sets the slider bar to the "small" side, the number of images arranged in each spread is reduced.

The merchandise designation unit 310 sets the merchandise of the album to be created. The size of the album and the paper type of the album can be set for the product. The cover type and binding type may be set individually.

The OK button 311 is a button for determining the conditions selected on the display screen 301 as album creation conditions. When the user selects the OK button 311 by clicking it with the mouse 107 , the album creation conditions are determined and transmitted to the automatic layout processing section 219 via the album creation condition setting section 201 . Specifically, the path input in the path box 302 is transmitted to the image acquisition unit 205 . A personal ID corresponding to an icon selected from a plurality of icons included in hero designation section 304 is transmitted to hero information input section 209 . The number of spreads entered in the number of spreads box 305 is transmitted to the number of spreads input unit 211 . The template information of the icon selected from the plurality of icons included in template specifying section 306 is transmitted to template setting section 214 . ON/OFF of image correction in the image correction check box is transmitted to the image correction condition input unit 216 .

A reset button 312 is a button for resetting each setting information on the display screen.

FIG. 4 is a processing flowchart for executing automatic layout of the album creating application according to this embodiment. The flowchart shown in FIG. 4 is realized, for example, by the CPU 101 reading a program stored in the storage device 104 to the ROM 102 or the RAM 103 and executing the program. The processing flow of automatic layout will be described below with reference to FIG. As described below, in this embodiment, when creating an album, a group of images for creating an album is divided into sub-image groups according to shooting time, and each divided sub-image group (hereinafter referred to as a “section”) is divided into sub-image groups. ) to be placed on a page or spread.

In S401, the album creation condition setting unit 201 sets album creation conditions. Here, information set regarding a moving image/image group for creating an album, a main character, the number of double-page spreads, a template, a mode, and image correction is set.

In S402, the moving image acquisition unit 202 acquires moving image data from the designated folder.

In S403, the moving image analysis unit 203 analyzes the acquired moving image data. The moving image analysis unit 203 calculates a total score value by combining the results of motion amount, camerawork, blurring, facial expression, and the like. Furthermore, the moving image analysis unit 203 may be divided into automatic layout candidates and editing candidates that the user replaces during editing work based on the calculated total score value. For example, if the calculated total score value is equal to or greater than a predetermined threshold value, it may be used as an automatic layout candidate.

In S404, the frame acquisition unit 204 cuts out frames having a total score value equal to or greater than a predetermined threshold value from the analyzed moving image data, and stores the frames as image data in the storage device 104. FIG. The analysis results (movement amount, camerawork, total score value) corresponding to the saved frames may be associated and saved in the storage device 104 .

In S405, the moving image analysis unit 203 determines whether or not S402 to S404 have been completed for all moving images in the moving image group in the storage device 104 specified by the album creation condition setting unit 201. FIG. If not finished (No in S405), the process returns to S402. If completed (Yes in S405), the process proceeds to S406.

In S<b>406 , the image acquisition unit 205 reads images included in the specified image group from the storage device 104 and develops them in the RAM 103 .

In S<b>407 , the image conversion unit 206 generates analysis image data from the image data developed in the RAM 103 . The image conversion unit 206 acquires from the storage device 104 an image group corresponding to the conditions specified by the album creation condition setting unit 201, and converts each image of the acquired image group into desired pixel count and color information. Note that the number of pixels and color information after conversion are determined in advance and stored in a program or a parameter file used by the program. In this embodiment, input image data is converted into analysis image data having a size of 420 pixels on the short side and an sRGB color space.

In S408, the image analysis unit 207 acquires information associated with the image data and the image quality feature amount of the analysis image data. The image analysis unit 207 acquires the shooting date and time from information attached to each image data included in the image group acquired from the storage device 104 . In this embodiment, the shooting date and time is acquired from the Exif information attached to each image data. Further, the image analysis unit 207 acquires the feature amount of image quality from the analysis image data generated in S407. For example, the feature amount of image quality includes focus. A known Sobel filter can be used as an edge detection method. An edge of an image is detected by a Sobel filter, and the gradient of the edge, that is, the luminance gradient is calculated by dividing the luminance difference between the start point and the end point of the edge by the distance between the start point and the end point. An average slope of edges in the image is calculated, and an image with a large average slope is determined to be more in focus than an image with a small average slope. In this embodiment, a plurality of tilt thresholds are set, and it is determined whether or not the focus amount is acceptable by determining which tilt threshold is greater than or equal to. In this embodiment, two different tilt thresholds are set, and the focus amount is determined in three stages of ◯Δ×. If the average slope of the edges in the image is greater than or equal to the first threshold, the tilt of the focus is ◯ (preferred). ), and if it is less than the second threshold, the tilt of focus is set to x (unacceptable).

In S409, the image analysis unit 207 executes object detection and recognition in each analysis image data. A face is detected from the analysis image data generated in S407. A known method can be used for face detection. As a known method, for example, there is Adaboost, which creates a strong classifier from a plurality of prepared weak classifiers. In this embodiment, the image analysis unit 207 detects the face of a person (object) using a strong classifier created by Adaboost. In S409, the image analysis unit 207 detects (extracts) the face image and obtains the upper left coordinate value and the lower right coordinate value of the position of the detected face image in the image. Note that the “image” referred to here is an image represented by the analysis image data generated in S407.
By specifying these two types of coordinates, the position and size of the person's face can be specified. Furthermore, the image analysis unit 207 also performs AdaBoost for detecting animals such as dogs and cats and food, as well as face detection. As a result, the image analysis unit 207 can detect objects such as people, animals, and food, and at the same time classify and recognize what the objects are. In addition, flowers, buildings, ornaments, etc. may be recognized without being limited to the above. Although object classification using AdaBoost has been described here, image recognition using a deep neural network may also be used. The image analysis unit 207 creates a face dictionary database by assigning personal IDs to the face images thus recognized.

In S410, the image analysis unit 207 performs individual recognition. First, the similarity between the face image extracted in S409 and the representative face image stored for each individual ID in the face dictionary database is compared. Then, the personal ID whose similarity is equal to or higher than the threshold and which has the highest similarity is determined as the personal ID of the extracted face image. That is, the image analysis unit 207 identifies the person whose similarity is equal to or greater than the threshold and who corresponds to the personal ID with the highest similarity as the person of the extracted face image. If the similarity is less than the threshold, a new personal ID is assigned to the extracted face image as a new person and registered in the face dictionary database.

The image analysis information of each analysis image data acquired in steps S408 to S410 is stored in the storage device 104 by distinguishing the IDs for identifying images, as shown in FIG. The image analysis unit 207 saves the photographing date/time information and focus determination result acquired in S408, and the number and position information of faces detected in S409 as image analysis information of each analysis image data. An image based on one piece of analysis image data may include a plurality of face images, but the position information of the face is stored separately for each individual ID acquired in S410. Note that if no face is detected in the image data, the shooting date/time information and the focus determination result acquired in S408 are saved. Further, the image analysis unit 207 stores the main objects included in the image in descending order of reliability when Adaboost is executed. Here, if no object is detected, no information about the object is stored. Objects other than faces are stored in the same way. AdaBoost continuously connects weak classifiers that discriminate patterns to form one strong classifier. Therefore, the reliability increases as the number of weak classifiers that match patterns learned in advance by Adaboost increases. Here, object determination has been described using AdaBoost as an example, but a discriminator using a Convolutional Neural Network such as a Deep Neural Network may also be used.

In S411, the image analysis unit 207 determines whether or not S407 to S410 have been completed for all images in the image group of the storage device 104 for which the album creation condition setting unit 201 has been designated. If not finished (No in S411), the process returns to S406. If completed (Yes in S411), the process proceeds to S412.

In S412, the image classification unit 208 performs image group division on the analysis image group. Image group division refers to dividing into a plurality of sub-image groups by grouping images having close photographing dates and times in the obtained analysis image group as a sub-image group. Specifically, the analysis image group is divided into a plurality of sub-image groups based on the photographing date and time information already acquired in S408, relying on the time difference between the analysis images. Examples of division criteria are as follows. In the analysis image group, for example, the image data with the oldest (or newest) shooting date and time is first focused on, and the time difference with the next oldest (newest) image data is referenced. While sequentially replacing the images of interest with those having newer (or older) shooting dates and times, it is determined which of the following conditions is met. Note that dividing in the following description means dividing two image data into a group of sub-images with a new shooting date and a group of sub-images with an old shooting date on the boundary between two pieces of image data.

First, if the shooting dates are not consecutive, that is, if there are days on which no shooting is performed between the images, the images are divided there. Next, a case where shooting dates are consecutive will be described. In this embodiment, when the difference between the photographing times of the images is 16 hours or more, the images are divided at that point. If the difference between the capture times of images is less than 16 hours, split by capture date if the time difference from the first capture to the last capture on each consecutive day is less than 4 hours. In the present embodiment, if the number of shots for each consecutive day is less than 50 for four hours or longer, it is divided by shooting day, and if it is 50 or more, it is not split. Note that the threshold for the time difference and the threshold for the number of sheets in division are not limited to these. FIG. 6A shows the result of division by the image group division method described above, and is divided into eight sub-image groups.

In S413, the image classification unit 208 performs scene classification. In the present embodiment, an example of classifying each sub-image group into three scenes of travel, daily life, and ceremony will be described, but the scenes to be classified are not limited to these.

First, the image classification unit 208 acquires a plurality of sub-image groups for which it has been determined in advance which of the travel, daily life, and ceremony scenes the images should be classified into. Then, the image classification unit 208 acquires the photographing feature amount for the sub-image group of each scene. The feature values acquired in this embodiment are, for example, the shooting period, the number of shots, and the number of shots. The shooting period is the time difference from the first shooting to the last shooting of the sub-image group. The number of shots is the number of images (the number of photographs) included in the sub-image group. The number of people photographed is the number of faces in images containing faces. That is, it is the number of faces included in one image. Then, the average value and standard deviation of the shooting period, the average value and standard deviation of the number of shots, and the average value and standard deviation of the number of people per image are obtained for the group of sub-images of each scene. Note that in the present embodiment, the number of faces per image is the number of people per image. FIG. 7 shows an example of average values and standard deviations of the shooting period (time), the number of shots (pictures), and the number of shots (persons). These obtained values are incorporated in the album creation application program in advance. That is, a group of sample images (teacher images) collected in advance is used to create parameters that characterize the scene through learning at the design stage, and the parameters are incorporated in the program in advance or stored in a state that the program can refer to. back.

After the album creation application is started, the image group specified by the user in the pass box 302 is acquired by the image acquiring unit 205 in S406, and the acquired image group is converted into the analysis image by the image converting unit 206 in S407. The analysis image group is divided into sub-image groups by the image classification unit 208 in S412. Then, the image classification unit 208 calculates a feature amount that is the average value of the shooting period, the number of shots, and the number of shots of each sub-image group. Then, the image classification unit 208 calculates the score of each scene of each sub-image group using the following formula based on the calculated feature amount of the sub-image group and the average value and standard deviation for each scene and feature amount obtained in advance. Ask for

Score for each scene and feature amount=50−|10×(Average value for each scene and feature amount−Feature amount for sub-image group)/Standard deviation for each scene and feature amount|Equation (1)

Average score for each scene = (Score for the shooting period for each scene + Score for the number of shots for each scene + Score for the number of shots for each scene)/feature item formula (2)

Note that the number of feature amount items in Equation (2) is 3 in this case. A score for each scene of the group of sub-images of interest is obtained by these equations (1) and (2). For example, the average value and standard deviation shown in FIG. 7 and the formula (1) can be used to obtain scores for the shooting period, the number of shots, and the number of shots of the travel scene. That is, the score for the travel scene shooting period is 50−|10×(33.221−sub image group shooting period)/4.778|. The score for the number of shots of the travel scene is 50−|10×(3.336−the number of shots of the sub-image group)/4.671|. The score for the number of people who took the travel scene is 50−|10×(4.634−the number of people who took the sub-image group)/1.532|. The score for each feature amount of the travel scene obtained by the formula (1) is averaged by the formula (2) to obtain the score for the travel scene. Similarly, scores can be obtained for other scenes such as daily life and ceremonies.

As described above, the average score for each scene of travel, daily life, and ceremony is calculated for each sub-image group divided in S412. In this manner, the image classification unit 208 classifies the sub-image group into the scene with the highest score among the calculated average scores for each scene of each sub-image group. If two or more scenes are tied, the scenes are sorted according to a predetermined scene priority. For example, in the present embodiment, it is predetermined that priority is given in the order of daily life>ceremony>travel, and the highest priority is given to daily life. Note that the order of priority is not limited to this, and the order of priority may be set by the user.

In the sub-image group (5) after scene division in FIG. 6A, the shooting period was 36 hours, the number of shots was 300, and the number of shots was 1.7. The average score for travel is 45.32, the average score for daily life is 18.38, and the average score for ceremony is -29.92, and the scene of sub-image group (5) is classified as travel. The sub-image group is managed by associating it with a scene ID so that the scene can be identified.

In S414, the image classification unit 208 determines whether or not the scene classification in S413 has been completed for all sub-image groups divided in S412. If not finished (No in S414), the process returns to S413. If completed (Yes in S414), the process proceeds to S415.

In S415, the image scoring unit 210 sets the hero. The main character setting is performed on a group of images specified by the user, and there are two types of setting: automatic and manual. When setting the main character automatically, the image scoring unit 210 sets based on the result of personal recognition in S410 and the result of image group division in S412. From the results obtained in S410, the number of appearances of each individual ID appearing in the sub-image group, the number of appearances of each individual ID appearing in each scene, the number of appearances of scenes in which each individual ID appears, etc. are specified, and these information are obtained. Automatically set the hero from. In this embodiment, when sub-image groups are classified into a plurality of scenes, a person ID appearing in a plurality of scenes is set as the main character ID, and when all sub-image groups are classified into a single scene, all sub-images are classified into a single scene. The personal ID that appears the most in the image group is set as the main character ID. When the user designates a specific icon in the hero designation section 304 , the personal ID corresponding to the icon selected via the hero information input section 209 is transmitted to the image scoring section 210 . If there is a user-designated personal ID, the user-designated personal ID is set as the main character ID, ignoring the above-described automatically set main character ID. In this way, the setting based on the user's designation by the main character designation unit 304 is the manual setting.

In S416, the image scoring unit 210 performs scoring. Scoring is a value that is referred to when selecting an image, which will be described later, and a score is given to each image data based on a viewpoint that will be described later. Furthermore, the image data clipped from the moving image is scored in consideration of the result of analysis by the moving image analysis unit 203 . Here, the scoring method will be described with reference to FIGS. 8 and 10. FIG.

FIG. 10 shows a template group for arranging three images. A template 1001 shown in (1-1) of FIG. 10 is one template and has a main slot 1002, a sub-slot 1003, and a sub-slot 1004. FIG. A main slot 1002 is a main slot (position where an image is arranged) in the template 1001 and is larger in size than the sub-slots 1003 and 1004 . That is, the main slot is the slot with the largest size in each template. A sub-slot is a slot other than the main slot in each template. In this embodiment, both the score for the main slot and the score for the sub-slot are set for each piece of image data.

Next, FIG. 8A shows criteria for scoring image data. More specifically, FIG. 8(a) is a table summarizing the features of images with high scores for travel, daily life, and ceremony scenes. Note that, as shown in FIG. 8(a), the feature of increasing the score differs depending on the slot type. For example, if the scene classification of the sub-image group is travel, the score for the main slot is high for the close-up image containing people and scenery, and the score for the close-up and profile images is high as the sub-slot. Become. On the other hand, when the scene classification of the sub-image group is everyday, close-up and side-face images score high as the main slot, and images with portraits and landscapes score high as the sub-slot. Become. Further, when the scene classification of the sub-image group is Ceremony, an image in which two people are close to each other has a high score as the main slot, and an image with a large number of people has a high score as a sub-slot. In this way, the features that give a higher score, in other words, the evaluation criteria, differ depending on the type of scene and slot. In this embodiment, the features or evaluation criteria that increase the score according to the type of scene and slot are set in advance in the application and included in the program.

Note that the features that increase the score are not limited to those described above. Each image data included in each sub-image group is scored based on the respective evaluation criteria for the main slot and sub-slot of each scene shown in FIG. 8(a).

Each feature amount, such as the number of faces in each image of a sample image group collected in advance, the position of the face in each image, and the size of the face in each image, is acquired. Then, the average value and standard deviation for each feature amount are obtained for each scene and each slot type (main slot and sub-slot), and incorporated in the album creation application program in advance.

After the scene classification of the image group designated by the user is completed, each feature amount of the number of faces of the main character ID, the position of the face, and the face size is calculated for each image of the image group. Also, the scene to which each image belongs (classified) is acquired from the scene classification result of S413. Then, based on the number of faces, the position of the face, and the size of the face of the main character ID in the image of interest, and the average value and standard deviation of each feature value for each scene and slot type obtained in advance, the following formula is used: A score is obtained for each scene of the image of interest and each slot type. In the following formulas, the score, average value, and standard deviation are obtained for each scene classification, slot type, and feature amount.

Score of each feature amount for each scene and slot type=50−|10×(average value of each feature amount for each scene and slot type−each feature amount for target image)/standard for each feature amount for each scene and slot type Deviation | Equation (3)

Average score for each scene and slot type = (Score for number of faces for each scene and slot type + Score for face position for each scene and slot type + Score for face size for each scene and slot type) / Feature item formula ( 4)

Note that the number of feature quantity items in Equation (4) is 3 in this case. In this way, the main slot score and the sub-slot score are given as scores for each image.

Furthermore, the calculated score is added to the score based on the mode designated by the album creation condition setting section 201 . When the mode is set to person, if the object classification in the image analysis information of FIG. The higher the reliability of the object classification result, the higher the score to be added. Besides, the score to be added may be changed according to the ranking of the object classification results. Here, for the sake of explanation, a case where a person is selected in the mode has been explained, but points are added under similar conditions when an animal or food is set in the mode. Also, the number of modes to be selected may be not only one, but two or more may be selected. In that case, it is sufficient to add the score by the amount that the selected object is included.

Further, a score may be added to the score of the image of the image ID whose focus feature amount is ◯ included in the image analysis information of FIG. 5 . This makes it possible to increase the score of an image that is in focus.

FIG. 8(b) shows an example of the score result of each image, in which each image ID is scored for the main slot and sub-slots. After both the main slot and sub-slot scores are set for each image, if the type of image included in the image analysis information in FIG. is added to both the main slot and subslot scores. Although the method of using the score value calculated by the moving image analysis unit 203 has been described here, the score value to be added may be increased according to the amount of motion. Alternatively, a predetermined score value may be added to a frame after zooming by camerawork or a frame after panning or tilting. By doing so, it is possible to reflect the intention of the user at the time of video shooting.

An image clipped from a moving image may have a higher score than a still image only by adding the score value at the time of shooting the moving image. In that case, the average value of the score values of the clipped images calculated by the moving image analysis unit 203 may be calculated and added to the score values of all still images or subtracted from the score values of the clipped images.

In S417, the image scoring unit 210 determines whether the image scoring in S416 has been completed for all images in the user-specified image group. If not finished (No in S417), the process returns to S416. If completed (Yes in S417), the process proceeds to S418.

In S<b>418 , the spread allocation unit 212 determines whether the number of sub-image groups obtained by the image group division of the image classification unit 208 is the same as the number of spreads input by the spread number input unit 211 . If not the same (No in S418), the process proceeds to S419. If they are the same (Yes in S418), the process proceeds to S422. For example, if the number of sub-image groups in FIG. 6A is 8 and the input number of the spread number input unit 211 is 8, S418 returns Yes, and the process proceeds to S422.

In step S<b>419 , the spread allocation unit 212 determines whether the number of sub-image groups obtained by the image group division of the image classification unit 208 is less than the number of spreads input by the spread number input unit 211 . If the number of sub-image groups is greater than or equal to the number of spreads (No in S419), the process proceeds to S421. If the number of sub-image groups is less than the number of spreads (Yes in S419), the process proceeds to S420. For example, if the number of divided sub-image groups in FIG. 6A is 8 and the number of input to the spread number input unit 211 is 10, the number of sub-image groups is equal to or greater than the number of spreads, so S419 is No, and S421. transition to

In S420, the spread allocation unit 212 performs sub-image group re-division. Sub-image group re-division refers to sub-dividing the divided sub-image group when the number of sub-image groups<the number of spreads.

A case in which the number of designated spreads is 10 for the number of divided sub-image groups of 8 in FIG. 6A will be described as an example. FIG. 6B shows the result of re-dividing a part of the sub-image group of FIG. 6A into sub-image groups and setting the number of sub-image groups to ten. The number of sub-image groups is set to 10 by further dividing the sub-image groups at the locations indicated by the dashed arrows in FIG. 6B. In FIG. 6B, there are dashed arrows between sub-image groups (2) and (3) and between (6) and (7). The criteria for determining the sub-image group to be divided are the number of images in the sub-image group and the shooting date and time. First, a sub-image group having a large number of images is specified among the sub-image groups in FIG. 6(a). Here, since it is desired to increase the number of sub-image groups from 8 to 10 by 2, two sub-image groups having a large number of images are specified. That is, the number of sub-image groups to be re-divided is designated in order from the sub-image group containing the largest number of images. For sub-image groups having the same number of images, the sub-image group having a larger maximum value of the difference in photographing date and time between the images is selected. If it still cannot be determined, it may be determined as appropriate, for example, by preferentially re-dividing the group of sub-images that are earlier in time.

In FIG. 6A, the sub-image group having the largest number of images is sub-image group (5), followed by sub-image group (1) and sub-image group (2). In this embodiment, when the number of images included in the sub-image group is the same, the sub-image group having the larger maximum time difference between the images included in the sub-image group is divided. Sub-image group (1) and sub-image group (2) contain the same number of images, but sub-image group (2) has a larger maximum time difference between the images. 2) is to be divided. That is, in FIG. 6A, sub-image group (5) and sub-image group (2) are further divided.

First, a method for re-dividing the sub-image group (2) will be described. The sub-image group (2) has two peaks of the number of images, and the image groups of these two peaks were taken on different dates. In such a case, the image is divided at the place where the photographing date changes (the dashed arrow between sub-image group (2) and sub-image group (3) in FIG. 6B).

Next, sub-image group (5) subdivision will be described. The sub-image group (5) has three peaks of the number of images, and the image groups of these three peaks are taken on different dates. In such a case as well, division is made at locations where the photographing date changes. At this time, although there are two places where the photographing date changes, the division is performed so that the difference in the number of images in each group after division becomes small. In FIG. 6B, in the sub-image group (5), splitting at the boundary between the second day and the third day is better than splitting at the boundary between the first day and the second day. Since the difference in the number of images in the group becomes small, the group is divided at the boundary between the second day and the third day. That is, the sub-image group (6) and the sub-image group (7) in FIG. 6B are divided at the dashed arrow.

As described above, the number of sub-image groups is changed from eight to ten. Here, the sub-images are divided at locations with different shooting dates, but if a group of sub-images with a large number of images were shot on a single day, they are split at locations with the largest time difference within a single day.

In S421, the spread allocation unit 212 integrates the sub-image groups. Integrating sub-image groups means integrating divided sub-image groups when the number of sub-image groups is greater than the number of spreads. Specifically, the sub-image groups are integrated so that the number of sub-image groups and the number of spreads match.

A case in which the number of designated spreads is 6 for the number of sub-image groups of 8 in FIG. 6A will be described as an example. FIG. 6(c) shows the result of integrating the sub-image groups of FIG. 6(a) and setting the number of sub-image groups to six. The number of sub-image groups is set to 6 by integrating the sub-image groups on both sides of the dashed line in FIG. 6C, in other words, by not dividing the dashed line. First, a sub-image group having a small number of images is specified among the sub-image groups in FIG. 6(a). Here, since the number of divisions is to be reduced from 8 to 6 by 2, two sub-image groups with a small number of images are specified. In FIG. 6(a), the sub-image group having the smallest number of images is the sub-image group (8), followed by the sub-image group (3) and the sub-image group (7). Therefore, first, the sub-image group (8) is to be integrated. Sub-image group (3) and sub-image group (7) contain the same number of images, but since sub-image group (8) adjacent to sub-image group (7) is to be integrated, sub-image group (3) is to be integrated.

Next, it is determined whether the group of sub-images to be integrated should be integrated with the group of sub-images captured earlier or with the group of sub-images captured later. First, the integration of sub-image group (3) will be described. First, the time difference between sub-image group (2) and sub-image group (4) before and after sub-image group (3) is compared. The difference in shooting time between the last image of sub-image group (3) and the first image of sub-image group (4) is the first image of sub-image group (3) and the last image of sub-image group (2). smaller than the difference in shooting time between Therefore, sub-image group (3) is determined to be merged into sub-image group (4). That is, the sub-image group is integrated at the broken line (3) in FIG. 6(c).

Next, the integration of sub-image group (8) will be described. Sub-image group (8) is integrated with previous sub-image group (7) because there is no subsequent sub-image group. That is, they are integrated at the broken line (6) in FIG. 6(c). Note that in the present embodiment, groups of sub-images having small differences in photographing time are integrated, but the present invention is not limited to this. For example, a group of sub-images to be integrated may be integrated into a group of sub-images of which the number of captured images is small.

In S422, the spread allocation unit 212 performs spread allocation. Through S418 to S421, the number of sub-image groups and the number of specified spreads are the same. In this embodiment, the group of sub-images with the shooting date and time at the top is assigned to the top of the double-page spread. That is, the sub-image groups are assigned to each spread page of the album in order of shooting date and time. As a result, it is possible to create an album in which sub-image groups are arranged in order of shooting date and time. It should be noted that, as will be described later, the images in one spread page do not have to be arranged according to the shooting date and time.

In S423, the image selection unit 213 selects a predetermined number of images from each sub-image group. An example of selecting four images from a group of sub-images assigned to a spread will be described with reference to FIG. Here, the number of pages of the spread is 1 or 2, the spreads other than the first spread and the last spread are two pages facing each other when opened, and the first spread and the last spread are one page.

FIG. 9A shows the time difference between the shooting dates and times from the first image to the last image included in the group of sub-images assigned to the spread, in other words, the shooting period of the group of sub-images. A method of selecting the first sheet when selecting four sheets will be described with reference to FIG. 9(b). A template 1001 shown in FIG. 10 has one main slot 1002 . For the first sheet, an image for the main slot is selected. The image selection unit 213 selects the image with the highest score for the main slot assigned in S416 from all the images shot during the shooting period of the sub-image group shown in FIG. 9B.

Then, for the second and subsequent sheets, images for sub-slots are selected. In the present embodiment, images are selected by subdividing the shooting period of the sub-image group so that the selected images are not concentrated in part of the shooting period of the sub-image group. First, the image selection unit 213 divides the imaging period of the sub-image group into two imaging time intervals as shown in FIG. 9C. That is, the sub-divided imaging period is divided into two equal imaging time intervals (grouped). Next, the image selection unit 213 selects, as the second image, the image with the highest sub-slot score from the shooting time interval (the solid line interval) in which the first image is not selected as shown in FIG. 9(d). do. Next, as shown in FIG. 9E, the image selection unit 213 divides each shooting time interval shown in FIG. 9D into two equal parts. The image selection unit 213 selects images (two imaging time The image with the highest score for the sub-slot is selected as the third image.

Next, an example in which an image cannot be selected because there is no image in the shooting time interval for selecting an image will be described by taking the selection of the fourth image as an example. In this embodiment, when selecting images to be laid out in a double-page spread from the sub-image group, the shooting time period of the sub-image group is divided according to time regardless of the number of images. Therefore, there is a case where an image does not exist in the imaging time interval obtained by division. For example, as shown in FIG. 9G, it is desired to select the fourth image from the imaging time interval (shaded imaging time interval) for which no image has been selected yet, but no image exists in this hatched imaging time interval. and In this case, the image selection unit 213 divides each shooting time interval in which the image has been selected into two, as shown in FIG. 9(h). Next, as shown in FIG. 9(i), the image selection unit 213 selects 4 images with the highest points for the sub-slots among the images captured in the solid-line capturing time interval in which the first to third frames are not selected. Select as second.

FIG. 11 is a diagram explaining in more detail the image selection unit 213 shown in FIG. The image selection unit 213 selects an image from a group of sub-images assigned to a spread.

A number setting unit 1101 sets the total number of images to be selected from the group of sub-images assigned to the spread to be processed. In other words, the total number of images to be laid out in each spread is set.

The image group acquisition unit 1102 acquires the sub-image group assigned to the spread from the image group acquired by the image acquisition unit 205 and the scene ID created by the image classification unit 208 . When acquiring an image group, the image group acquisition unit 205 acquires shooting date/time information, image type, object information, main slot score, and sub-slot score for each image. In this embodiment, the image types are classified into still images, moving image clipped images, and SNS images, but are not limited to these. Image types other than the three described above may be distinguished, and still images may be distinguished in more detail, such as still images taken with a digital camera and still images taken with a smart device.

A loop counter 1103 manages the number of times the process of selecting an image to be laid out in a double-page spread from the group of sub-images has been executed. Note that the process of selecting this image is executed by the method described with reference to FIG. The managed number of times is used to determine the score axis to be used and to set how many shooting time intervals the shooting period of the sub-image group assigned to the spread to be processed is divided into.

A score axis setting unit 1104 sets a score axis to be used according to the number of times of processing counted by the loop counter 1103 . "Set the score axis to be used" means "set the image evaluation criteria". Here, the main slot score axis (main slot evaluation criteria) and the sub Set the score axis for slots (evaluation criteria for sub-slots). In this embodiment, the evaluation criteria are set for each slot type by switching between these multiple evaluation criteria. Here, for example, for the first time, the score axis for the main slot is set, and the processing for selecting the image for the main slot is set to be executed. Set to execute the process of selecting an image.

In the present embodiment, the main slot scoring axis and the sub-slot scoring axis have been described as examples of the scoring axis to be used, but another scoring axis may be used. For example, in addition to the main slot score axis and the sub slot score axis, a person relevance score axis may be used. Note that the degree of relevance to a person can be scored based on, for example, the number of appearances of a person or the number of times they are photographed together. Specifically, it can be determined that a person who appears many times has a high degree of relevance to the group of sub-images that were evaluated, and it is determined that people who appear together a large number of times have a high degree of relevance to each other. can do. In addition, although scoring is performed by switching each scoring axis, an integrated scoring axis that combines the scoring axis for the main slot, the scoring axis for the sub-slots, the scoring axis for the degree of person relevance, and the like may be used. At this time, the weight of each score axis when calculating the integrated score may be changed. Also, the score axis to be used may be changed at random according to the number of times of processing, or the score axis may be switched by referring to a table prepared in advance.

A dividing unit 1105 divides the capturing period of the sub-image group into a predetermined number of segments based on the capturing date/time information acquired by the image analyzing unit 207 . First, from the shooting times of the images included in the sub-image group acquired by the image group acquiring unit 1102, the shooting start time and shooting end time of the sub-image group assigned to the spread to be processed are specified. Calculate the shooting period. That is, the imaging time of the image captured earliest and the imaging time of the image captured latest among the images included in the sub-image group are specified, and the imaging period is calculated.

Next, based on the number of times of processing counted by the loop counter 1103, the calculated imaging period is equally divided into imaging time intervals. The division number of the imaging time interval is set by a power of 2 based on the count. In the present embodiment, the method of equally dividing the imaging period by the power of 2 based on the count has been described, but the method is not limited to this, and the imaging period may be equally divided by the count number. The division unit 1105 also outputs the start time and the end photography time of each divided photography time interval.

An image type setting unit 1106 sets the type of image selected in image selection according to the number of times of processing counted by the loop counter 1103 . In this embodiment, the SNS image is selected for the first processing, the moving image clipped image is selected for the second processing, and the still image is selected for the third and subsequent processing.

A section information setting unit 1107 divides the images included in the sub-image group acquired by the image group acquisition unit 1102 into groups according to the shooting time sections divided by the dividing unit 1105, and obtains the shooting information of the images and the score of each image. , etc., for each shooting time interval.

A mode setting unit 1108 sets the mode specified by the album creation condition setting unit 201, and performs the subsequent processing so that objects corresponding to the set mode are arranged. Examples of modes are person, animal, and food, as described above.

The image selection unit 1109 selects one image with a high score including an object corresponding to the mode from each imaging time period. Image selection by the image selection unit 1109 is based on the score axis set by the score axis setting unit 1104, the mode set by the mode setting unit 1108, and the score of the image set by the section information setting unit 1107. is done. As described with reference to FIG. 9, no image is selected from the photographing time interval in which there is no image. In this embodiment, one image is selected from each imaging time interval, but multiple images may be selected from each imaging time interval. In the present embodiment, determination of an image including an object corresponding to the set mode includes whether an object corresponding to the mode set up to TOP 3 of the object classification of each image at the time of analysis shown in FIG. 5 is included. is a determination condition, but is not limited to this.

Each time an image is selected from the shooting time period, the similarity determination unit 1110 performs similarity determination between the selected image and the previously selected image. If the result of similarity determination, which will be described later, is equal to or greater than the determination threshold, it is determined that the images are similar, and if the result is less than the determination threshold, it is determined that they are not similar. If it is determined that they are not similar, the image selected from that section is held, and if it is determined that they are similar, the selected image of that section is discarded.

For similarity determination, determination by pattern matching, determination using the SHIFT method, or the like can be used. The determination threshold used for similarity determination is changed according to the shooting time difference between images to be compared. For example, when the similarity determination result is output in the range of 0 to 100, the closer the value is to 100, the more similar it is. I judge. If the shooting time difference is short (for example, within 15 seconds) due to continuous shooting or the like, the determination threshold is set low (for example, set to 60). Otherwise, it is desirable to set the determination threshold to 80. By lowering the determination threshold, it becomes easier to determine that the images are similar, and even if there is a slight similarity, it is determined that the images are similar. In this embodiment, in this way, the determination criteria are changed according to the imaging time difference.

The integration unit 1111 selects an image to be finally left from the images selected in each section determined to be dissimilar by the similarity determination unit 1110 . When images are selected from a plurality of sections, it is determined which section of the image is to be selected. When judging an image, the same score as judged by the image selection unit 1109 is used, and an image with a high score is selected. In this embodiment, an image with a high score is selected, but the selection is not limited to this. For example, among sections in which an image is selected, a section in which an image is not selected from adjacent sections may be selected.

The image management unit 1112 adds the image selected by the integration unit 1111 as a selected image, and determines whether the required number set by the number setting unit 1101 is exceeded. When the required number of images has been reached, the image selection process is terminated. If the required number of sheets has not been reached, the loop counter 1103 is counted up, and the image selection process described above is executed again.

Here, for the sake of explanation, an example in which images containing the same object are selected in the main image and the sub-image has been described, but different objects can be selected in the main image and the sub-image. For example, when the loop count is 1, the main image is selected. By switching the object to be selected when the loop count is 1 and when the loop count is other than 1, different objects can be selected for the main image and the sub-image.

FIG. 12 is a flow chart of processing for selecting an image to be arranged in a template slot from a group of sub-images performed in the configuration shown in FIG. A detailed flow for selecting an image from the group of sub-images assigned to the spread to be processed will be described. Note that FIG. 12 is a flowchart for explaining the detailed processing of S423.

In S1201, the number setting unit 1101 sets the total number of images to be selected from the group of sub-images assigned to the spread to be processed. Note that this total number of images may be determined based on the setting value of 309 in FIG. 3, or may be a value set by default.

In S<b>1202 , the image group acquisition unit 1102 acquires image data corresponding to the sub-image group assigned to the spread to be processed from the image data of the image group acquired by the image acquisition unit 205 .

In S1203, the mode setting unit 1108 sets the mode. Here, the person mode, animal mode, or food mode described above is set.

In S1204, the loop counter 1103 counts the number of times the image selection process is executed for each sub-image group. Note that the initial value is 1.

In S1205, the score axis setting unit 1104 sets the score axis to be used. In this embodiment, the score axis to be used is set according to the number of image selection processes managed by the loop counter 1103 . A specific setting method has been described above in the description of the score axis setting unit 1104 .

In S1206, the image type setting unit 1106 sets the image type to be selected. When the loop count is 1, that is, when the number of times of processing is the first time, in the present embodiment, the SNS image is set to be selected. When the loop count is 2, that is, when the number of times of processing is the second time, setting is made so as to select a moving image clipped image. When the loop count is 3 or more, that is, when the number of times of processing is 3 or more, a still image is set to be selected.

In S1207, the dividing unit 1105 calculates the shooting period of the sub-image group from the start and end shooting times of the sub-image group assigned to the spread to be processed, and divides the sub-image group based on the number of times of processing counted by the loop counter 1103. Evenly divide the shooting time interval of . Note that when the number of times of processing is the first, no image is selected at this stage, so no division is performed. When the number of times of processing is the second time, one image is selected, and since there is one shooting time interval in the shooting period of the sub-image group, the entire shooting period is evenly divided into two to create two shooting time intervals. . When the number of times of processing is 3, two images are selected and there are two shooting time intervals, so each of the two shooting time intervals is divided into two to create four shooting time intervals.

In step S1208, the section information setting unit 1107 groups the acquired image data of the sub-image group for each divided shooting time section, and acquires the shooting information of each image and the score of each image for each shooting time section. do.

In S<b>1209 , it is determined whether or not the image of the shooting time period of interest divided by the division unit 1105 has been selected. If the shooting time interval of interest is an already selected shooting time interval, the process advances to S1214. If it is an unselected shooting time section, the process proceeds to S1210. Note that in the first loop, there is one shooting time interval of interest because the shooting time interval is not divided.

In S1210, it is determined whether or not an object of the set mode is included in the shooting time interval of interest, which was an unselected shooting time interval. For example, if an animal is set as the mode, it is determined whether or not there is an image containing an object identified as an animal. If there is no image containing the designated object, the process advances to S1214. If the specified object is included, the process proceeds to S1211.

In S1211, the image selection unit 1109 selects one image including the designated object from the images within the shooting time period of interest. Note that if there are a plurality of images including the specified object within the shooting time interval of interest, one image with the highest score is selected.

In S1212, the similarity determination unit 1110 performs similarity determination between the image selected in the imaging time interval of interest and the image selected in another previously selected imaging time interval. If it is determined that there is no similar image in another imaging time interval, the image selected in the imaging time interval of interest is retained as it is, and the process proceeds to S1214. If it is determined that a similar image exists in another shooting time interval, the process proceeds to S1213, cancels the image selected in the target shooting time interval, and proceeds to S1214.

In S1214, it is determined whether or not the image selection processing has been performed for all the shooting time intervals divided in S1207. If it has not been executed for all the shooting time intervals divided in S1205, the process returns to S1208, and image selection is performed for the shooting time intervals for which images have not been selected.

In S<b>1215 , the integration unit 1111 selects an image with the highest score to be finally left from the images of each shooting time period determined to be dissimilar by the similarity determination unit 1110 . That is, if two or more images remain in S1209 to S1214, the remaining image is selected.

In S1216, the image management unit 1112 adds the images selected in S1215 to the selected images and manages the total number of selected images.

In S<b>1217 , it is determined whether the total selected number of sheets managed by the image management unit 1112 matches the required number of sheets set by the number setting unit 1101 . If the required number of images and the total selected number of images match, the image selection process is terminated. If the total selected number is less than the required number, the process proceeds to S1218.

In S1218, it is determined whether or not all images including objects of the designated mode have been selected from the sub-image group. If all the images including the object of the designated mode have not been selected and still exist in the sub-image group, the process returns to S1204 to continue image selection processing until the total number of selected images reaches the required number. When returning to S1204, the counter managed by the loop counter 1103 is incremented by one. As a result, the dividing unit 1105, which works in conjunction, sets twice as many divisions of the imaging time interval as the previous time. Since the imaging time interval is divided more finely and an image is selected in an unselected imaging time interval, an image that has not been selected so far is selected. When all the images including the object of the mode specified in S1218 have been selected from the sub-image group, the process proceeds to S1219.

In S1219, the designated mode is changed if there is no image of the designated mode even if all the unselected images among the images of the sub-image group are determined. As for the method of change, the mode may be changed to the mode with the highest priority other than the specified mode according to a predetermined priority, or the mode may be changed based on the objects included in the sub-image group. The process returns to S1203, the mode is reset to the changed mode, and the process is repeated until the required number of sheets is acquired.

Returning to FIG. 4, in S424, the template setting unit 214 sets a plurality of templates to be used for layout from the storage device 104. FIG. The set template has a design designated by the album creation condition setting section 201 .

In S425, the image layout unit 215 determines the spread image layout to be processed. A template suitable for arranging the selected image is determined from among the plurality of templates set by the template setting unit 214 . A template determination method will be described with reference to the block diagram of FIG. 13 illustrating the configuration of software for template determination.

A selected image information acquisition unit 1301 acquires the number of selected images selected by the image selection unit 213 and image information about each selected image. The image information to be acquired includes the width and height of the image, information on the shooting date and time, and the score calculated by the image scoring unit 210 .

A template selection unit (number of slots) 1302 selects a template candidate having a number of slots that matches the number of selected images from those set by the template setting unit 214 .

A selected image sorting unit 1303 sorts the selected images in order of shooting date and time.

In the template selection section (main image position) 1304, among the templates selected by the template selection section 1302, the arrangement order of the main slots in which the main images are arranged matches the arrangement order of the main images when arranged in order of shooting date and time. Select template suggestions. Next, a template candidate having the same aspect ratio between the main image and the main slot is selected. In this embodiment, an image with an older shooting date and time is arranged in the upper left of the template, and an image with a newer shooting date and time is arranged in the lower right.

A template selection unit (sub-image position) 1305 selects a template suitable for the image selected by the image selection unit 213 from among the templates selected by the template selection unit (main image position) 1304 . A template selection unit (sub-image position) 1305 selects a template in which the sub-slots in which sub-images are arranged match the order of the sub-images arranged in order of photographing date and time, and the aspect ratios of the sub-images and sub-slots match. .

FIG. 14 is a processing flow diagram of template determination.

In S1401, the selected image information acquisition unit 1301 acquires the number of selected images of the selected images and the image information of each selected image. Here, a case is assumed in which three images are selected by the processing of FIG. 12 . Therefore, 3 is acquired as the number of selected images in S1401.

In S1402, the template selection unit 1302 (number of slots) selects a template candidate whose number of selected images matches the number of slots. Here, a template with 3 slots is selected. Note that templates (1-1) to (4-4) shown in FIG. 10(a) are examples of templates with three slots.

In S1403, the selected image sorting unit 1303 arranges the selected images in order of shooting time. Assume that the result of arranging the selected images in the order of shooting date and time is that shown in FIG. Here, image 1005 is the main image selected for the main slot, and images 1006 and 1007 are sub-images selected for the sub-slot.

In S1404, when the template selection unit (main image position) 1304 arranges the selected images in the shooting order, the arrangement order of the main images and the arrangement order of the main slots match, and the aspect ratios of the main images and the main slots match. template candidates are selected. Since the image 1005 has the latest shooting date and time and is the main image for the main slot, templates (3-1) to (3-4) shown in FIG. 10A are candidates.

In S1405, the template selection unit (sub-image position) 1305 selects a template in which the positions and aspect ratios of the sub-image and sub-slot match. In the example shown in FIG. 10(b), the old image 1006 for the sub-slot, that is, the image to be placed at the upper left is the vertical image, and the new image 1007 is the horizontal image. Template (3-2) is therefore selected as the most suitable template for the selected image.

Thus, at S425, information about which template to use and which image to place in which slot of the template is determined.

In S426, the image correction unit 217 performs image correction. When the image correction condition input to the image correction unit 217 is ON, image correction is performed. In this embodiment, dodging correction, red-eye correction, and contrast correction are automatically performed as image corrections. When the image correction condition input to the image correction unit 217 is OFF, image correction is not performed. The corrected image is converted to 1200 pixels on the short side and sRGB color space.

In S427, the layout information output unit 218 creates layout information. Each image is placed on the template as determined in S425. At this time, the image to be used is scaled and arranged according to the size information of each slot of the template. The image used here is the corrected image generated in S426, and is an image different from the analysis image used in S408 to S418. Then, the layout information output unit 218 generates bitmap data in which images are arranged in the template as layout information.

In S428, it is determined whether or not S423 to S427 have been completed for all spreads. If not finished (No in S428), the process returns to S423. If completed (Yes in S428), the automatic layout process is terminated.

In this way, an album is created according to the flow described above, and the result is previewed for the user. The user confirms the previewed completed album and performs editing operations as necessary.

From here, a method of adding spreads or pages according to user instructions to a completed album having existing spreads or existing pages and performing automatic layout for additional spreads or additional pages will be described.

FIG. 15 is a block diagram illustrating the configuration of software for automatically selecting images to be used for additional double-page spreads.

A double-page spread addition position setting unit 1501 determines the position at which a double-page spread is added based on the user's input in the automatic layout result. Designation of the spread addition position will be described with reference to FIG.

FIG. 21 is a diagram showing an example of a UI for designating a spread addition position for an automatic layout result. A window 2101 is a display window for previewing the result of automatic layout. All layout results can be checked by moving the slider bar arranged at the bottom of the display window. A double-page spread 2102 shows the layout result of each double-page spread created by automatic layout. The user can use the pointer 2103 to specify the position at which the spread is to be added to the automatic layout result. When the user uses the mouse 107 to move the pointer 2103 to a position where the user wants to add a double-page spread, an "add double-page spread" pop-up is displayed. By clicking at the position where the popup is displayed, the position to add the spread can be confirmed.

The additional image setting unit 1502 presents a group of unused images in the album to the user and allows the user to select an image to be used for the additional spread from the group of presented images. Designation of an image by the user will be described with reference to FIG.

FIG. 22 is a diagram showing an example of a UI for selecting images to be used for additional spreads. A window 2201 is a display window for displaying a list of specifiable images. An image 2202 is an image that can be specified. Here, unselected images are displayed in automatic layout. The user can use the pointer 2203 to designate an image to be used for the additional spread. The example of FIG. 22 shows a situation where the pointer 2203 is moved and selected on the image 7 . Multiple images may be selected to be used for additional spreads. A button 2204 is a button for confirming that the selection of the designated image has been completed. When a button 2204 is pressed, the selected image is confirmed as the image to be used for the additional spread.

Here, for convenience of explanation, it is assumed that the images to be used for the additional double-page spread are selected from the group of images contained in the directory specified by the album creation condition setting section 201, but the user selects the group of images contained in another directory. can be specified.

Returning the description to FIG. A designated image analysis unit 1503 analyzes the image selected by the user in the additional image setting unit 1502 . Here, a case where one image is specified in the additional image setting unit 1502 will be described. The designated image analysis unit 1503 performs image analysis similar to that of the image analysis unit 207 on the designated image. If the image is analyzed during automatic layout, the analysis result is stored. By using the result, the processing in the designated image analysis unit 1503 can be omitted, thereby speeding up the processing. If an image included in a different directory from the image specified by the album creation condition setting unit 201 is specified, the specified image analysis unit 1503 performs analysis processing because there is no analysis result. As described above, the analysis results include object determination in image data, face detection, expression recognition of detected faces, personal recognition results of detected faces, and shooting date/time information. Here, the shooting date/time information will be described as an example of the analysis result.

An image selection condition setting unit 1504 sets conditions for automatically selecting image candidates to be used for additional spreads other than the images specified by the additional image setting unit 1502 . For the sake of explanation, from among the group of sub-images divided by the image classification unit 208, from the photographing date and time of the image specified by the additional image setting unit 1502, Set the image selection condition (range) to be classified. Here, it is assumed that the images specified in the additional image setting section 1502 are selected from the group of images included in the directory specified in the album creation condition setting section 201, as described above. The image group included in the directory specified by the album creation condition setting unit 201 has already been divided into sub-image groups based on the shooting date and time, and scene-classified. Therefore, it is possible to identify which sub-image group and which scene the specified image belongs to based on the shooting date and time.

If the image specified in the additional image setting section 1502 does not have shooting date and time information, the image selection condition (range) is set from the spreads before and after the spread addition position specified in the spread addition position setting section 1501. good too. For example, the latest shooting date and time of the image used in the spread before the additional spread is set as the start shooting date and time. Also, the earliest shooting date and time of the image used in the spread after the additional spread is set as the end shooting date and time. The start date and time and the end date and time are set as image selection conditions.

When an image included in a different directory from the image specified in the album creation condition setting unit 201 is specified, all images in the different directory may be set in the image selection range. Also, images in another directory are analyzed in the same manner as the image classification unit 208, image groups are divided, and scenes are classified. You can set the range.

An image analysis unit 1505 acquires images that satisfy the image selection conditions set by the image selection condition setting unit 1504, excluding images specified by the additional image setting unit 1502, and performs analysis processing similar to that of the image analysis unit 207. . If image analysis has been performed in advance and the analysis results have been saved, it may be used. If the result is not saved, the analysis process is executed.

The image scoring unit 1506 scores the image acquired by the image analyzing unit 1505 based on the image analysis result. When scoring an image, the same processing as that of the image scoring unit 210 is performed. If the image score is calculated in advance and the result is saved, it may be used. If the image score is not stored, the image score is calculated based on the analysis results.

The number of images to be selected setting unit 1507 sets the number of images to be arranged in the additional spread and the number of images to be automatically selected, which is the number of images to be arranged in the additional spread minus the number of images specified in the additional image setting unit 1502. do. Here, the average number of images per one page spread is calculated from the number of images used in each spread during automatic layout, and the average number of images is set as the number of images to be arranged in the additional spread. Also, the number of images obtained by subtracting the number of images specified by the additional image setting unit 1502 from the average number of images is set as the number of automatically selected images to be used from among the images acquired based on the image selection conditions.

Here, the number of images obtained by subtracting the number of images specified by the user from the average number of images is set as the number of automatically selected images, but the minimum and maximum values of the images to be arranged may be set. For example, as the number of images to be placed in an additional spread, search for the minimum number of images in the spread in the album and set it to the minimum value, and similarly search for the maximum number of images in the spread in the album and set it to the maximum value. may The minimum and maximum number of images to be arranged in the additional spread may be set as fixed values instead of the number of images to be used in the album. Also, the minimum and maximum values may be set according to the number of images that can be arranged in the held template. Also, the number of automatically selected images may be set according to the number of unused images included in the range set by the image selection condition setting unit 1504 . For example, when the number of unused images is less than the average number of images or the minimum value, the number of unused images may be set as the number of automatically selected images.

An automatic image selection unit 1508 selects an image to be laid out in an additional spread from images acquired based on image selection conditions, in addition to images specified by the user in the additional image setting unit 1502 . Similar image determination is performed for images specified by the user in the unused image group included in the range set by the image selection condition setting unit 1504, and the automatic image selection set by the image selection number setting unit 1507 is performed from images that are not similar. The number of selected images is selected in descending order of the score. If the number of unused images included in the range set by the image selection condition setting unit 1504 is small, the threshold set during similar image determination may be changed according to the number of unused images. For example, at the time of selection, the same processing as that of the image selection unit 213 may be performed using shooting date/time information.

An additional spread layout section 1509 automatically lays out the images set by the additional image setting section 1502 and the images automatically selected by the image automatic selection section 1508 in an additional spread. Here, for the sake of explanation, the automatic layout process performs the same processing as the layout method performed by the image layout unit 215 shown in FIG. 2 and the template determination method explained with reference to FIGS.

Here, the case where the same layout method as the automatic layout of the entire album is used for the additional double-page spread has been described, but other layout methods may be used. Templates may also be dynamically generated based on the aspect ratio of the selected image, time sequence, location of large slots, and the like. Furthermore, without using a template, the image layout, such as the position and size of the image in the spread, may be set from the selected image. In the present disclosure, the arrangement of an image means the position of the outer edge of the image, the position of each side constituting the outer edge, or the position of each vertex.

A layout information acquisition unit 1510 acquires layout information related to the layout result created by the additional spread layout unit 1509 . The layout information includes the template used, the background color, the background pattern, the setting of the margins, the features of the arranged images, the arrangement order of the images with respect to the date and time of shooting, the images arranged in the large slots, and the like. The layout information acquisition unit 1510 acquires the layout information regarding the additional double-page spread.

A layout change determination unit 1511 determines whether or not to change the layout of existing laid-out spreads other than additional spreads. Also, the layout change determination unit 1511 sets the portion to be changed in the existing spread and the layout change method based on the layout information of the additional spread acquired by the layout information acquisition unit 1510 .

Here, for the sake of explanation, it is determined whether or not the additional spread and the adjacent front and back spreads have similar layout results. As a method of determining whether or not the layout results are similar, the number of slots, the size of the slots, and the positions of the slots of each size of the templates adopted in the additional spread and the adjacent front and back spreads are compared. judge. As another method of determining whether or not the layout results are similar, a number may be assigned to each template, and templates having the same assigned number may be determined to be similar.

The layout change determination unit 1511 compares the template used in the layout of the additional spread with the template used in the layout of the existing spread adjacent to the additional spread. Here, for the sake of explanation, it is assumed that the similar template is used in the layout of the existing spread immediately before the additional spread.

When it is determined that a similar template is used, it is determined that the existing double-page spread determined to use a similar template requires a layout change to use a template different from the template in use.

Here, for the sake of explanation, the determination process is performed on the spread adjacent to the additional spread, but the determination process may also be performed on the spread that is not adjacent. As a determination target criterion at that time, for example, an existing spread using an image of the same scene as the image used in the additional spread may be determined.

The existing double-page spread automatic layout unit 1512 changes the layout of the existing double-page spread determined by the layout change determination unit 1511 to require layout change. Based on the change method set by the layout change determination unit 1511, the existing spread is automatically laid out for the existing spread whose layout is to be changed. For the sake of explanation, here is an example of how to change the template to be used.

The automatic layout process is similar to the layout method performed by the image layout unit 215 shown in FIG. 2 and the template determination method explained with reference to FIGS. At this time, based on the determination result of the layout change determination unit 1511, control is performed so that the images are arranged using a template other than the template determined to be similar. Specifically, in the template determination method described with reference to FIGS. 13 and 14, the templates used in the additional spread layout are excluded from the template candidates. As a result, a layout result using another template can be automatically created. Further, when performing the automatic layout of the additional double-page spread, a template set different from the template set used in the automatic layout of the existing double-page spread may be prepared. Furthermore, by using a different template set for automatic layout of additional spreads when changing the layout of existing spreads, it is possible to reduce the influence on other existing spreads whose layouts are not changed.

For the sake of explanation, the additional double-page spread layout section 1509 and the existing double-page spread automatic layout section 1512 are described as separate configurations, but they may be implemented as one configuration.

Also, although the method for changing the layout of the existing double-page spread has been described, images used in the layout of the existing double-spread may be changed or added. For example, if an image similar to the image used in the existing spread is used in the additional spread, an image similar to the image used in the additional spread used in the existing spread is replaced with an image that is not similar. can be changed to

The above is the description of the block diagram for executing the automatic layout for additional spreads and existing spreads.

23A and 23B are diagrams for explaining layout results obtained by performing automatic layout on additional spreads and existing spreads. FIG. 23(a) shows a spread after automatic layout before addition of a spread. Rectangles within the spread indicate the arranged images. FIG. 23B shows the layout result after the additional spread layout unit 1509 adds a spread at the additional position specified by the user. When a spread is added to an existing album, the same template is used for the additional spread and the N-spread adjacent to the additional spread. Layout aesthetics may collapse, such as giving In order to avoid such continuation of similar spreads, the layout information acquisition unit 1510 acquires the layout results of the additional spreads, and the layout change determination unit 1511 determines whether to change the layout of the existing spreads. In the case shown in FIG. 23B, it is determined that the template used for the N spread and the additional spread is the same, and the layout of the images in the layout of the existing spread is changed. FIG. 23C shows the result of changing the layout of the existing double-page spread by the existing double-page spread automatic layout unit 1512 . Here, for the sake of explanation, the result of changing the arrangement of images in an existing double-page spread by selecting a template whose chronological order does not change is shown. By adjusting the layout of the existing double-page spread in this way, it is possible to avoid deterioration of the overall aesthetics of the album due to the continuation of the elements constituting the double-page spread.

FIG. 16 is a flow chart for executing automatic layout for additional spreads and existing spreads.

In S1601, the spread addition position setting unit 1501 sets the spread addition position specified by the user with the pointer for the preview of the automatic layout result.

In S1602, the additional image setting unit 1502 presents the album unused images to the user, prompts the user to select an image to be placed in the additional spread from the presented image group, and instructs the user to place the image specified by the user in the additional spread. Set as selected image.

In S<b>1603 , the designated image analysis unit 1503 analyzes the image set by the additional image setting unit 1502 . A designated image analysis unit 1503 performs analysis processing similar to that of the image analysis unit 207 .

In step S1604, the image selection condition setting unit 1504 automatically selects image candidates to be arranged in the additional spread, in addition to the images set by the additional image setting unit 1502, based on the analysis results of the images set by the additional image setting unit 1502. Set the conditions for selection. The condition set here is that the image is classified into the same scene as the image set by the additional image setting unit 1502, for example.

In S<b>1605 , the image analysis unit 1505 sequentially analyzes images that meet the conditions set by the image selection condition setting unit 1504 . An image analysis unit 1505 performs analysis processing similar to that of the image analysis unit 207 .

In S<b>1606 , the image scoring unit 1506 scores the image based on the image analysis result of the image analysis unit 1505 . An image scoring unit 1506 performs scoring processing similar to that of the image scoring unit 210 .

In S1607, it is determined whether or not the image corresponding to the condition set by the image selection condition setting unit 1504 has been analyzed and scored. If the analysis and scoring of all images that meet the specified conditions are completed, the process advances to S1608. If there are still images that have not been analyzed and scored, the process returns to S1605 to continue processing the images that have not been analyzed and scored.

In S1608, the image selection number setting unit 1507 sets the number of images to be arranged in the additional spread and the number of automatically selected images, which is the number of images to be used from among the images acquired based on the image selection conditions. Note that a specific setting method has been described above in the description of the image selection number setting unit 1507, and is therefore omitted.

In S1609, in addition to the images set by the additional image setting unit 1502, the image automatic selection unit 1508 selects the images to be arranged in the additional spread from among the images acquired based on the image selection conditions in descending order of score.

In S1610, the additional spread layout unit 1509 uses the images automatically selected by the automatic image selection unit 1508 to automatically lay out the additional spread.

In S<b>1611 , the layout information acquisition unit 1510 acquires information on the layout result of the additional spread created by the additional spread layout unit 1509 .

In S1612, the layout change determination unit 1511 compares the layout results of the additional spread and existing spreads that have already been laid out other than the additional spread, and determines whether to change the layout of the existing spread. Note that a specific setting method has been described above in the description of the layout change determination unit 1511, and is therefore omitted.

In S1613, the existing double-page spread automatic layout unit 1512 changes the layout of the existing double-page spread determined by the layout change determination unit 1511 to require layout change. For example, the existing spread automatic layout unit 1512 executes S1613 by changing the template used by the existing spread to another template. Alternatively, S1613 may be realized by changing the slot size or performing trimming.

In S1614, it is determined whether or not automatic layout has been performed for all existing double-page spreads determined to be changed by the layout change determination unit 1511, and if it has been performed, the process ends. If not, the process returns to S1613 to perform spread processing for which layout has not been performed.

The above is the description of the flow chart for executing the automatic layout for additional spreads and existing spreads.

In S1612, the layout results of the additional spread and the existing spread that has already been laid out other than the additional spread are compared, and it is determined whether or not the layout results are similar. Then, when it is determined that both are similar, it is decided to change the layout of the existing double-page spread. However, other criteria may be used for determination. For example, in S1612, the layout results of the additional spread and the existing spread that has already been laid out other than the additional spread are compared to determine whether or not the layout results are similar. Then, if it is determined that the two are not similar, it may be decided to change the layout of the existing double-page spread. At this time, the layout of the existing page is changed so as to resemble the layout of the added page. For example, the layout of the existing page is changed by applying the same template as the additional page to the existing page.

In this embodiment, when an additional spread is added to a specific position of the album, by changing the layout of the existing spread in accordance with the layout of the additional spread, the album is highly aesthetically harmonious even after the addition of the spread. can be created automatically.

(Embodiment 2)
The second embodiment describes a method of automatically laying out an additional spread and an existing spread.

FIG. 17 is a block diagram for executing automatic layout by combining additional spreads and existing spreads. Since the same numbers as in FIG. 15 represent the same devices and perform the same processing, the description is omitted here.

An image selection number setting unit 1507 sets images to be arranged in the additional spread. Here, the minimum number of images in the existing double-page spread in the album is searched for, and the minimum number of images to be arranged in the additional double-spread is set.

The layout change range determination unit 1701 is a device that determines the range of changing the layout of the existing spread based on the image selected by the image automatic selection unit 1508 in the spread addition position setting unit 1501 . Here, the layout change range is determined depending on whether or not the photographing date and time information of the images arranged in the existing spread and the images set in the additional image setting unit 1502 and the images selected in the automatic image selection unit 1508 overlap. judge. For example, a case will be described in which an image to be arranged in an additional spread is selected from a group of unused images in the same scene as the image arranged in the existing spread immediately before the additional spread. In this case, since the images arranged in the existing spread and the images arranged in the additional spread are images of the same scene, the images are arranged in a consistent predetermined order from the existing spread to the additional spread, for example, in order of shooting date and time. is desirable. However, the images selected for use in the additional spread are not necessarily newer than the shooting dates and times of all the images placed in the existing spread just before the additional spread. Therefore, it is necessary to change the allocation of images to the existing and additional spreads and lay out the images newly allocated to the existing and additional spreads.

In addition, when images are selected by the image automatic selection unit 1508, if there are few unused images in the range set by the image selection condition setting unit 1504, or if the number of images set by the image selection number setting unit 1507 cannot be selected. There is Alternatively, even if there are more than the set number of unused images, if the image quality is poor or not suitable for use in the layout, the set number of images may not be selected. In such a case, select the image placed in the existing spread as an image to be placed in the additional spread, and perform automatic layout of the additional spread, and at the same time, automatically lay out the existing spread with the image moved to the additional spread. I can think of a way. If an image is moved from an existing spread to an additional spread, it is necessary to change the arrangement of the images in the existing spread. set as

A spread allocation unit 1702 allocates images to be laid out to existing spreads and additional spreads determined by the layout change range determination unit 1701 . Here, a spread to which each image is assigned is set based on the image capturing time difference. First, the group of images to be arranged in the existing spread and the additional spread included in the change range is divided by the section with the largest shooting time difference. Next, among the divided sections, a section having a large photographing time difference is divided. This is repeated until the number of existing spreads and additional spreads included in the change range is reached, and when the image group is divided up to the necessary number of spreads, the images are assigned to each spread within the change range in section units. At this time, the division position may be adjusted so that the number of images included in each divided section becomes the average number of images, in addition to the shooting time difference. Also, when the number of unused images is small or the unused images are not suitable for the layout, if many images are moved from the existing spread to the additional spread, the existing spread structure can be adjusted, such as reducing the number of existing spreads. You can change it.

The automatic layout unit 1703 performs automatic layout for each spread determined by the layout change range determination unit 1701 using the images assigned to each spread by the spread allocation unit 1702 . The layout method performed by the automatic layout unit 1703 is the same as the method described in the first embodiment.

The above is the description of the block diagram for executing the automatic layout by combining the additional spread and the existing spread.

24A and 24B are diagrams for explaining the result of automatic layout in which the additional spread and the existing spread are combined. FIG. 24(a) shows existing spreads after automatic layout before addition of spreads. Rectangles within the spread indicate the arranged images. Here, for the sake of explanation, it is assumed that all images have the same aspect ratio. Event described in each image indicates the event of the image. Numerical values written under Event are IDs that schematically represent the order of photographing date and time. Intermittent numbers indicate that there are images captured between them and not adopted as images to be arranged in the double-page spread. FIG. 24(b) shows a layout result obtained by adding a double-page spread to a user-specified portion and performing automatic layout only within the additional double-page spread. Although the same Event 1 image is arranged in the additional spread and the adjacent N spread, the time series between the N spread and the additional spread is broken. On the other hand, FIG. 24C shows the layout result processed in this embodiment. A layout change range determination unit 1701 specifies a change range of an existing spread, and a spread assignment unit 1702 performs spread allocation including the existing spread. After that, the automatic layout unit 1703 automatically lays out each spread, thereby providing a layout in which the existing spread and the additional spread are arranged in chronological order.

FIG. 18 is a flow chart for executing automatic layout by combining additional spreads and existing spreads. Since the same numbers as in FIG. 16 represent the same devices and perform the same processing, the description is omitted here.

In S<b>1801 , the layout change range determination unit 1701 determines the range of changing the layout of the existing double-page spread based on the image selected by the automatic image selection unit 1508 . For example, in the example of FIG. 24, the photographing date/time information (2-16) of the image arranged in the N spread and the photographing date/time information (1-15) of the image arranged in the additional spread overlap in FIG. 24(b). are doing. On the other hand, it is assumed that the shooting date/time information of the images arranged in the N+1 spread does not overlap with the shooting date/time information of the images arranged in the additional spread. In this case, in S1801, N-page spreads are specified as the range for changing the layout of the existing two-page spread.

In step S1802, the spread assignment unit 1702 changes the group of images arranged in the existing spread determined by the layout change range determination unit 1701 to be changed and the group of images selected for the additional spread based on the shooting date and time. Divide into sections equal to the number of spreads. Then, the spread allocation unit 1702 allocates the divided image to each spread to be changed in units of sections. For example, in the example of FIG. 24, the N-page spread and the additional page-spread are targeted, so they are divided into two sections.

In S1803, the automatic layout unit 1703 automatically lays out each spread using the images assigned to the spread by the spread assigning unit 1702 for each spread within the change range. In FIG. 24, automatic layout is executed for the N spread and the additional spread.

In S1804, it is determined whether or not automatic layout has been performed for all double-page spreads determined by the layout change range determination unit 1701. If it has been performed, the process ends. If not, the process returns to S1803 to continue processing for the two-page spread for which automatic layout has not been performed.

The above is a description of the flow chart for executing automatic layout by combining additional spreads and existing spreads.

In this embodiment, the additional and existing spreads to which the images are assigned are changed, and the additional and existing spreads containing the images to which the spread allocation has been changed are automatically laid out, thereby matching the existing and additional spreads. A highly flexible layout can be realized.

(Embodiment 3)
The third embodiment will describe a method for switching the execution of spread allocation when performing automatic layout by combining additional spreads and existing spreads.

FIG. 19 is a block diagram for switching the execution of spread allocation when performing automatic layout by combining additional spreads and existing spreads. Since the same numbers as in FIG. 17 represent the same devices and perform the same processing, the description is omitted here.

A spread change determination unit 1901 determines whether or not to change allocation of spreads when automatically laying out additional spreads. For the sake of explanation, it is determined whether or not the shooting time of the image selected for the additional spread is within the shooting time interval of the images arranged in the existing spread in the front and back spread adjacent to the additional spread. do. If the shooting date and time of the image selected in the additional spread does not fall within the shooting time interval of the existing spread, the assignment of the spread is not changed. On the other hand, if the shooting date and time of the image selected in the additional spread overlaps the shooting time interval of the existing spread, the allocation of the spread is changed.

If the allocation of spreads is not changed, an additional spread is automatically laid out by the additional spread automatic layout unit 1509 using the images selected by the automatic image selection unit 1508 according to the procedure described in the first embodiment. Then, a layout information acquisition unit 1510, a layout change determination unit 1511, and an existing spread automatic layout unit 1512 appropriately change the layout of the existing spread.

When changing the allocation of spreads, the layout change range determination unit 1701, the spread allocation unit 1702, and the layout unit 1703 perform automatic layout for the additional spreads and the existing spreads within the change range according to the procedure described in the second embodiment. implement.

Here, for the sake of explanation, the determination of the spread to be changed is limited to adjacent spreads, but other than the existing adjacent spreads may be subject to determination. For example, if an existing spread includes an image of a related event that has features common to the image selected in the additional spread, other than the adjacent spread may be included in the layout change range determination target.

Further, when the allocation of spreads is changed, if the allocation of spreads is delimited at a place where the shooting time difference is large, the difference in the number of images may become larger than the average number of images. In that case, the double-page spread may be divided so as to be close to the average number of images instead of the location where the shooting time difference is large. If the number of images per double-page spread exceeds the average number of images, the images selected by the automatic image selection unit 1508 may be deselected starting with the lowest score up to the average number of images. If the number of images per spread is less than the average number of images, additional images may be selected within the assigned shooting time interval.

The above is a description of the block diagram for switching the execution of spread assignment when performing automatic layout by combining additional spreads and existing spreads.

Next, a description will be given of a flow for switching the allocation of spreads when performing automatic layout by combining additional spreads and existing spreads.

FIG. 20 is a flowchart for switching the execution of spread allocation when performing automatic layout by combining additional spreads and existing spreads. Since the same numbers as in FIG. 18 represent the same devices and perform the same processing, the description is omitted here.

In S2001, the spread change determination unit 1901 determines whether or not to change the allocation of spread images when automatically laying out additional spreads. As a result of the determination, if the spread assignment is not to be changed, the process branches to S1610 to execute processing. If the spread assignment is to be changed, the process branches to S1801 to execute processing.

By executing automatic layout at each branch, spreads including additional spreads can be laid out automatically.

The above is a description of the flow for switching the execution of spread assignment when performing automatic layout by combining additional spreads and existing spreads.

In this embodiment, by switching the execution of spread allocation, it is possible to realize an optimum layout according to the images to be arranged.

(Other embodiments)
The invention is not limited to the embodiments described above. For example, in the above-described embodiment, an album creation application was described as an example, but an application or an image processing apparatus having image selection processing for automatically selecting recommended images may be used.

Also, in FIG. 1, the image processing apparatus assuming a local environment has been described as an example, but the image processing apparatus is not limited to this. For example, it may be an image processing device on a server. In this case, an image stored in the server may be used or an image may be uploaded.

In the above-described embodiment, the images are assumed to be photographs, and the images are grouped based on the date and time when they were taken.

In the above-described embodiment, the mode is specified by the user, and the arrangement is made based on the specified mode. The mode may be set automatically from the distribution of . The present invention can also be realized by executing the following processing. That is, the software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program. This is the process to be executed. Also, the program may be executed by one computer or may be executed by interlocking a plurality of computers. Further, it is not necessary to implement all of the above-described processes by software, and part or all of them may be implemented by hardware.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Claims (10)

display control means for displaying album data including an existing page on which a plurality of images are arranged;
additional means for adding additional pages to the album data;
layout means for laying out a plurality of image data including image data selected by a user for the additional page;
An image processing apparatus comprising a changing unit that changes the arrangement of the plurality of image data on the existing page based on the layout result of the plurality of image data for the additional page and the layout result of the plurality of image data on the existing page. 2. The image processing apparatus according to claim 1, wherein said changing means changes components of said existing page and said additional page. When the plurality of image data whose photographing date and time are within a predetermined time range are not arranged in the order of photographing date and time across the existing page and the additional page, the changing means changes the plurality of image data from the predetermined existing page. 3. The image processing apparatus according to claim 2, wherein the image arranged on the existing page and the image arranged on the additional page are exchanged so that the images arranged on the additional page are arranged in order of shooting date and time. 4. The method according to any one of claims 1 to 3, wherein said changing means changes the arrangement of said plurality of image data on said existing page by changing a template used for said existing page. The described image processing device. When it is determined that the number of image data arranged in the template used in the additional page is the same as the number of the plurality of image data in the existing page, the changing means changes the number of the plurality of image data in the existing page. 5. The image processing apparatus according to any one of claims 1 to 4, wherein the arrangement of image data is changed. further comprising additional position setting means for setting an additional position for adding the additional page in the album data based on a user input;
6. The image processing apparatus according to any one of claims 1 to 5, wherein the existing page is an existing page within a predetermined number of pages from the addition position. 6. The image processing apparatus according to any one of claims 1 to 5, wherein the existing page contains image data having a predetermined characteristic in common with the image arranged on the additional page. 7. The image processing apparatus according to any one of claims 1 to 6, wherein the additional pages are added in units of double-page spreads consisting of two pages facing each other when opened. displaying album data including an existing page with a plurality of images;
adding additional pages to the album data;
laying out a plurality of image data including image data selected by a user for the additional page;
An image processing method comprising: changing the arrangement of the plurality of image data on the existing page based on the layout result of the plurality of image data for the additional page; and the layout result of the plurality of image data on the existing page. A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 8.

Images