JP2020204944A - Apparatus, method and program - Google Patents

Abstract

To solve a problem in which layout processing considering a nozzle of a printing apparatus is not performed at the time of layout of image data.SOLUTION: An apparatus includes: layout means that determine a layout method of image data to pages; identification means that identifies information on a nozzle of a printing apparatus which is not used in printing processing of the printing apparatus based on the determined layout method; and determination means that determines whether or not to change the layout method based on the identified information on the nozzle.SELECTED DRAWING: Figure 2

Description

The present application relates to devices, methods and programs, and more specifically to devices, methods and programs for creating albums.

In recent years, in addition to the spread of digital cameras, the number of photographs taken by users has increased sharply due to the spread of smart devices and the improvement of camera performance of smart devices. As the number of shots increases, so does the variation of the subject to be shot.

Patent Document 1 is disclosed as a method of creating an album by performing automatic layout on a plurality of photographs taken by a digital camera.

JP-A-2010-108165

In an inkjet printer, if the recording head has a nozzle that has not been used for a long time, it is necessary to take measures such as preliminary ejection to avoid ink sticking. However, Patent Document 1 does not disclose layout processing in consideration of the use and non-use of nozzles, which may reduce user convenience.

One embodiment of the present invention specifies a layout means that determines a method of laying out image data for a page and specifies information about nozzles of the printing device that are not used in the printing process of the printing device based on the determined layout method. The apparatus is characterized by comprising means and a deciding means for determining whether or not to change the layout method based on the specified information about the nozzle.

According to the present application, the convenience of the user can be improved.

It is a block diagram which shows the hardware structure of the image processing apparatus in 1st Embodiment. It is a block diagram which shows the software structure of the image processing apparatus in 1st Embodiment. It is a figure which shows the GUI of the album creation application in 1st Embodiment. It is a flowchart of automatic layout processing in 1st Embodiment. It is a figure explaining the image analysis information in 1st Embodiment. It is a figure explaining the image group division in 1st Embodiment. It is a figure explaining the scene classification in 1st Embodiment. It is a figure explaining the image scoring in 1st Embodiment. It is a figure explaining image selection in 1st Embodiment. It is a figure explaining the creation of the basic layout in 1st Embodiment. It is a figure explaining the creation of the basic layout in 1st Embodiment. It is a figure explaining the unused nozzle in 1st Embodiment. It is a figure explaining the complementary layout in 1st Embodiment. It is a flowchart of automatic layout processing in 2nd Embodiment.

[First Embodiment]
In the present embodiment, the image processing device operates an application for creating an album (hereinafter, also referred to as "application" or simply "application"). This application creates layout information for outputting an album in which multiple photographic images are automatically laid out. Then, by recording using this layout information, an album as a printed matter is created.

<About the hardware configuration of the image processing device>
Hereinafter, the hardware configuration of the image processing apparatus according to the present embodiment will be described with reference to FIG. In the following, the case where the image processing device is an information processing device (PC) will be described, but another device such as a smartphone may be adopted as the image processing device.

As shown in FIG. 1, the image processing device 100 includes a CPU 101, a ROM 102, a RAM 103, an HDD 104, a display 105, a keyboard 106, a mouse 107, and a data communication device 108. These components are connected by a data bus 109 and can transmit or receive data to and from each other.

The CPU 101 controls the entire image processing device 100. Further, the CPU 101 executes the image processing method described in the present embodiment according to a program. In FIG. 1, the image processing device has one CPU, but the image processing device may have a plurality of CPUs.

The ROM 102 stores a program executed by the CPU 101. The RAM 103 provides a memory for temporarily storing various information when the program is executed by the CPU 101. The HDD 104 stores a database or the like that holds processing results such as image files and image analysis, and in the present embodiment, the HDD 104 stores an application program for creating an album. This application program is sometimes called an album creation application, and will be described later with reference to FIG. 3 and the like.

The display 105 is a device for displaying and presenting the layout result of the user interface (hereinafter referred to as UI) and the image of the present embodiment to the user. The display 105 may have a touch sensor function. The keyboard 106 is one of the input devices included in the image processing device 100, and is used, for example, to input predetermined information on the GUI displayed on the display 105. In this embodiment, the user inputs the number of spreads of the album via the keyboard 106. The mouse 107 is one of the input devices included in the image processing device 100, and is used, for example, to click and press a button on the GUI displayed on the display 105.

The data communication device 108 is a device for communicating with an external device such as a recording device or a server. For example, the information about the image layout acquired as a result of the automatic layout is transmitted to the recording device or the server connected to the image processing device 100 via the data communication device 108. The data bus 109 connects each of the above-described components to the CPU 101. The above is the contents of the hardware configuration of the image processing device in the present embodiment.

<About the software configuration of the image processing device>
Hereinafter, the software configuration of the image processing device in the present embodiment, in other words, the functional configuration realized by the album creation application installed in the image processing device will be described with reference to FIG. The album creation application is started by the user double-clicking the application icon displayed on the display 105 using the mouse 107 or the like. Although the album creation application has various functions, the automatic layout function provided by the automatic layout processing unit 210, which is a particularly important function in the present embodiment, will be mainly described below.

As shown in FIG. 2, this application has an album creation condition setting unit 201, an automatic layout processing unit 210, and an album display unit 202. The "automatic layout function" is a function for creating layout information by classifying or selecting an image of a photograph taken based on its contents and attributes and then laying it out. The album image based on the created layout information is displayed on the display 105.

The album creation condition setting unit 201 sets the album creation condition according to the UI operation by the mouse 107 in the automatic layout processing unit 210. In the present embodiment, the image group used for the album, the number of spreads of the album, the mode for creating the album, and the product for creating the album are set as the album creation conditions. The image group setting may be set using the accompanying information or attribute information of individual image data such as the shooting date and time, or is based on the structure of the file system in which the image data is stored, such as the device and directory settings. May be set. The "spread" corresponds to, for example, one display window in display, and corresponds to a pair of adjacent pages printed on different sheets in printing.

The image acquisition unit 211 acquires an image group (image data group) that satisfies the album creation condition set by the album creation condition setting unit 201 from the images stored in the HDD 104. The image group referred to here refers to a candidate image group that is a candidate to be arranged in a spread when creating one album. For example, when the album creation condition setting unit 201 specifies January 1, XX to December 31, XX, all the images taken from January 1, XX to December 31, XX are all taken. , Corresponds to the image group of layout candidates. Examples of the image stored in the HDD 104 include a still image and a cut-out image cut out from a moving image. Still images and cropped images are acquired from imaging devices such as digital cameras and smart devices. The image pickup device may be provided by the image processing device 100, or may be provided by an external device of the image processing device 100. When the imaging device is an external device, the image acquisition unit 211 acquires an image via the data communication device 108. Further, the still image or the cut-out image may be an image acquired from an external network or server via the data communication device 108. Examples of the image acquired from the network or the server include a social networking service image (hereinafter, SNS image). By executing the OS program, the CPU 101 analyzes the data associated with the image for each image and obtains the storage source which is the acquisition destination of the image. However, by acquiring an image from the SNS via the application, the acquisition destination of the image may be managed within the application. The image acquired by the image acquisition unit 211 is not limited to the above-mentioned image, and may be another type of image.

The image analysis unit 212 executes an analysis process on the image data acquired by the image acquisition unit 211. In the present embodiment, the feature amount of the image is acquired, and object determination in the image, face detection, facial expression recognition of the detected face, and personal recognition of the detected face are executed. Further, the image analysis unit 212 acquires information on the shooting date and time by referring to the data (for example, Exif information) associated with the image acquired from the HDD 104. The information obtained as a result of the image analysis unit 212 analyzing the image data is referred to as "image analysis information".

The image classification unit 213 executes scene division for classifying the image group acquired by the image acquisition unit 211 for each scene by using the image analysis information derived by the image analysis unit 212. This scene division is executed using information on the shooting date and time, information on the number of acquired images, and information on the detected face. An image of a scene can be, for example, a collection of images (sub-image group) taken at a temporary shooting opportunity for a certain shooting target, and such shooting opportunities include, for example, travel, daily life, wedding, etc. Can be mentioned.

The image scoring unit 214 scores each image based on the album creation conditions set by the album creation condition setting unit 201 and the image analysis information derived by the image analysis unit 212. In the present embodiment, each image is scored so that the image suitable for the layout has a high score. When scoring an image, image information including image analysis information, image classification results, and album creation conditions are used. However, other information may be used additionally or as an alternative. An image suitable for layout includes, for example, an image having high aesthetics such as high contrast and sharp edges.

The spread allocation unit 215 divides the image group and allocates it to each spread. The spread referred to here corresponds to, for example, a spread in an album, that is, two pages. Although the form of allocating the image group to the spread is described here, the form of allocating the image group to the page may be used. The spread allocation unit 215 divides the image group based on the input number of spreads and the image classification result, and allocates a part of the image group to each spread. For example, when the number of input spreads is 5, the acquired image group is divided into 5 small image groups, and one small image group is assigned to each spread.

The image selection unit 216 selects an image to be used for the album based on the score given by the image scoring unit 214 in the small image group assigned to each spread by the spread allocation unit 215.

The layout creation unit 217 creates a spread layout. Specifically, the layout creation unit 217 determines the arrangement position of each image based on the image selected by the image selection unit 216 and the unused nozzle determined by the unused nozzle determination unit 219.

The layout information output unit 218 outputs layout information according to the layout created by the layout creation unit 217. The layout information is, for example, bitmap format image data for drawing a spread in which each image is arranged at a designated position. The image data output from the layout information output unit 218 is displayed on the display 105 when a user instruction is input via the GUI as shown in FIG. 3, for example.

The unused nozzle determination unit 219 determines which nozzle of the recording device is not used when creating an album, based on the layout information output from the layout information output unit 218.

The album display unit 202 displays the album image on the display 105 based on the bitmap format image data output from the layout information output unit 218.

When the program of the album creation application in this embodiment is installed in the image processing device 100, the start icon of this application is displayed on the top screen (desktop) of the OS (operating system) running on the image processing device 100. To. When the user double-clicks the startup icon on the desktop displayed on the display 105 with the mouse 107, the program of the album creation application stored in the HDD 104 is loaded into the RAM 103. Then, the program loaded in the RAM 103 is executed by the CPU 101, and the album creation application is started. The above is the contents of the software configuration of the image processing device in the present embodiment. The application may be in another form, for example, a Web application that displays a screen or the like in a browser operating on the image processing device 100.

<About GUI>
Hereinafter, the GUI of the album creation application in this embodiment will be described with reference to FIG. FIG. 3 is a diagram showing GUI 301 provided by the activated album creation application and displayed on the display 105. The user can set the album creation condition via GUI301.

The GUI 301 has a pass box 302 and a folder selection button 303 as setting units for photographic images to be included in the album. The pass box 302 is a box for indicating the storage location (path) of the image group to be created in the album in the HDD 104. The folder selection button 303 is a button for selecting a folder containing an image group to be created as an album. When the user clicks the folder selection button 303 with the mouse 107, a tree composed of a plurality of folders is displayed. Then, when the user selects a folder containing the image group to be created as an album, the folder path of the selected folder is displayed in the pass box 302.

The spread number box 304 is a box for designating the spread number per album. The user can use the keyboard 106 to enter numbers directly into the spread box 304, or use the mouse 107 to enter numbers from the list into the spread box 304.

In the mode designation unit 307, a plurality of illustration images showing the objects to be considered when creating an album are displayed as icons. The user can specify the object to be considered by selecting an arbitrary icon from the plurality of displayed icons. In this embodiment, a case where three objects of "human", "animal", and "cooking" are adopted as objects that may be considered will be described. However, objects other than this may be adopted, and the object to be considered may be selected from three or more objects. It is also possible to specify multiple objects at the same time, but here it is assumed that one object is selected for the sake of brevity. The user specifies the object to consider by clicking the icon with the mouse 107.

The spread photo number setting unit 309 is a slider bar for setting the number of images to be arranged in each spread of the album to be created. When the slider is moved to the "many" side, the number of images placed in each spread increases. On the other hand, when the slider is moved to the "small" side, the number of images arranged in each spread decreases.

The product designation unit 310 is an element for setting the product of the album to be created. As specific items related to the product to be set, the size of the album and the paper type of the album can be adopted. You may set the cover type and the binding part type.

The OK button 311 is a button for determining the selected condition as an album creation condition. When the user clicks the OK button 311 with the mouse 107, the album creation condition is confirmed, and the album creation condition is transmitted to the automatic layout processing unit 210 via the album creation condition setting unit 201. More specifically, the path information input to the pass box 302 is transmitted to the image acquisition unit 211. Further, the value of the number of spreads input in the number of spreads box 304 is transmitted to the spread allocation unit 215.

The reset button 312 is a button for resetting each setting content on the display screen. The above is the contents of the GUI of the album creation application in this embodiment.

<About automatic layout>
Hereinafter, the automatic layout in this embodiment will be described with reference to FIG. 4A. FIG. 4A is a flowchart of a process for executing the automatic layout of the album creation application according to the present embodiment. The flowcharts shown in FIGS. 4A to 4D are realized, for example, by the CPU 101 reading the program stored in the HDD 104 into the ROM 102 or the RAM 103 and executing the program. In the present embodiment, when creating an album, the image group for creating the album is divided according to the shooting time of the image, and the page is divided from each sub-image group (hereinafter, also referred to as "section"). Select the image to place in.

In step S401, the album creation condition setting unit 201 sets the album creation condition. In the following, "step S ~" is simply abbreviated as "S ~". In the present embodiment, the image group used for the album, the number of spreads, the mode for creating the album, and the product for creating the album are set as the album creation conditions.

In S402, the image acquisition unit 211 reads an image satisfying the conditions set by the album creation condition setting unit 201 from the HDD 104 and develops it in the RAM 103.

In S403, the image analysis unit 212 analyzes the image data expanded in the RAM 103. The analysis of this step will be described with reference to FIG. 4 (b).

In S4031, the image analysis unit 212 refers to the information associated with the image data acquired by the image acquisition unit 211, and acquires the information on the shooting date and time corresponding to the image data. In the present embodiment, information on the shooting date and time is acquired from the Exif information attached to each image data.

In S4032, the image analysis unit 212 derives the feature amount of the image quality with respect to the image data acquired by the image acquisition unit 211. Examples of the feature amount of image quality include the focus amount. An edge detection method can be used as a focus amount determination method, and a known sobel filter can be used as an edge detection method. The edge of the image is detected by the Sobel filter, and the brightness gradient (that is, the slope of the edge) is calculated by dividing the brightness difference between the start point and the end point of the edge by the distance between the start point and the end point. By calculating the average slope of the edges in the image, it can be determined that the image having a large average slope is in focus more than the image having a small average slope. In the present embodiment, a plurality of inclination threshold values for measuring the magnitude of the average inclination of the edge are set, and by determining which inclination threshold or more the calculated average inclination of the edge is equal to or greater than, it is determined whether the focus amount is acceptable. Specifically, a first tilt threshold value and a second tilt threshold value (note that the first tilt threshold value> the second tilt threshold value) are set as two different tilt threshold values, and the focus is set in three stages of ○ △ ×. Determine the amount. If the average inclination of the edges in the image is equal to or greater than the first threshold value, a suitable focus amount is set (indicated by ◯). If the average slope of the edges in the image is less than the first threshold value and greater than or equal to the second threshold value, the focus amount is acceptable (indicated by Δ), and the average slope of the edges in the image is less than the second threshold value. If there is, the focus amount is unacceptable (indicated by x).

In S4033, the image analysis unit 212 detects objects and classifies the detected objects with respect to the image data acquired by the image acquisition unit 211. In this embodiment, the face is detected. Any known method can be adopted as the face detection method. As such a known method, for example, AdaBoost in which a strong classifier is created from a plurality of weak classifiers prepared can be mentioned. In this embodiment, a person's face is detected using a strong classifier created by AdaBoost. In S4033, the face is detected, and the upper left coordinate value and the lower right coordinate value of the detected face region in the image are acquired. By obtaining these two types of coordinate values, the position of the face and the size of the face can be specified. Furthermore, by also executing AdaBoost, which detects animals such as dogs and cats and food, as well as the face of a person, objects of people, animals, and food are detected, and at the same time, what are the objects in the image? Can be classified. The object to be detected is not limited to the above, and may be a flower, a building, a figurine, or the like.

As shown in FIG. 5, the image analysis information as the analysis result of each image data acquired in S4031 to S4033 is stored in the HDD 104 separately for each ID that identifies the image. The values related to the main objects contained in the image are retained from the one with the highest reliability at the time of AdaBoost execution. On the other hand, the values for undetected objects are not retained. AdaBoost continuously connects weak discriminators that discriminate patterns to form one strong discriminator. Therefore, the reliability increases as the number of weak discriminators that match the pattern learned in advance by AdaBoost increases. Here, although the object determination has been described by taking AdaBoost as an example, a discriminator using a ConvolutionalNeuralNetwork such as DeepNeuralNetwork may be used.

Returning to the description of FIG. 4 (a). In S411, the image analysis unit 212 determines whether the processing of S402 to S403 is completed for all the images of the image group satisfying the conditions set in S401. If the determination result of this step is true, the process proceeds to S412, while if the determination result is false, the process returns to S402.

In S412, the image classification unit 213 executes scene division and scene classification for an image group that satisfies the conditions set by the album creation condition setting unit 201. “Scene division” refers to dividing the acquired image group into a plurality of sub image groups according to the scene. Specifically, using the information on the shooting date and time acquired in S4031, the image group is divided into a plurality of sub-image groups based on the time difference between the images. An example of the actual division is as follows.

Among the image groups, for example, the image with the oldest (or newest) shooting date and time is first focused on, the time difference from the next oldest (or newer) image is calculated, and whether or not the calculated time difference is equal to or greater than a predetermined threshold value. judge. Such processing is performed on all images while sequentially replacing the images of interest with new (or old) images. The "division" in S412 means that the image group is divided into the newer one and the older one in the shooting date and time at the boundary between the two images. If the shooting date of one image and the shooting date of another image are not the same and are not continuous, that is, if there is a non-shooting day between the images, the image is divided.

Next, a case where the shooting dates are continuous will be described. In the present embodiment, when the difference between the shooting date and time of the image of interest and the shooting date and time of the next oldest (or new) image is 16 hours or more, it is divided. When the difference in shooting time between images is less than 16 hours, if the time difference between the first shooting and the last shooting of each consecutive day is less than 4 hours, the image is divided. On the other hand, when the time difference from the first shooting to the last shooting of each consecutive day is 4 hours or more, if the number of shots taken each consecutive day is less than 50, it is divided, and if it is 50 or more, it is not divided. And. The time difference threshold value and the image number number threshold value used at the time of division are not limited to this. FIG. 6A shows an example of the result of dividing the image group by using the above-mentioned scene division method. That is, it is shown that it is divided into eight sub-image groups.

After the scene division, the image classification unit 213 classifies the scene. In the present embodiment, a case where each sub-image group (each scene) is classified into three categories of travel, daily life, and ceremony will be described as an example, but the classification items are not limited to these.

First, a plurality of sub-image groups determined in advance as to which of travel, daily life, and ceremony are classified are acquired, and the feature amount of photography for each of the acquired plurality of sub-image groups is acquired. The feature quantities acquired in this embodiment are, for example, the shooting period, the number of shots, and the number of shots. The shooting period is the time difference between the shooting time of the oldest image and the shooting time of the newest image among the images included in the sub-image group. The number of shots is the number of images (the number of pictures) included in the sub-image group. The number of photographed persons is the number of faces in an image showing faces, and is the number of faces included in one image. Then, the mean value and standard deviation of the shooting period, the mean value and standard deviation of the number of shots, and the mean value and standard deviation of the number of people per image are obtained for the plurality of sub-image groups. In the present embodiment, the number of faces per image is the number of people per image.

FIG. 7 shows an example of the average and standard deviation of the shooting period (time), the average and standard deviation of the number of shots, and the average and standard deviation of the number of people per image obtained for a plurality of sub-image groups. These obtained values are incorporated in the program of the album creation application in advance. That is, at the design stage of the application, parameters are generated by learning using images collected in advance, and the generated parameters are incorporated into the program. After the album creation application is started, the shooting period, the number of shots, and the number of people per image are determined for each sub-image group obtained as a result of scene division for the image group specified by the user via the pass box 302. Calculate the average value of each. Then, for each scene, the parameters (specifically, the average value and standard deviation of the shooting period, the number of shots, and the feature amount of the number of shots of each sub-image group) built into the above-mentioned application are used. , Calculate the score according to the following formula.

From the formula (1), the score for each scene and each feature amount of the sub-image group of interest is obtained. For example, for a travel scene, the shooting period, the number of shots, and the number of shots are required. These scores are averaged by the formula (2) to obtain the score for the travel scene. Similarly, you can get points for other scenes such as daily life and ceremonies. By the method described above, the average score of the travel scene, the average score of the daily scene, and the average score of the ceremony scene are calculated for each sub-image group. In this embodiment, the number of feature quantity items in the formula (2) is 3.

Of the scores for each scene calculated for each sub-image group, the scene with the highest score is specified as the scene for that sub-image group. If two or more scenes are tied, the scenes are classified according to the predetermined priority. For example, in the present embodiment, priority is given in the order of daily life> ceremony> travel, and the priority of daily life is the highest. The order of priorities is not limited to this, and the user may be able to change the order of priorities.

Sub-image groups (1) to (8) in FIG. 6A show sub-image groups obtained by dividing the image group into scenes, respectively. Of these sub-image groups, it is assumed that the sub-image group (5) has a shooting period of 36 hours, the number of shots is 300, and the number of shots is 1.7. On the other hand, the average score of travel is 45.32, the average score of daily life is 18.38, and the average score of ceremonies is -29.92. In that case, the scene of the sub-image group (5) is classified as a trip. The classified sub-image group is managed by associating the scene ID so that the scene can be identified.

In S416, the image scoring unit 214 scores each image. This process is called image scoring. The "score" derived in this step is a parameter for evaluating the suitability for use in an album, and a score evaluated from the viewpoint described later is given to each image data, which is referred to when selecting an image described later. Hereinafter, the method of scoring an image will be described with reference to FIGS. 8 (a) and 8 (b).

FIG. 8A shows the criteria for scoring image data. More specifically, FIG. 8A is a table summarizing the characteristics of images in which the score is high for each of the three scenes of travel, daily life, and ceremony. As shown in FIG. 8A, the feature of increasing the score differs depending on the type of scene. For example, if the scene classification of the sub-image group is travel, the subtracted image containing the person and the landscape will have a higher score. Further, for example, when the scene classification of the sub image group is daily, the score is high for the close-up of the face and the profile image. Further, for example, when the scene classification of the sub image group is a ceremony, the score is high for the images in which the two people are close to each other.

In this way, the characteristics of high scores, in other words, the evaluation criteria, differ depending on the type of scene. In the present embodiment, the feature that the score increases according to the type of the scene is preset in the application and is included in the program. The feature of increasing the score is not limited to those described above. In the present embodiment, a score is given to each image data included in each sub-image group based on the characteristics of each scene shown in FIG. 8A. Using the number of faces in each image acquired in S4033, the position of the face in each image, and the size of the face in each image, the mean value and the standard deviation were obtained for each scene type, and the obtained value was obtained. Remember according to the program of the album creation application. Which scene each image of the image group specified by the user belongs to (classified) is specified from the result of the scene classification of S412. Then, based on the mean value and standard deviation obtained in advance corresponding to the scene of the attention image, and the feature quantities of the number of faces, the position of the face, and the face size in the attention image, the following equations (3) and equations (3) and equations are used. Using (4), the score and the average score are calculated. The score, mean value, and standard deviation in the following formula are obtained for each scene and each feature amount.

Further, points are added to the score calculated by using the above formula according to the mode specified by the album creation condition setting unit 201. For example, when the album creation mode is set to person, for a certain image, if the values of TOP1 to TOP3 of the object classification held in the table of FIG. 5 include the reliability of the person in the category. , Add points to the score of the certain image. At this time, the higher the reliability of the object classification result, the more points may be added. In addition, the score to be added may be changed according to the rank of the object classification result (that is, any of TOP1 to TOP3). Here, for the sake of explanation, the case where a person is selected in the album creation mode has been described, but other modes such as animals and cooking may be used. Further, the number to be selected is not limited to one, and two or more may be selected. In that case, points will be added to the score as much as the selected object is included. Further, for the image whose focus feature amount shown in FIG. 5 is ◯, points may be added to the score calculated by using the above formula. As a result, the score of the in-focus image can be increased. FIG. 8B shows a table in which the scores for each image ID are held as an example of the results of scoring each image.

In the present embodiment, the image analysis result of S403 and the image classification result of S412 are used for scoring, but the ones used are not limited to this. For example, the image size may be used, the shooting information such as the lens information used at the time of shooting may be used, or the compression format information of the image input to the application may be used.

In S417, the image scoring unit 214 determines whether the image scoring of S416 is completed for all the images in the user-designated image group. If the determination result of this step is true, the process proceeds to S418, while if the determination result is false, the process returns to S416.

In S418, the spread allocation unit 215 allocates all the images of the user-specified image group to each spread in the album. FIG. 4C is a detailed flowchart of the allocation process in this step.

In S4181, the spread allocation unit 215 determines whether the number of scenes (the number of sub-image groups) obtained by the scene division of the image classification unit 213 is the same as the number of spreads set by the album creation condition setting unit 201. If the determination result of this step is true, the process proceeds to S4185, while if the determination result is false, the process proceeds to S4182. For example, if the number of scenes is 8 and the number of spreads is 8 as shown in FIG. 6A, the process proceeds to S4185.

In S4182, the spread allocation unit 215 determines whether the number of scenes is less than the number of spreads set by the album creation condition setting unit 201. If the determination result of this step is true, the process proceeds to S4183, while if the determination result is false, the process proceeds to S4184. For example, if the number of scenes is 8 and the number of spreads is 10, as shown in FIG. 6A, the number of scenes is not less than the number of spreads, so the process proceeds to S4183.

In S4183, the spread allocation unit 215 divides the sub-scene. "Sub-scene division" refers to further subdividing the divided scenes when the number of scenes is smaller than the number of spreads. A case where the number of scenes as shown in FIG. 6A is 8 and the number of designated spreads is 10 will be described as an example. FIG. 6B shows the result of sub-scene division in the state of FIG. 6A. The number of scenes is set to 10 by further dividing the sub-image group at the locations of the dashed arrows 601 and 602 in the figure. Such sub-scene division is performed using the number of images and the shooting date and time as a reference. In the sub-scene division from the state shown in FIG. 6A, a sub-image group having a large number of images is specified. Here, since we want to increase the number of scenes by two from 8 to 10, we specify two sub-image groups with a large number of images. That is, the number of scenes to be subdivided is specified by the number of scenes (here, 2) to be increased in order from the scene containing the most images. When there are multiple scenes with the same number of images, the scene with the larger maximum difference in shooting date and time between the images is selected. If the scene to be divided cannot be determined even after that, a rule may be determined in advance, for example, a scene having an earlier time is preferentially divided. In FIG. 6A, the sub-image group having the largest number of images is the sub-image group (5) from the largest number, followed by the sub-image group (1) and the sub-image group (2). The number of images in the sub image group (1) is equal to the number of images in the sub image group (2). In the present embodiment, when the number of images included in the sub-image group is the same, the sub-image group having a larger time difference between the first image and the last image included in the sub-image group is divided. The sub-image group (1) and the sub-image group (2) contain the same number of images, but the sub-image group (2) has a larger time difference between the first image and the last image. (2) is the target of division. As described above, in the example of FIG. 6A, the sub-image group (5) and the sub-image group (2) are further divided.

First, the division of the sub-image group (2) will be described. As shown in the figure, the sub-image group (2) has two peaks of the number of images, and it is assumed that these two have different shooting dates. In this case, as shown by the broken line arrow 601, it is divided at a place where the shooting date changes. Next, the division of the sub-image group (5) will be described. The sub-image group (5) has three peaks of the number of images, and it is assumed that these three have different shooting dates. Also in this case, it is divided at the place where the shooting date changes. At this time, there are two places where the shooting date changes, but the images are divided so that the difference in the number of images of each image group after division becomes small. Regarding the sub-image group (5) of FIG. 6A, each division after the division at the boundary between the second day and the third day is more effective than the division at the boundary between the first day and the second day. Since the difference in the number of image groups becomes small, it is divided at the boundary between the second day and the third day. That is, it is divided at the portion indicated by the broken line arrow 602. The number of scenes is changed from 8 to 10 by the sub-scene division described above. Here, the images are divided at different locations on the shooting date, but if the sub-image group with a large number of images was shot on a single day, it is divided at the location with the largest time difference in the single day. ..

In S4184, the spread allocation unit 215 integrates the scenes. "Scene integration" refers to integrating divided scenes (sub-image groups) when the number of scenes is larger than the number of spreads. Specifically, the number of scenes is reduced so that the number of scenes and the number of spreads are equal. A case where the number of scenes is 8 and the number of designated spreads is 6 as shown in FIG. 6A will be described as an example. FIG. 6C shows the result of scene integration in the state of FIG. 6A. By integrating the parts indicated by the broken lines 603 and 604 in FIG. 6C, in other words, the number of scenes (the number of sub-image groups) is set to 6 by not dividing the parts indicated by the broken lines. In scene integration, sub-image groups with a small number of images are specified as integration targets. Here, it is necessary to specify two sub-image groups having a small number of images for the purpose of reducing the number of scenes by two from 8 to 6. In FIG. 6A, the sub-image group with the smallest number of images is the sub-image group (8), followed by the sub-image group (3) and the sub-image group (7). The number of images in the sub image group (3) is equal to the number of images in the sub image group (7). Therefore, first, the sub-image group (8) having the smallest number of images is targeted for integration. The sub-image group (3) and the sub-image group (7), which are considered as another integration target, contain the same number of images, but the sub-image group (8) adjacent to the sub-image group (7) is integrated. Since it is a target, the sub-image group (3) that is not adjacent to the sub-image group (8) is targeted for integration.

After determining the sub-image group to be integrated, it is determined whether the sub-image group to be integrated is integrated into the sub-image group whose shooting date and time is earlier or the sub-image group whose shooting date and time is later. First, the integration of the sub-image group (3) of FIG. 6A will be described. Comparing the time difference between the sub-image group (3) and the previous sub-image group (2) and the time difference between the sub-image group (3) and the subsequent sub-image group (4), the latter Is smaller than the former. Therefore, it is determined that the sub-image group (3) is integrated with the sub-image group (4). Then, the boundary between the different sub-images at the location of the broken line 603 disappears, and the sub-image groups are integrated. Next, the integration of the sub-image group (8) will be described. Since the sub-image group (8) does not have a sub-image group after this, it is determined to integrate with the sub-image group (7) before the sub-image group (8). Then, the boundary between the different sub-images at the location of the broken line 604 disappears, and the sub-image groups are integrated. In the present embodiment, the sub-image groups having a small difference in shooting time are integrated, but the method for integrating the sub-image groups is not limited to this. For example, the sub-image group to be integrated may be integrated into the sub-image group having a small number of images.

In S4185, the spread allocation unit 215 performs spread allocation. By the processing of S4181 to S4184, the number of sub-image groups is the same as the designated number of spreads. In the present embodiment, the sub-image group having the oldest shooting date and time is assigned to the first spread, and the sub-image group is assigned to each spread page of the album in the order of shooting date and time. This makes it possible to create an album in which sub-image groups are arranged in order of shooting date and time. However, as will be described later, the images may or may not be arranged in the order of shooting date and time in one spread page.

In S419, the image selection unit 216 selects a predetermined number of images from each sub-image group. Hereinafter, a case where four images are selected from a sub-image group assigned to a certain spread will be described with reference to FIG. 9 as an example.

In FIG. 9, (A) shows the time difference between the shooting date and time from the first image to the last image included in the sub-image group assigned to the spread, in other words, the shooting period of the sub-image group. A method of selecting the first sheet when selecting four sheets will be described with reference to FIG. 9B. Of all the images taken during the shooting period of the sub-image group shown in FIG. 9B, the image with the highest score given in S416 is selected as the first image. Then, in the selection of the second and subsequent images, the shooting period of the sub image group is subdivided and the images are selected so that the selected images are not concentrated on a part of the shooting period of the sub image group. First, as shown in FIG. 9C, the shooting period of the sub-image group is divided into two equal parts (grouping) into two shooting sections (periods). Next, as shown in (D) of FIG. 9, the image having the highest score among the images belonging to the shooting section (solid line section) in which the first image is not selected is set as the second image. select. Next, as shown in (E) of FIG. 9, each shooting section of (D) is divided into two equal parts. The score of the image taken in the shooting section shown by the solid line in (F) of FIG. 9, that is, the two shooting sections in which the first and second shots are not selected (images corresponding to the two shooting sections) is The highest image is selected as the third image. Next, a case where an image does not exist in the shooting section for selecting an image and the image cannot be selected will be described by taking the fourth image selection as an example. In the present embodiment, the shooting period is divided according to the time regardless of the number of images. Therefore, the image may not exist in the shooting section obtained by dividing. For example, as shown in FIG. 9 (G), it is assumed that the fourth image is selected from the shooting section (diagonal line shooting section) in which the image has not been selected yet, but the image does not exist in this shooting section. .. In such a case, as shown in FIG. 9H, each image-selected shooting section is divided into two. Next, as shown in FIG. 9 (I), the image with the highest score is selected as the fourth image from the images taken in the solid line shooting section in which the first to third images are not selected. To do. In the case where the number of images to be selected is increased or decreased from 4, the images may be selected in the same manner as described above.

Returning to the description of FIG. 4 (a). In S420, the layout creation unit 217 determines which image is to be arranged in the spread of the processing target (determined by determining the image layout). In the present embodiment, after creating a layout having a margin at the end of the spread (as a basic layout), a layout for using all the nozzles of the recording head during recording (as a complementary layout) is created. FIG. 4D is a detailed flowchart of S420.

In S4201, the layout creation unit 217 creates a basic layout for each spread. Here, regarding a method of creating a basic layout based on the shooting date and time acquired in S403 and the score acquired in S416, FIG. 10 and FIG. 11 are used with reference to an example in which five images are selected in S419. explain. FIG. 10 shows the shooting date and time and the score for each of the five selected images. First, the five images selected as the images used for the spread are assigned to any of the two pages in the spread. In the present embodiment, the images are arranged in order of shooting time and divided at the center. When the number of selected images is an odd number as in this example, the image is divided at a location close to the center where the shooting time difference between the preceding and following images is large. Next, the page is divided according to the ratio of the highest score to the total of the other scores on each page. Here, the score ratio is divided into vertical or horizontal line segments so that the area ratio of the divided area on the divided page is the same. It is desirable that the direction of the long side of the larger rectangle after division and the image with the highest score are aligned. The area to be divided is an area excluding a certain margin at the edge of the page. This margin width information is incorporated in the album creation application in advance.

FIG. 11A shows the state after the first page division when the basic layout is created. After dividing each page that constitutes the spread, the image with the highest score is placed in the larger rectangle. Of the rectangle, the area excluding a certain margin at the end is used as the image area, and the target image is enlarged, reduced, and trimmed so that it fits in the area before being arranged (see FIG. 11B). .. After that, the same processing is performed using the unarranged image and the area in the page (see FIG. 11C). At this time, if there is only one image that is not arranged, it is arranged in the area in the remaining page. Here, as the basic layout, the arrangement of the images is determined based on the shooting date and time and the score of the image, but the method of creating the basic layout is not limited to this. The image may be arranged based on the color of the image or the result of object detection, or the image may be arranged based on information accompanying the image data other than the shooting date and time, such as position information and shooting conditions.

In S4202, the layout creation unit 217 determines whether the basic layout creation of S4201 is completed for all the spreads. If the determination result of this step is true, the process proceeds to S4203. On the other hand, if the determination result of this step is false, the process returns to S4201.

In S4203, the layout creation unit 217 creates a complementary layout for each spread. In the present embodiment, as a layout method for reducing unused nozzles of the recording head, a method of arranging an image over the entire spread is adopted. More specifically, any of the images assigned to the target spread in S419 is arranged as the background image in the entire spread, and the other images are arranged as the foreground image on the background image. At this time, the foreground image is arranged in the same manner as in creating the basic layout of S4201.

In the complementary layout, each image assigned to the spread is used as a background image, and the same number as the number of images is created. Here, as an image arrangement method, one background image is arranged over the entire spread, but the image arrangement method is not limited to this. There may be a plurality of background images, or only one page of the spread may be arranged on the entire page (FIG. 13 (a)) so that the margin area does not continue in the same direction as the paper transport direction. They may be arranged alternately (FIG. 13 (b)). Further, although the foreground images are arranged in the same manner as the basic layout, they may be collected on one page of the spread (FIG. 13 (c)) or arranged separately (FIG. 13 (d)).

In S4204, the unused nozzle determination unit 219 calculates the number of unused nozzles. Hereinafter, the method of calculating the unused nozzles in this step will be described with reference to FIG. 12A. FIG. 12A shows the recording result 1203 of each spread and the unused nozzle 1204 when recording on the recording medium 1201 using the recording head 1202. FIG. 12B is a graph showing the relationship between the nozzle position and the ink ejection amount of a certain color when recording on the recording medium 1201 while transporting the recording medium 1201 in the transport direction shown in FIG. 12A. Is. This corresponds to the sum of the ink ejection amounts in the direction perpendicular to the transport direction.

When actually recording with a recording device, the colors that make up the image data (for example, red (R), green (G), blue (B)) are the colors that match the recording head (for example, CMYK cyan (C), magenta). (M), yellow (Y), black (K)). Therefore, it is conceivable to calculate the number of unused nozzles using the pixel values of the four channels (CMYK). However, in the present embodiment, the number of non-ejection nozzles is calculated based on each pixel value of RGB generally used in image data instead of each pixel value of CMYK. In other words, instead of the ink ejection amount at each nozzle, the R value, G value, and B value are integrated along the column direction (conveyance direction of the recording medium) of the bitmap data of the layout result. Calculate the number of non-ejection nozzles. When any of the integrated values of RGB is 0, the nozzle at the corresponding position is regarded as an unused nozzle. Finally, the number of unused nozzles is calculated. Here, the number of non-ejection nozzles is calculated using each pixel value of RGB, but the calculation method is not limited to this. Other color spaces may be used, color separation and ink ejection amount calculation similar to those in the actual recording process may be performed, and arrangement on the paper surface during the recording process such as imposition may be considered. For example, when recording two spreads side by side in a direction perpendicular to the transport direction of the recording medium, the number of unused nozzles is set for each of the spreads recorded on the left side of the recording head and the spreads recorded on the right side. calculate. Further, it is not always necessary to determine whether or not all the nozzles of the recording head are non-ink ejection. The number of unused nozzles is the number of nozzles located outside the recording surface of the recording medium (nozzles outside the recording area) and nozzles that do not continue beyond a predetermined threshold value, that is, nozzles that cannot be visually recognized as defective nozzle non-discharge results after output. It is not necessary to count the number. For example, consider a case where nozzles are arranged at intervals of 2400 dpi (10.6 μm pitch) and recording is performed by a recording device having a dot diameter of 50 μm on a recording medium. At this time, since the non-ejection of the three consecutive nozzles can be filled by the ejection by the adjacent nozzles, it is not necessary to consider the three consecutive nozzles as unused nozzles. The threshold value may be changed according to the resolution of the recording device, recording conditions, characteristics of the recording medium, and the like.

In S4205, the layout creation unit 217 selects one of the complementary layouts created in S4203. In the present embodiment, as the complementary layout corresponding to the target spread, the complementary layout having the smallest value of the number of unused nozzles calculated in S4204 is selected.

In S4206, the layout creation unit 217 determines whether the processes of S4203 to S4205 have been completed for all the spreads. If the determination result of this step is true, the process proceeds to S4207, while if the determination result is false, the process returns to S4203.

In S4207, the layout creation unit 217 determines the spread to use the complementary layout. In the present embodiment, the spread with the smallest number of unused nozzles calculated in S4204 is a spread using the complementary layout, and the other spreads are spreads using the basic layout. Here, one spread with the smallest number of unused nozzles is selected, but the selection method is not limited to this. The number of unused nozzles may be reduced by selecting a plurality of spreads, or a spread may be selected so that the number of unused nozzles in a specific range is reduced.

In S4208, the unused nozzle determination unit 219 determines whether the number of unused nozzles in the entire album is equal to or less than a predetermined threshold value. If the determination result of this step is true, the process proceeds to S421. On the other hand, if the determination result of this step is false, the process returns to S4203 while maintaining the determination of the spread using the complementary layout determined in S4207. In the flowchart of FIG. 4D, the process returns to S4203, but this is not the case. For example, returning to S4207, the layout creation unit 217 may determine in S4207 that the spread determined to have the second smallest number of unused nozzles in the previous process of S4207 is a complementary spread. Here, when the number of unused nozzles in the entire album becomes equal to or less than the threshold value, the process shifts to S421, but the termination condition of S420 is not limited to this. The complementary layout may be increased even after the number of unused nozzles in the entire album is below the predetermined threshold, or the complementary layout may be increased until the number of unused nozzles when creating a part of the album is below the predetermined threshold. May be good. The predetermined threshold value used in this step may be incorporated in the album creation application in advance, may be derived from the value set by the album creation condition setting unit 201, or may be derived based on the image data. .. For example, if you select a paper type such as matte paper that has less noticeable defects, you can raise the threshold value, and if there are many low-frequency components in the image data and the defects are more noticeable, you can lower the threshold value. Good.

In S421, the layout information output unit 218 creates layout information. For example, the layout information output unit 218 creates bitmap format image data representing a spread image in which each image is laid out as determined in S420.

The layout information (bitmap format image data) created in S421 is transmitted to the printing apparatus to execute the printing process.

According to this embodiment, for example, when recording with a recording device such as a full multi-printer to create an album, the possibility that the nozzle is not used for a long time can be reduced. As a result, maintenance processing such as preliminary ejection can be reduced, so that the convenience of the user (or operator) can be improved. Further, the cost can be reduced by arranging the images so as to reduce the unused nozzles in the recording head based on the analysis result of the images.

[Second Embodiment]
In this embodiment, increasing the number of nozzles used is considered when selecting an image.

<About automatic layout>
Hereinafter, the automatic layout in this embodiment will be described with reference to FIG. 14A. FIG. 14A is a flowchart of a process for executing the automatic layout of the album creation application according to the present embodiment. Since the processing represented by the same reference numerals as those in the above-described embodiment (see FIG. 4) is the same as the processing in the above-described embodiment, the description thereof is omitted here.

In S1401, the image selection unit 216 selects a predetermined number of images from each sub-image group. Here, as an example of the selection method of this step, a method of selecting an image in which many colors are used in each spread will be described with reference to FIG. 14 (b).

In S14011, the image selection unit 216 performs color reduction processing on each image. In the color reduction treatment, for example, a known k-means method is used to reduce the colors in the image so that there are a predetermined number of types. In the present embodiment, the colors are reduced so that there are five types of colors in the image, but the number of types after the color reduction is not limited to five.

In S14012, the image selection unit 216 selects the first first image from the sub-image group. In the present embodiment, the image with the highest score given in S416 is selected as the first image.

In S14013, it is determined whether or not a predetermined number of images have been selected. If the determination result of this step is true, a series of processes is completed, while if the determination result is false, the process proceeds to S14014.

In S14014, the color distance between the already selected image and each of the images in the sub-image group excluding the already selected image is calculated. Here, as an example of color distance calculation in this step, color distance calculation using hue will be described. First, a color having the maximum area is derived from the previously selected image based on the color-reduced image obtained in S14011, and the derived color is used as a representative color. Next, the representative color is similarly determined for each image of the sub-image group excluding the already selected image based on the color-reduced image. Here, the representative color is represented by converting it into each value of HSV (hue, again, brightness). Finally, the difference between the representative color of the previously selected image and the representative color of each image in the sub-image group excluding the already selected image is calculated. Here, paying attention to the hue, the difference in hue is used. Although the difference in hue is used as the color distance, the values of other parameters may be used as the color distance. For example, it may be a distance in another color space.

In S14015, the image selection unit 216 selects an image having a color different from the selected image. In the present embodiment, among the color distances calculated in S14014, the image having the maximum average color distance from the selected image is selected. After selection, the process returns to S14013.

According to the present embodiment, by selecting an image to be used for an album based on the color of the image, the number of unused nozzles in the recording head can be reduced, and as a result, the cost can be reduced.

[Other Embodiments]
The idea of the present application is not limited to the above-described embodiment. For example, in the above-described embodiment, the album creation application has been described as an example, but an application or an image processing device having an image selection process for automatically selecting a recommended image may be used. That is, the above-described embodiment may be executed in a layout application that is not limited to album creation.

Further, as shown in FIG. 1, an image processing device assuming a local environment has been described as an example, but the image processing device is not limited to this. For example, it may be an image processing device on a server. In this case, the image stored in the server may be used or the image may be uploaded.

Further, in the above-described embodiment (particularly S4208), the complementary spread is determined so that the number of unused nozzles is equal to or less than a predetermined threshold value, but this is not the case. For example, the layout creation unit 217 identifies nozzles that are not used in the processes up to S4207. Then, the layout creation unit 217 determines the frequency of use of the specified unused nozzle, and may determine No in S4208 when the unused nozzle whose frequency of use is equal to or less than the threshold value is included. That is, even if the number of unused nozzles specified in S4204 is large, if the frequency of use of all the unused nozzles is high, the layout generation means may determine Yes in S4208. Further, for example, the layout creation unit 217 identifies nozzles that are not used in the processes up to S4207. Then, the layout creation unit 217 may determine the unused time of the specified unused nozzle, and may determine No in S4208 when the unused nozzle having the unused time equal to or longer than the threshold value is included.

In the above-described embodiment, it is assumed that the image is a photograph, and the grouping is performed based on the shooting date and time of the photograph. However, the grouping may be performed based on the date and time when the image was created.

In the above-described embodiment, the user specifies the mode, and the layout is performed based on the specified mode. However, image analysis is performed on the image group for creating the album, and the objects included in the image group are laid out. The mode may be set automatically based on the distribution of types.

The above-described embodiment is also realized by executing the following processing. That is, a program that realizes one or more functions of the above-described embodiment is supplied to the system or device via a network or a storage medium. Then, it can be realized by the process of reading and executing the program by one or more processors in the computer of the system or the device. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions. Regarding the above-mentioned first and second embodiments, the configurations of the respective embodiments may be appropriately combined and used.

100 Image processing device 212 Image analysis unit 215 Spread allocation unit 216 Image selection unit 217 Layout creation unit

Claims (12)

Layout means that determine how to lay out image data for a page,
Specific means for identifying information about nozzles of the printing apparatus that are not used in the printing process of the printing apparatus based on the determined layout method.
A determinant that determines whether or not to change the layout method based on the information about the identified nozzle.
A device characterized by comprising. The information about the identified nozzles is information about the number of nozzles of the printing apparatus that are not used in the printing process.
When the number of nozzles of the printing apparatus not used in the printing process is equal to or less than the threshold value, the determining means determines that the layout method is not changed.
The apparatus according to claim 1, wherein when the number of nozzles of the printing apparatus not used in the printing process is not equal to or less than the threshold value, the determining means determines to change the layout method. The identified information about the nozzle is information about the frequency of use of the nozzle of the printing apparatus that is not used in the printing process.
When the frequency of use of the nozzles of the printing apparatus not used in the printing process is equal to or less than the threshold value, the determining means determines that the layout method is not changed.
The apparatus according to claim 1, wherein when the frequency of use of the nozzles of the printing apparatus not used in the printing process is not equal to or less than a threshold value, the determining means determines to change the layout method. The information regarding the specified nozzle is information regarding the non-use time of the nozzle of the printing apparatus that is not used in the printing process.
When the non-use time of the printing apparatus not used in the printing process is equal to or less than the threshold value, the determining means determines that the layout method is not changed.
The apparatus according to claim 1, wherein when the non-use time of the printing apparatus not used in the printing process is not equal to or less than a threshold value, the determining means determines to change the layout method. The layout means is a first layout method in which an image is arranged so as to have a margin at an end as a method of laying out image data for a spread page of an album, and a nozzle corresponding to the width of the entire rectangle of the spread page. Determined with a second layout method in which the images were arranged so that all were used,
The apparatus according to any one of claims 1 to 4, wherein it is determined to use the second layout method for at least one spread page of the double-page spread of the album. The album is created by the recording device transporting the recording medium along the transport direction and recording on the recording medium.
The device according to claim 5, wherein the width is a width in a direction perpendicular to the transport direction. In the second layout method,
Is one image data arranged as a background image on the entire spread page?
Claim 5 or 6 characterized in that one image data is arranged on the entire surface of any one of the two pages constituting the spread page, or rectangular images and rectangular margins are arranged alternately. The device described in. The apparatus according to any one of claims 1 to 7, further comprising a selection means for selecting image data laid out from a plurality of image data based on information on colors used. The apparatus according to claim 8, wherein the selection means selects an image based on a color distance between the already selected image and each of the images of the image group excluding the already selected image. .. The apparatus according to claim 9, wherein the selection means selects an image having the maximum color distance from an image group excluding the already selected image. Steps to determine how to lay out image data for a page,
A step of identifying information about nozzles of the printing device that are not used in the printing process of the printing device based on the determined layout method.
A step of deciding whether or not to change the layout method based on the information about the identified nozzle.
A method characterized by providing. A program for causing a computer to perform the method according to claim 11.

Images