JP5484038B2 - Image processing apparatus and control method thereof - Google Patents

Description

The present invention relates to an image processing apparatus that supports album creation and a control method thereof.

In recent years, with the widespread use of digital cameras, anyone can easily shoot digital images. One way to enjoy the captured images is to create an album.

Generally, when creating an album using a template including a plurality of image frames, images are registered one by one in all the image frames in the template. At this time, since the image is trimmed in the shape of the image frame, it is necessary to adjust the position and size of the image in the image frame so that a desired display is obtained. In addition, in order to finally make an album that looks good, it is also necessary to adjust the color tone and the like while considering the relationship with other images. This operation is basically performed manually by the user, but there are systems that are automatically performed.

However, when this operation is performed manually, the user must select one image to be put in each image frame from a large number of candidate images, and further create a layout while considering the arrangement in the image frame. Therefore, this work is very troublesome. In addition, when this operation is automatically performed, it is difficult to completely reflect the user's intention in the final layout.

There are several methods aimed at reducing the user's labor associated with registering an image in an image frame in a template.

For example, according to the method described in Patent Document 1, the image to be registered in the image frame is automatically selected based on the state designation information associated with the image frame. Can be reduced.

Further, according to the method described in Patent Document 2, a similar image related to an image to be placed in the image frame is displayed in the save area. When this method is used, the user can select an image having a preferred arrangement and color tone from similar images in the save area when the arrangement and color tone of the image registered in the image frame are not preferred. As a result, the user can reduce the labor involved in image adjustment.

Japanese Patent Laying-Open No. 2005-210689 JP 2004-257257 A

However, in the above-described method, when an image to be used by the user is determined with respect to the image frame, the user performs a layout at the time of creating an album so that the image is arranged in the user's favorite arrangement and size. This requires time and effort for image adjustment.

The present invention has been made to solve this problem, and it is an object of the present invention to reduce the user's trouble concerning the layout when creating an album.

An image processing apparatus according to the present invention that achieves the above object is as follows.
Holding means for holding a template having one or more image frames inside and one or more images;
Association means for selecting one of the one or more image frames as a selected image frame and associating the selected image selected from the one or more images with the selected image frame;
Setting means for setting a plurality of types of editing processing on the selected image so as to apply to the shape of the selected image frame based on the inclination and size of the circumscribed rectangle of the selected image frame;
Generating means for generating a plurality of candidate images by processing the selected image by a plurality of types of editing processing set by the setting means;
Presenting means for presenting each of the plurality of candidate images generated by the generating means in the shape of the selected image frame;
It is characterized by providing.

It is possible to reduce the user's trouble concerning the layout when creating the album.

The figure which shows the system configuration | structure of an image processing. The figure which shows the example of GUI of an application. The flowchart showing the flow of user operation at the time of album creation. The flowchart showing the flow of the process of an application when a user registers an image into an image frame. The figure which shows the example of an edit process variation list. The figure showing the example of the circumscribed rectangle and the inscribed rectangle of an image frame. The figure showing the default state with respect to the image frame of an image. The flowchart showing the flow of the process of an application at the time of determining the variation of the rotation angle of an image. The flowchart showing the flow of the process of an application at the time of determining the magnification of an image and the variation of movement amount. The figure which shows the example of the variation application result of the magnification of an image and a movement amount. The flowchart showing the flow of the process of an application when a user registers an image into an image frame. The flowchart showing the flow of the process of an application at the time of determining the variation of the toning parameter of an image.

First Embodiment FIG. 1 is a diagram illustrating an outline of a system that realizes a first embodiment. Here, the CPU 101 controls the operation of the entire computer system, and executes a program stored in the primary storage 102. The primary storage 102 is a main memory, and reads and stores programs stored in the secondary storage 103. Further, the secondary storage 103 corresponds to, for example, a hard disk. Generally, the capacity of the primary storage is smaller than the capacity of the secondary storage, and programs and data that cannot be stored in the primary storage are stored in the secondary storage. Generally, album templates, image frames, and the like are held in this secondary storage. Data that must be stored for a long time is also stored in the secondary storage. In this embodiment, a program that controls the processing procedure of the embodiment is stored in the secondary storage 103, and is read into the primary storage 102 when the program is executed, and the CPU 101 executes the execution processing.

The input device 104 corresponds to, for example, a mouse or a keyboard. The output device 105 corresponds to, for example, a monitor or a printer. The reading device 106 is an imaging device configured with a known CCD element or the like, and reads a captured image directly or indirectly into a primary storage device or a secondary storage device.

FIG. 2 is a diagram illustrating an outline of a GUI of the image processing apparatus (hereinafter also referred to as “application”) according to the first embodiment. On the preview screen 201, an arbitrary page among the pages constituting the album is preview-displayed. The page to be displayed for preview can be changed by a user operation via the input device 104.

The image list 202 displays a list of images for use in the album. The page list 203 displays a list of thumbnails of all pages constituting the album. When the print execution button 204 is pressed, album print processing is executed. The image frame 205 is included in the album and displayed on the preview screen 201 together with the album page. When an image is registered in the image frame, the position and size of the image can be adjusted by a user operation via the input device 104.

FIG. 3 is a flowchart showing the flow of operations performed by the user when creating an album using the application having the GUI shown in FIG. In step S301, the user selects a template from one or more templates of the album to be created. The template is specified by dragging and dropping a file from the file management system of the operating system onto the GUI of the application.

In step S302, the user selects an image to be used for creating an album. The user designates an image to be selected by dragging and dropping an image file on the GUI of the application from the file management system of the operating system. The selected image is added to the image list 202.

In step S303, the user determines an image to be preliminarily registered in the image frame inside the template. The user selects an arbitrary image on the image list 202 and drags and drops it on the image frame 205 in order to pre-register an image in the image frame. Hereinafter, an image frame in which the user intends to pre-register an image is referred to as a “selected image frame”, and an image to be pre-registered in the selected image frame is referred to as a “selected image”. In step S304, the user completes registration of images in the image frames in the album, and executes album printing. The user presses the print execution button 204 to execute album printing. Hereinafter, a rectangular area circumscribing the shape of the selected image frame is referred to as a “frame circumscribing rectangle”, and a rectangular area inscribed in the shape of the selected image frame is referred to as a “frame inscribed rectangle”.

FIG. 6 shows an example of a frame circumscribed rectangle 601 and a frame inscribed rectangle 603. When the frame circumscribed rectangle and the frame inscribed rectangle are inclined with respect to the page, it is assumed that the inclination is preset for each image frame 602. In the coordinate system described below, the upper left point of the rectangular area circumscribing the shape of the selected image frame is the origin, the right direction from the origin is the + x direction, and the lower direction is the + y direction.

Here, the state in which the long side of the selected image and the long side of the frame circumscribing rectangle are parallel, the selected image is circumscribed to the frame circumscribing rectangle, and the center of the selected image and the center of the frame circumscribing rectangle are matched. It is defined as the default state for the selection frame of the selected image.

FIG. 7 is a diagram showing how this default state is set. FIG. 7A shows a state in which the frame circumscribing rectangle 601 is inclined on the spread page 701 and the image frame 602 is within the frame circumscribing rectangle 601. FIG. 7B shows the state of the selected selected image. FIG. 7C shows a state where the selected image is arranged according to the inclination of the frame circumscribed rectangle 601 and the image frame 602, that is, the default state.

In addition, the parameter of the editing process for the selected image is represented by a difference from the default state. When the editing process is performed on the selected image, the CPU 101 executes the editing process after setting the selected image once to the default state. In the first embodiment, it is assumed that editing processing performed on an image is rotation, scaling, and movement of the image.

A rotation variation is generated for the rotation angle of the image as a parameter, an enlargement rate variation is generated for the enlargement ratio, and a movement amount variation is generated for the movement amount. An editing process variation is configured by a combination of the rotation angle, the movement amount, and the enlargement ratio. As for the contents of the editing process variation, as shown in the editing process variation list shown in FIG. 5A, a rotation angle, an enlargement ratio, an x-direction movement amount, and a y-direction movement amount corresponding to the variation number are created. Saved in a table.

FIG. 4 is a flowchart showing an operation flow of the application when the user registers an image in the image frame in step S303 in FIG. In step S <b> 401, the CPU 101 creates a rotation angle variation that is a parameter of rotation processing performed on the selected image. The created rotation angle variation is added to the edit processing variation list. A method for determining the variation of the rotation angle will be described later with reference to FIG. Here, the number of variations of the rotation angle created, and N _r. In addition, when R (x) is described, it represents the x-th value among the created variations of the rotation angle.

In step S402, the CPU 101 assigns 0 to the variable i. In step S403, the CPU 101 performs a rotation process on the selected image with the rotation amount of the rotation angle of R (i). In step S <b> 404, the CPU 101 creates variations for the enlargement ratio and movement amount, which are parameters for scaling and movement processing performed on the rotated image created in step S <b> 403. The created enlargement ratio and movement amount variation are added to the edit processing variation list. A method for determining the enlargement ratio and the movement amount variation will be described later with reference to FIG.

In step S405, the CPU 101 adds 1 to the variable i. In step S406, CPU 101 compares the variable i, the number _{N r} of rotation variations created in step S401. If i> N _r, the process proceeds to step S407. Otherwise, the process returns to step S403.

In step S407, the CPU 101 performs a plurality of types of editing processing on the selected image for all variations described in the editing processing variation list, and generates candidate images. The plurality of candidate images generated at this time are stored in the primary storage 102 or the secondary storage 103 in association with the variation information of the editing process variation list.

In step S408, the CPU 101 presents the plurality of candidate images created in step S407 to the user through the output device 105 such as a display. For example, the CPU 101 may open a new window near the selected image frame 205 on the GUI, for example, and display a list of candidate images on the window. Further, by providing an image save area on the GUI of the application, a list of candidate images may be displayed in the save area. In addition, the CPU 101 presents candidate images in a state of being cut out so as to fit the shape of the selected image frame. Further, the contents of the editing process variation corresponding to the candidate image may be displayed simultaneously with this window. In step S409, the CPU 101 registers an image selected by the user from the plurality of candidate images presented in step S408 in the selected image frame.

FIG. 8 is a flowchart showing the processing flow of the application when creating a variation in the rotation angle of the selected image in step S401 of FIG. In step S801, the CPU 101 adds 0 to the edit processing variation list as one of the rotation angle variations. In step S <b> 802, the CPU 101 obtains the inclination θ ₀ of the selected image with respect to the frame circumscribed rectangle and adds −θ ₀ as one of the rotation angle variations to the edit processing variation list.

In step S803, the CPU 101 detects the longest line segment in the selected image. The line segment is detected by a known method using Hough transform. In step S804, the CPU 101 checks whether a line segment is detected in step S803. If a line segment is detected, the process proceeds to step S805. Otherwise, the process ends.

In step S805, the CPU 101 obtains the inclination θ of the line segment detected in step S803 with respect to the long side of the frame circumscribed rectangle. Here, it is assumed that the unit of inclination is degrees. θ is determined in the range of −90 degrees to +90 degrees. In step S806, the CPU 101 checks how many values of θ are. If it is any of −10 degrees <θ <+10 degrees, θ> 80 degrees, and θ <−80 degrees, the process proceeds to step S807. Otherwise, the process ends. In step S807, the value of −θ is added to the editing process variation list as one of the rotation angle variations.

FIG. 9 shows a flow of processing when creating a variation of the enlargement ratio and movement amount of the selected image in step S404 of FIG. In step S <b> 901, the CPU 101 adds a value of 1.0 for the enlargement factor, 0 for the movement amount in the x direction, and 0 for the movement amount in the y direction, as one of the variations. In step S902, the CPU 101 detects a region of a specific subject included in the selected image. In this example, the specific subject is a human face area. Note that the detection of the face area in the image follows a known method.

For example, Japanese Patent Application Laid-Open No. 09-251534 discloses a facial area extracted using a standard face image (template) registered in advance, and features such as an eyeball (black eye) and a nostril from the extracted facial area. A technique for further extracting point candidates is shown. Also described is a technique for detecting facial parts such as eyes, nose and mouth from the arrangement of candidate feature points and similarities with templates such as eye regions, nose regions and mouth regions registered in advance. .

In step S903, the CPU 101 checks whether a face area is detected in step S902. If a face area is detected, the process proceeds to step S904. Otherwise, the process ends. In step S904, the CPU 101 obtains the center of the face area detected in step S902. When a plurality of faces are detected, the face having the largest area is used.

In step S905, the CPU 101 obtains the coordinates of the vertices of the frame inscribed rectangle. The upper left coordinate of the inscribed rectangle is P _i1 (x _i1 , y _i1 ), and the lower right coordinate is P _i2 (x _i2 , y _i2 ). In step S906, the values of the width W _i = x _i2 -x _i1 and the height H _i = y _i2 -y _i1 of the frame inscribed rectangle are compared. If W _i > H _i , the process proceeds to step S907. Otherwise, the process proceeds to step S911.

In step S907, the CPU 101 obtains center coordinates P _c1 , P _c2 , and P _c3 of the rectangular areas obtained by dividing the width of the frame inscribed rectangle into three equal parts. The positions of the center coordinates P _c1 , P _c2 , and P _c3 at this time are shown in FIG. As described above, the CPU 101 sets a plurality of relative positional relationships between the selected image and the selected image frame based on the three divided positions. In step S908, the CPU 101 generates a moving image by moving the selected image shown in FIG. 10B so that the coordinate P _c1 in the frame inscribed rectangle matches the center of the face area of the selected image. Do. Further, the size (magnification rate) of the moving image is determined so that the moving image circumscribes the frame circumscribing rectangle, the size of the moving image is changed based on the magnification rate, and the size-changed image is generated. The movement amount and the enlargement ratio are added to the editing process variation list as one of variations. An example of the editing result at this time is shown in FIG. The small figure on the left is an example in which the face of the selected image is matched with P _c1 .

In step S909, the CPU 101 sets the enlargement ratio and the moving amount of the selected image so that the coordinates P _c2 in the frame inscribed rectangle match the center of the face area of the selected image and circumscribes the frame circumscribed rectangle. Decide and add it to the edit process variation list as one of the variations. An example of the editing result at this time is shown in FIG. The small figure in the center is an example in which the face of the selected image is aligned with _Pc2 .

In step S910, the coordinate P _c3 in the frame inscribed rectangle, match the center of the face region of the selected image, and, as circumscribing the frame circumscribing rectangles, determines the magnification rate and the amount of movement of the selected image, Add to the edit process variation list as one of the variations. An example of the editing result at this time is shown in FIG. The small figure on the right is an example in which the face of the selected image is matched with _Pc3 .

In step S911, the enlargement ratio and the moving amount of the selected image are determined so that the center coordinate _Pc of the frame inscribed rectangle matches the center of the face area and circumscribes the frame circumscribed rectangle. Add to edit process variation list. The plurality of size-changed images that have been enlarged are presented as candidate images by the presenting means in step S408, and the candidate images selected by the user are registered in the selected image frame in step S409.

Through the above processing, in this embodiment, it is possible to create a variation of the editing processing in consideration of the inclination and shape of the selected image frame. Further, the number of edit processing variations that can be created is the number of (number of rotation variations) × (number of scaling and movement variations).

As described above, by using the processing described in the first embodiment, when an image is registered in the image frame of the album template, a variation of the editing process based on the characteristics of the image frame is created, and a preview of the image editing result is displayed by the user. Can be presented. Further, the user can register an image subjected to the editing process in the image frame by selecting a favorite one from the presented ones. As a result, the user's labor associated with the image adjustment processing for each image frame can be greatly reduced.

<Embodiment 2> In Embodiment 2, the editing process performed on an image considers a case where toning is added to rotation, scaling, and movement of the image. Toning is performed by adjusting the color of the selected image based on the difference between the average color information of all the images associated with the image frames in the same page of the template and the average color information of the selected image. Is called. As in the first embodiment, a plurality of types of editing processing variations are configured by combinations of parameters. The contents of the editing process variations are stored as an editing process variation list shown in FIG.

Here, the toning process is to add values to the R, G, and B values of all the pixels of the selected image. The toning process parameters are represented by ΔR, ΔG, and ΔB, which are addition values for the R, G, and B values, respectively. The R, G, and B addition values are determined by the difference between the R, G, and B averages of all the images laid out in the candidate image page and the individual R, G, and B average values. Is done.

FIG. 11 shows the flow of the operation of the second embodiment when the user registers an image in the image frame in step S303 of FIG. In step S1101, the CPU 101 creates a variation of the toning parameter to be applied to the selected image. The created variation of the toning parameter is added to the editing process variation list. A method for determining the variation of the toning parameter will be described later with reference to the flowchart of FIG.

Here, let _{Nc be} the number of variations of the created toning parameter. When C (x) is described, it represents the x-th value among the variations of the created toning parameter.

In step S1102, the CPU 101 substitutes 0 for a variable i. In step S1103, the CPU 101 performs a toning process on the selected image with the parameter C (i) to create a toned image. In step S1104, the CPU 101 creates variations of parameters for rotation, scaling, and movement processing performed on the image created in step S1103. The method for determining the variation of the rotation angle, the enlargement ratio, and the movement amount follows the method described in steps S401 to S406 in FIG.

In step S1105, the CPU 101 adds 1 to the variable i. In step S1106, the CPU 101 compares the variable i with _Nc of the toning parameter variation created in step S1101. If i> _Nc , the process proceeds to step S1107. Otherwise, the process returns to step S1103.

In step S1107, the CPU 101 executes the editing process on the selected image for all variations described in the editing process variation list, and creates candidate images. The candidate image created at this time is stored in association with each piece of variation information in the editing process variation list.

In step S1108, the CPU 101 presents the candidate image created in step S1107 to the user via the display device that is the output device 105 or the like. For example, the CPU 101 may open a window on the side of the selected image frame on the GUI and display a list of candidate images on the window. In step S1108, the CPU 101 executes an editing process corresponding to the image selected by the user among the candidate images presented in step S408, and registers it in the selected image frame.

FIG. 12 shows the flow of application processing when creating a variation of the toning parameter of the selected image in step S1101 of FIG. 11 in the second embodiment. In step S1201, the CPU 101 assigns 0 to each of the variables R _all , G _all , and B _all . In step S1202, the CPU 101 assigns 0 to the variable N _img . In step S1203, the CPU 101 assigns the number of image frames on the same spread page as the selected image frame to the variable N _frm .

In step S1204, the CPU 101 assigns 0 to the variable i. In step S1205, the CPU 101 confirms that the i-th image frame is not the selected image frame. If the i-th image frame is not the selected image frame, the process proceeds to step S1206. Otherwise, the process proceeds to step S1209.

In step S1206, the CPU 101 determines whether an image is registered in the i-th image frame. If an image is registered, the process proceeds to step S1207. Otherwise, the process proceeds to step S1209. In step S1207, the CPU 101 adds 1 to the variable N _img .

In step S1208, the CPU 101 obtains averages of R, G, and B values of all pixels of the image registered in the i-th image frame and adds them to R _all , G _all , and B _all , respectively. In step S1209, the CPU 101 adds 1 to the variable i.

In step S1210, CPU 101 compares the values of variable i and N _frm . If i> N _frm , the process proceeds to step _S1211 . Otherwise, the process returns to step S1205. In step S1211, the CPU 101 obtains the average R _ave , G _ave , and B _ave of the R, G, and B values of all the pixels of the selected image.

In step S <b> 1212, the CPU 101 calculates a value of ΔR = (R _all / N _img ) −R _ave and adds it to the editing process variation list as one of the variations of the toning parameter together with ΔG = 0 and ΔB = 0. . In step S1213, the CPU 101 calculates a value of ΔG = (G _all / N _img ) −G _ave and adds it to the editing process variation list as one of the variations of the toning parameter together with ΔR = 0 and ΔB = 0. .

In step S1214, the CPU 101 calculates a value of ΔB = (B _all / N _img ) −B _ave and adds it to the editing process variation list as one of the variations of the toning parameter together with ΔG = 0 and ΔR = 0. . In step S1215, the CPU 101 sets the values of ΔR = (R _all / N _img ) −R _ave , ΔG = (G _all / N _img ) −G _ave , ΔB = (B _all / N _img ) −B _ave. Is added to the editing process variation list as one of the variations of the toning parameter.

Through the above processing, in the present embodiment, it is possible to create a variation of the editing processing in consideration of the color tone of the image registered in the image frame other than the selected image frame. Further, the number of edit processing variations that can be created is the number of toning variations) × (number of rotation variations) × (number of scaling and movement variations). By using the image processing described in the second embodiment, when an image is registered in the image frame of the album template, it is possible to register an image that has been subjected to color tone processing in addition to the image processing in the first embodiment. As a result, it is possible to greatly reduce the user's labor associated with the image adjustment processing for improving the appearance of the entire album.

In the embodiment described this time, the editing process performed on the image is rotation, scaling, movement, and toning of the image. However, in addition to the above processing, arbitrary image processing such as other coordinate conversion processing or matrix calculation may be added to the variation of the editing processing.

For example, left / right reversal processing may be applied so that the face in the selected image faces the face in the image registered in the image frame on the same page. Further, for example, sharpness processing may be applied so that the sharpness is the same as that of an image registered in another image frame on the same page.

In the embodiment described this time, a variation of the editing process is created based on the characteristics of the image frame, and the candidate image to be created is changed. However, a candidate image may be created using a predetermined editing process variation, and the candidate image to be presented may be changed based on the characteristics of the image frame.

Other Embodiments The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

Holding means for holding a template having one or more image frames inside and one or more images;
Association means for selecting one of the one or more image frames as a selected image frame and associating the selected image selected from the one or more images with the selected image frame;
Setting means for setting a plurality of types of editing processing on the selected image so as to apply to the shape of the selected image frame based on the inclination and size of the circumscribed rectangle of the selected image frame;
Generating means for generating a plurality of candidate images by processing the selected image by a plurality of types of editing processing set by the setting means;
Presenting means for presenting each of the plurality of candidate images generated by the generating means in the shape of the selected image frame;
An image processing apparatus comprising:   The plurality of types of editing processing includes rotation processing by a plurality of rotation amounts with respect to the selected image, and the plurality of rotation amounts include 0 degrees and a rotation amount that matches the selected image with the inclination of the circumscribed rectangle. The image processing apparatus according to claim 1.   The plurality of rotation amounts further include a rotation amount for extracting the longest line segment from the selected image and making the direction of the long side of the circumscribed rectangle coincide with the direction of the line segment. Item 3. The image processing apparatus according to Item 2.   A specific subject is detected from the selected image, and the selected image and the selected image frame are arranged such that the center of the detected region of the specific subject matches each of the plurality of positions set in the selected image frame. The image processing apparatus according to claim 3, wherein a plurality of relative positional relationships are set.   5. The image processing apparatus according to claim 4, wherein, when arranged so as to have a relative positional relationship, the selected image is scaled so that the selected image circumscribes the circumscribed rectangle. 6.   The plurality of types of editing processing is based on a difference between an average of color information of all images associated with an image frame in the same page of the template and an average of color information of the selected image. The image processing apparatus according to claim 1, further comprising adjusting a color of the image. A method for controlling image processing in an image processing apparatus, comprising:
A holding step in which the holding means holds a template having one or more image frames inside and one or more images;
An association step of selecting one of the one or more image frames as a selected image frame, and associating the selected image selected from the one or more images with the selected image frame;
A setting step for setting a plurality of types of editing processing on the selected image so that the setting means applies to the shape of the selected image frame based on the inclination and size of the circumscribed rectangle of the selected image frame;
A generating step of generating a plurality of candidate images by processing the selected image by a plurality of types of editing processing set in the setting step;
A presenting means for presenting each of the plurality of candidate images generated in the generating step in the shape of the selected image frame; and
A method comprising the steps of:   The program for making a computer perform each process of the method of controlling the image processing of Claim 7.

Images