JP2012084184A - Album creation device, album creating method, program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To create an album with a fine appearance by compensating required number of images for creating the album using existing still images, and effectively using insufficient still images.SOLUTION: A value N, which is obtained by subtracting the total number of selected images from the total number of images that a template of a photo album requires (the total number of combination frames), is determined as the minimum number of shortfall images which are the minimum number of images required for compensating shortage of images. An image (an original image for creation) appropriate for a creation source for images, for compensating the shortage of images, is selected from among selected images. New images are created by applying predetermined image processing to the selected image of the creation source, until the minimum number N of the shortfall images is satisfied.

Description

  The present invention relates to creation of an album in which a predetermined number of images are arranged, and more particularly to adjustment of the number of images arranged in an album.

  Patent Documents 1 and 2 are conventional techniques for creating an album by electronically arranging images. In Patent Document 1, the number of images for each page is acquired based on the total number of pages and the number of images, and the images are distributed to each page. Here, the total number of images arranged in the album is within a predetermined range.

  In Patent Document 2, a user image is analyzed, a plurality of trimming area candidates are displayed, one of them is selected, and trimming processing is executed.

  On the other hand, Patent Document 3 is cited as a conventional technique for creating a still image from a moving image. In Patent Document 3, a still image is created from a moving image, and the still images are grouped according to the similarity of the still images.

JP 2006-74590 A JP 2006-279634 A Japanese Patent Laid-Open No. 2003-9069

  In order to create a good-looking album as a whole, it is necessary to have a certain number of images or more, but it is difficult if the user has few images. In this respect, Patent Document 1 only distributes existing images to each page, and Patent Document 2 can obtain only one trimmed image from one image, and there is no consideration on measures against a shortage of images. It has not been.

  Further, when the user does not have a spare still image but only a moving image, it is difficult to select how to obtain an image suitable for the album in a shortage.

  An object of the present invention is to supplement the number of images necessary for creating an album using existing still images, and to create a good-looking album by effectively using still images with little handheld.

  Another object of the present invention is to supplement the required number of images so that an album can be created by using an existing moving image, and to create a good-looking album by effectively using the moving image.

  The method for creating an album according to the present invention includes a step of acquiring an original image, a step of determining the number of insufficient images by comparing the number of images necessary for creating an arranged album and the number of acquired original images, Performing predetermined image processing on the acquired image to create at least one complementary image corresponding to the number of insufficient images; and creating an album based on at least one of the original image and the complementary image. Including.

  The predetermined image processing includes trimming of a subject area, synthesis of a background, synthesis of a graphic showing a facial expression, change of viewpoint or morphing by pseudo three-dimensionalization.

  A step of obtaining a selected image by selecting an image satisfying a predetermined condition from among the original images; a step of generating a complementary image at least for the number of deficient images by performing predetermined image processing on the selected image; and the original Creating the album based on at least one of an image, the complementary image and the selected image.

  The album creating method according to the present invention determines a minimum number of images required for creating a moving image, a representative image that is a still image captured from the moving image, and creating an album in which images are arranged in a predetermined template. Steps, by performing predetermined image processing on the acquired moving image, creating at least the number of still images of the same scene as the representative image necessary for creating an album, and the same as the created representative image Creating an album based on at least one of the still images of the scene.

  A program for causing a computer to execute the above album creating method is also included in the present invention.

  The album creating apparatus according to the present invention is a means for determining the number of deficient images by comparing the means for acquiring an original image with the number of images required for creating an album in which the images are arranged and the number of acquired original images. And means for creating at least one of complementary images by performing predetermined image processing on the acquired image, and means for creating an album based on at least one of the original image and the complementary image ,including.

  The album creating apparatus according to the present invention discriminates a minimum number of images required for creating a moving image, a means for obtaining a representative image that is a still image captured from the moving image, and an album in which images are arranged in a predetermined template. Means for performing predetermined image processing on the obtained moving image to create at least the number of still images of the same scene as the representative image necessary for creating the album, and the same as the representative image created Creating an album based on at least one of the still images of the scene.

  According to the present invention, an image deficiency can be compensated by performing predetermined image processing on an existing image. In particular, a trimmed image focusing on the subject in the image, an illustration corresponding to the subject of the image, an image viewed from different viewpoints by creating a pseudo 3D, an intermediate image showing the state of the object changing due to morphing, If an image obtained by capturing an image of the same scene as a certain representative image from a moving image is used to compensate for the lack of images, an album with a good appearance can be created.

Configuration diagram of image editing device Diagram showing an example of a template Photo album creation process flowchart The figure which shows an example of the selection image in which several subjects exist The figure which shows an example of the trimming area | region which includes several subjects separately The figure which shows an example of a trimming image The figure which illustrated a trimming image being arranged on a template The figure which shows an example of the selection image which does not contain a person subject The figure which shows an example of the selection picture where the illustration was synthesized The figure which shows an example of the selection picture where the illustration was synthesized The figure which shows another example of the selection image containing a person subject. The figure which shows an example of the selection image with which the illustration related to the expression of a person subject was combined The figure which shows an example of the selection image to which the image process which produces the change of a facial expression with which the illustration related to the facial expression of a person subject was combined The figure which illustrated the state that the selection picture where the illustration was synthesized was arranged on the template Schematic diagram of pseudo 3D processing A diagram exemplifying how pseudo 3D images viewed from different viewpoints are arranged on a template. Schematic diagram of morphing Diagram illustrating how the morphed image is placed on the template The figure which shows an example of a representative image A diagram exemplifying an image of the same scene captured from a video centering on a representative image A diagram illustrating how captured images of the same scene are placed on a template The figure which illustrated a situation where captured images of the same scene were continuously arranged on different pages

  FIG. 1 is a block diagram showing a schematic configuration of a photo album creating apparatus 10 to which the present invention is applied. The photo album creating apparatus 10 reads, for example, moving image data and still image data (including still images obtained by capturing moving images) stored in various recording media 40 such as a CD and a DVD, a memory card 42, and the like. A photo album (data for displaying an image on a display device with a predetermined layout and design or a print medium such as paper on which an image is printed with a predetermined layout and design) is created. The photo album creating apparatus 10 is operated by an operation unit 26 including a pointing device capable of instructing movement of a pointer (cursor) such as a mouse, a touch pad, and a drag ball, and various operation devices such as a keyboard and a touch panel.

  The photo album creating apparatus 10 includes a CPU 12 that controls the entire photo album creating apparatus 10, a drive 11 that reads and writes data from and to a recording medium 40, a card slot 16 that reads and writes data from and to a memory card 42, and a photo album creating apparatus 10. A hard disk drive (storage medium) 18 (hereinafter referred to as HDD 18) in which programs to be controlled, moving images and still images are stored, a RAM 20 that functions as a working memory when the CPU 12 performs various processes, a RAM 20 and the like A pointer that moves on the window in accordance with an input operation including a list of stored images, a window including a template image to be described later, and other various graphical user interfaces related to image editing, and a moving direction and moving speed instruction from the operation unit 26 Video signal A video encoder 22 to be output and displayed on the display 50, a communication I / F 28 for connecting to the Internet and other networks, and a data recording unit (operation history recording means) 30 each responsible for specialized processing. Yes. These units are connected to each other via a bus 36, and the CPU 12 controls the respective units of the apparatus according to an operation program stored in the HDD 18.

  The photo album creating apparatus 10 is connected to a printer 44 for printing the designated still image on plain paper or photographic paper via a printer driver 38.

  The card slot 16 detachably holds, for example, a memory card 42 such as a compact flash (registered trademark), an SD card, and smart media, and reads or writes still image data recorded on the memory card 42. Thereby, still image data captured by a digital still camera or the like can be read. In addition to the multi-drive 11 and the card slot 16, for example, a communication port such as USB or IEEE1394 may be provided so that direct communication with an electronic device such as a digital still camera or a PDA can be performed.

  The HDD 18 stores, in addition to one or a plurality of still image data serving as material for the photo album, decoration image data combined with these still image data when the photo album is generated. Examples of the decoration image include a mask image that covers an unnecessary part of the target image, a template image (see FIG. 2) that is provided with a decoration frame such as an illustration in which a fitting frame of the target image is defined. These decoration images add color to the photo album, such as decorating the background of still images as materials and adding one-point decorations.

  The template is provided with a predetermined number of composite frames X1, X2,... For fitting the selected still image data, and the selected images are sequentially arranged in the composite frames of the template according to an arbitrary arrangement order. It is assumed that the photo album is completed when the images are arranged in all the composite frames, and no blank is generated in the composite frames, and some image is arranged.

  Each still image data is stored in the HDD 18 in the form of an image file such as JPEG, TIFF, or EXIF standardized for a digital still camera. These image files are provided with a data area for storing still image data and a tag area for storing metadata indicating the contents of the data. Hereinafter, still image data is simply referred to as “image”.

  The moving image data corresponds to a compression recording standard such as motion JPEG or MPEG4.

  The communication I / F 28 connects the photo album creating apparatus 10 to the Internet, displays a home page on the Internet on the display 50, and transmits and receives image data via an electronic mail (hereinafter referred to as mail). The communication I / F 28 is a broadband compatible modem or the like. The communication I / F 28 may be connected to the Internet via an optical communication network or a cable network. The photo album generated by the photo album generation unit 34 can be distributed to a specific terminal via the Internet.

  The data recording unit 30 and the photo album generation unit 34 are so-called coprocessors, and each assists the processing operation of the CPU 12 by taking charge of specialized processing.

  When the photo album creation apparatus 10 is a storefront print terminal, the photo album created by the photo album generation unit 34 is output to the display 50 and displayed as a preview. If the content is confirmed, the printer 44 is set according to a user instruction. Can be output to a print medium, or the photo album such as a DVD can be transferred to a recording medium 40 that can be recorded as video data. Further, when the photo album creating apparatus 10 accepts a photo album order from a terminal connected from the outside via the Internet, the photo album preview is converted into a Web page that can be displayed on the terminal instead of the display 50 and the like. You may transmit to a terminal. Alternatively, the completed photo album may be displayed as one of the photo album display functions of various electronic devices such as a mobile phone and a digital camera without being premised on outputting to a recording medium or a print medium.

  The system configuration in FIG. 1 is an example, and the present invention can be implemented with other electronic devices (for example, mobile phones and PDAs) having the same configuration.

  The flow of photo album creation processing will be described below with reference to the flowchart of FIG. This process is executed by the photo album generation unit 34 (or CPU 12). A program for causing the photo album generation unit 34 to execute this processing is recorded on a computer-readable recording medium such as the HDD 18 or the RAM 20. Such a program may be imported from the outside via the Internet.

  In S <b> 1, images to be arranged in the photo album are read from the recording medium 40, the memory card 42, or a terminal connected to the communication I / F 28 from the outside via the Internet and stored in the HDD 18. The image includes a captured image of a moving image file in addition to a still image file such as JPEG.

  In S2, an image having conditions suitable for creating a photo album is selected from the original images stored in the HDD 18. For example, an image that satisfies a predetermined condition (number of pixels of 800,000 pixels or more, no image blur or blur, non-backlight, etc.) is determined from the original image, and an image that matches the condition is determined from the stored image. The image is sorted and separately stored in the HDD 18 as a sorted image distinguished from the original image. The predetermined condition may be arbitrarily designated by the user from the operation unit 26. In addition, the user may be able to arbitrarily specify whether or not to execute the processing of S2 itself from the operation unit 26. Note that the occurrence of out-of-focus or camera shake can be determined by a known method (for example, see JP-A-2005-332382).

  In S3, the minimum number of images (total number of composite frames) necessary for creating the photo album is compared with the total number of the selected images, and it is determined whether or not the minimum number of selected images necessary for creating the album is secured. . If secured, the process proceeds to S7, and if not secured, the process proceeds to S4.

  In S4, the value N obtained by subtracting the total number of the selected images from the total number of images (total number of composite frames) required by the photo album template is set to the minimum number of images necessary to compensate for the shortage of images. The minimum number of missing images, which is a number.

  In S5, an appropriate image (preparation source image) is selected from the selection images as a generation source of an image that supplements the deficient image. For example, an image having a plurality of subjects or an image with good image quality (for example, an image in which no red eyes are generated) is selected. This process will be described in detail later. In addition, when the image quality is set as the selection condition, it can be set as a milder condition than S2, a strict condition, or a condition unrelated to S2. This condition may be arbitrarily specified by the user.

  In S6, by performing predetermined image processing on the creation source image selected in S5, new images are generated until the minimum number N of insufficient images is reached. This process will be described in detail later.

  In S7, the image generated in S6 and the original sorted image are arranged in a desired order in the template combination frame, and both are combined. Note that if the minimum number N of insufficient images is sufficient with only the image generated in S6, the original selected image may not be arranged. The arrangement of the images may be automatically performed in order of shooting date and time, or may be in response to an instruction from the operation unit 26.

  Note that the number of images generated in S6 may not be exactly the same as the minimum number N of insufficient images, and may be larger than this. In this case, in S7, the original selected image (the image acquired in S1 when S2 is not implemented) and which image among the generated images to be arranged in the album are selected through an operation on the operation unit 26. It may be possible. If extra images are generated, it is up to the user to choose which one to use, and they can be used in whole or in part. Furthermore, it is also possible to create an album using only the generated image without using the original selected image (the image acquired in S1 when S2 is not performed). In short, it is not necessary to arrange all of the images generated in S6 in the album without omission, and it is preferable to give a room for selection that the user can select and place in the album.

  Hereinafter, details of the processes of S4 and S5 will be described. Various modes can be adopted for this processing.

  First, as a first mode, an image including a plurality of subjects is selected in S4, a region including each subject is set as a trimming region in S5, and an image is cut out according to the set trimming region. A plurality of trimmed images corresponding to the subject are obtained.

  For example, from the sorted image I1 as shown in FIG. 4, three subjects, a flower X1, a butterfly X2, and a person X3, are recognized. As a method for recognizing a subject, a well-known method can be used. For example, an image feature amount of a sample subject is stored in the HDD 18 and the presence of a specific subject is recognized by pattern matching between the feature amount of the sample subject and the feature amount. If there are a plurality of recognized subjects, the selected image is selected as a creation source image.

  Next, a plurality of trimming regions that individually include each of the subjects recognized from the creation source image are set in the creation source image. In FIG. 5, three trimming regions L1, L2, and L3 that individually include the flower X1, the butterfly X2, and the person X3 of the creation source image I1 are set. The shape of the trimming region is not particularly limited, such as a rectangle or a circle, but may be adapted to the shape of the composite frame to be fitted. The user may be able to freely specify / change the trimming area via the operation unit 26. Further, the number of created images is not limited to one, and the larger the number, the better.

  As shown in FIG. 6, when the creation source image I1 is trimmed according to the three trimming regions L1, L2, and L3, three trimmed images I1-1, I1-2, and I1-3 are obtained.

  After that, as shown in FIG. 7, the trimmed images I1-1, I1-2, and I1-3 are arranged in the corresponding synthesis frames A1, A2, and A3 and synthesized into the template. A new image can be created by pasting trimmed images together like a collage.

  In this way, by extracting the subject from the image and arranging it, it is possible to make up for the shortage of the image using an important part of the image.

  As a second aspect, a new image is created by synthesizing different background images and graphics with the creation source image.

  For example, an illustration in which a butterfly is dancing around is synthesized with an image I2 having a flower as a subject as shown in FIG. As a result, a new image I2-1 as shown in FIG. 9 is obtained. Alternatively, an illustration in which the peripheral portion of the image is colored with respect to the image I2 is synthesized. As a result, a new image I2-2 as shown in FIG. 10 is obtained.

  Alternatively, an illustration (tears) representing a person's emotion is synthesized with an image I3 having a person as a subject as shown in FIG. As a result, a new image I3-1 as shown in FIG. 12 is obtained. Also, in addition to composing star-shaped illustrations, the emotions are changed by processing some of the human face parts (eyes, mouth, nose). As a result, a new image I3-2 as shown in FIG. 13 is obtained.

  The type of illustration to be combined depends on the content of the subject. For example, if a facial part of a human subject is detected, the illustrations of FIGS. 12 and 13 are combined. If a facial part is not detected, the illustrations of FIGS. 9 and 10 are combined. The user may be able to specify an illustration to be synthesized via the operation unit 26.

  Thereafter, as shown in FIG. 14, the images I2-1, I2-2, I3-1, and I3-2 are arranged in the corresponding synthesis frames B1, B2, C1, and C2 and synthesized.

  In this way, it is possible to easily compensate for the lack of images by generating an image in which the original images are arranged.

  As a third aspect, the creation source image is pseudo-three-dimensionalized, and a planar image in which the pseudo three-dimensional image is viewed from a desired viewpoint is created. A well-known technique can be used to create a pseudo three-dimensional image from a planar image, for example, using “Make3D” of Stanford University.

  For example, a pseudo three-dimensional image is created from a planar creation source image I4 as shown in FIG. Then, an image obtained by viewing the pseudo three-dimensional image I4-1 as shown in FIG. 15B from the viewpoint on the right side from the front is set as a new image I4-1, and the pseudo 3D as shown in FIG. An image obtained by viewing the dimensional image I4-1 from the viewpoint on the left side from the front is defined as a new image I4-2.

  After that, as shown in FIG. 14, the images I4-1 and I4-2 are arranged in the corresponding synthesis frames D2 and D1 and synthesized into a template.

  In this way, the lack of images can be easily compensated using a plurality of images of the same person or landscape viewed from different viewpoints, and the interest of the album is enhanced.

  As a fourth aspect, a technique called morphing that shows a state of deformation from one object to another object is used to create an intermediate image showing a state during the deformation to a different object, and this is used as a new image.

  For example, by morphing from the subject of the pre-deformation image I5 with the pet as the subject as shown in FIG. 17A to the subject of the post-deformation image I6 as the subject with the pet owner as shown in FIG. 17C, FIG. An intermediate image I5-6 as shown in (b) is created. That is, it is necessary to secure two images for creating the intermediate image. Note that a plurality of intermediate images can be created instead of only one.

  Thereafter, as shown in FIG. 18, the pre-deformation image I5, the intermediate image I5-6, and the post-deformation image I6 are arranged in this order in the synthesis frames E1, E2, and E3 and synthesized into the template.

  In this way, an intermediate image is created from the two images using morphing to compensate for the lack of images, and the interest of the album is enhanced.

  As a fifth aspect, if the selected image is originally obtained by capturing a part of a moving image such as motion JPEG or MPEG4, another image in the same scene as the selected image is extracted from the same moving image. Capture it and make it a new image.

  For example, if the selected image (representative image) I7-0 as shown in FIG. 19 is a captured image of a part of a certain moving image, the captured image I7-1 of the same scene as the selected image as shown in FIG. I7-2 and I7-3 are generated from the moving image. A known method can be used for scene discrimination. For example, you may use a thing like patent document 3. FIG. Alternatively, a still image frame (for example, a frame image for 5 seconds before and after the representative image) that moves back and forth within a predetermined range with respect to the representative image as a center is captured, and as shown in Japanese Patent Laid-Open No. 2008-22216. The color temperatures of the representative image and each frame image are compared, and if they are the same, they are determined to be the same scene, and if they are different, they are determined to be different scenes, and the frame images determined to be the same scene are selected (see FIG. 20).

  After that, as shown in FIG. 21, the frame images I7-1, I7-2, and I7-3 in the same scene as the representative image I7-0 are combined in the order along the playback time axis of the moving images F1, F2, Place in F3 and F4 and synthesize. In FIG. 21, four frames of the same scene are arranged on the spread page, but frames of different scenes can be arranged on other spread pages.

  The layout method is not limited to that shown in FIG. For example, as shown in FIG. 21, one page may be allocated to each frame and arranged as a so-called flip book.

  In this way, images of the same scene having temporal continuity can be arranged, and the appearance can be improved.

  In addition, it is necessary to create at least the minimum number N of the number of missing images, but more images are created, and those that satisfy the predetermined conditions (image quality and size) are only the minimum number N of the missing images. You may select and arrange | position to a synthetic | combination frame.

  In the above description, it is assumed that an image is arranged in a template synthesis frame. However, a template synthesis frame is not essential for the implementation of the invention, and each image is arbitrarily assigned to each page. The present invention can also be applied to a photo album having a free layout that is arranged at the position. For example, when the number of original images is smaller than the number of images (typically, the number of album pages) N required for creating a photo album, images are generated up to at least N as in S6. Which original image / generated image is arranged on which page is left to the selection operation to the operation unit 26 by the user.

10: Photo album creation device, 18: HDD, 34: Photo album generation unit

Claims (5)

Means for obtaining a representative image that is a moving image and a still image captured from the moving image;
Means for determining the minimum number of images required to create an album with images,
Means for performing predetermined image processing on the acquired moving image to create at least the number of images necessary for creating the album, at least as many still images of the same scene as the representative image;
Means for creating an album based on at least one of still images of the same scene as the created representative image;
Album creation device including   2. The album creating apparatus according to claim 1, wherein the predetermined image processing includes trimming of a subject area, synthesis of a background, synthesis of a graphic indicating a facial expression, viewpoint change or morphing by pseudo three-dimensionalization. Photo album creation device
Obtaining a video and a representative image that is a still image captured from the video;
Determining the minimum number of images required to create an album with images,
By performing predetermined image processing on the acquired moving image, creating at least the number of images necessary for creating the album at least as many still images as the representative image,
Creating an album based on at least one of still images of the same scene as the created representative image;
To create an album.   The album creating method according to claim 3, wherein the predetermined image processing includes trimming of a subject area, synthesis of a background, synthesis of a graphic indicating a facial expression, change of a viewpoint by pseudo three-dimensionalization, or morphing.   A program for causing a computer to function as the album creating apparatus according to claim 1.

Images