JP2015073325A - Image generation device, image generation method, image generation program, and image generation system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a still image from a moving image to instantly obtain an image capable of being appreciated while reproducing motion of a subject.SOLUTION: An image generation device comprises: a main subject determination unit for determining a main subject from among persons included in a moving image composed of frames at a predetermined cycle; a duplication unit which selects, as selection frames, a plurality of images including the main subject in the frames at the predetermined cycle and generates a duplication image using the main subject's image part from each of the plurality of selection frames; and a combining unit for generating a combined image obtained by combining a plurality of generated duplication images. The duplication unit selects the selection frames after photographing of the moving image, in each predetermined period different from the predetermined cycle.

Description

  The present invention relates to an image generation apparatus, an image generation method, an image generation program, and an image generation system that generate a composite image using a plurality of images.

  In recent years, many portable devices (photographing devices) with a photographing function such as a digital camera have not only a still image but also a moving image photographing function. For example, imaging devices that capture still images during moving image shooting are widely known.

  However, while still images can be viewed in an instant and are easy, moving images and continuous shooting (hereinafter referred to as moving images) require a relatively long time for both shooting and viewing, and are difficult to spread.

  The good point of the moving image is that it can reproduce the motion, and the good point of the still image is that the momentary image with a high degree of perfection that is cut off the moment can be viewed at a glance. Therefore, Patent Document 1 discloses a technique for creating a still image using a moving image. In the invention of Patent Document 1, the partial moving image cut out by the moving image cutting means is stored in the moving image storage means in association with moving image data as a thumbnail moving image.

Japanese Patent No. 4189575

  However, although the invention of Patent Document 1 is for moving picture search and is effective for displaying a scene change, the state of the main subject has not been understood.

  The present invention provides an image generation apparatus, an image generation method, and an image generation method that can easily create a fun image such as generating a still image from a moving image and obtaining an image that can be viewed instantly while reproducing the movement of the subject. It is an object to provide a program and an image generation system.

  An image generation apparatus according to the present invention includes a main subject determination unit that determines a main subject from persons included in a moving image including frames with a predetermined cycle, and the main subject among the frames with the predetermined cycle. A duplicating unit that selects a plurality of images as a selected frame, generates a duplicate image using the image portion of the main subject from each of the plurality of selected frames, and a synthesizing unit that generates a synthesized image by synthesizing the plurality of duplicate images And the duplicating unit selects the selected frame at a predetermined time different from the predetermined cycle after the moving image is captured.

  The image generation method according to the present invention includes a main subject determination step of determining a main subject from persons included in a moving image having frames with a predetermined cycle, and the main subject out of the frames with the predetermined cycle. A plurality of images included are selected as selected frames, a duplication step of generating a duplicate image using the image portion of the main subject from each of the plurality of selected frames, and a composite image obtained by synthesizing the plurality of duplicate images is generated. A composition step, and the duplication step performs selection of the selected frame at a predetermined time different from the predetermined period after the moving image is captured.

  In addition, an image generation program according to the present invention includes a main subject determination step of determining a main subject from persons included in a moving image having frames with a predetermined cycle, and the above-mentioned frames among the frames with the predetermined cycle. A duplication step of selecting a plurality of images including a main subject as a selected frame, and generating a duplicate image using an image portion of the main subject from each of the plurality of selected frames, and a composite image obtained by synthesizing the plurality of the duplicate images And a synthesizing step for generating the selected image. In the duplicating step, the selected frame is selected at a predetermined time different from the predetermined period after the moving image is captured.

  In addition, an image generation system according to the present invention includes an imaging unit that captures a moving image including frames having a predetermined cycle, and the moving image that is given from the imaging unit, and the person included in the moving image A main subject determination unit for determining a subject, and a plurality of images including the main subject among the frames of the predetermined cycle are selected as selected frames, and an image portion of the main subject is used from each of the plurality of selected frames. A duplicating unit that creates a duplicated image; and a synthesizing unit that produces a synthesized image obtained by synthesizing a plurality of the duplicated images, and the duplicating unit selects the selected frame after the moving image is captured. It is performed every predetermined time different from the predetermined cycle.

  According to the present invention, it is possible to extract an appropriate part from the results of continuous shooting, generate a new image, and summarize the result while reproducing the movement of the subject to obtain an easily viewable image. It has the effect of being able to.

1 is a block diagram showing a circuit configuration of an image generation apparatus according to a first embodiment of the present invention. Explanatory drawing for demonstrating the information recorded on the temporary recording part 4b. Explanatory drawing for demonstrating the process of the mount determination part 1c and the duplication determination part 1d. FIG. 4 is a flowchart showing camera control. The flowchart which shows the determination flow of the main subject. The flowchart which shows the decision flow of a selection top. Explanatory drawing which shows an example of the synthesized image synthesized from a moving image. The flowchart which shows the 2nd Embodiment of this invention. The flowchart which shows a composite image preparation process. 6 is a flowchart showing subject composition processing. Explanatory drawing for demonstrating the image composition by the 3rd Embodiment of this invention. 5 is a flowchart showing subject duplication / combination processing. Explanatory drawing for demonstrating a duplicate image. The flowchart which shows the determination flow of an empty space. The flowchart which shows a frame selection process and a synthetic | combination process. The flowchart which shows a frame selection process and a synthetic | combination process. Explanatory drawing for demonstrating a synthesized image.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a circuit configuration of an image generation apparatus according to the first embodiment of the present invention. The present embodiment shows an example in which the present invention is applied to a photographing apparatus.

  In this embodiment, a person existing in a moving image is detected, a person as a main subject is specified from one or more detected persons, and a plurality of image portions of the specified main subject are extracted from a moving image. Are synthesized on a still image to generate a composite image. Thus, a moving subject can be viewed as a still image that can be viewed instantly. In the following description, it is assumed that a composite image is created from a moving image. However, the image may be a continuously captured image, and a composite image may be generated from a series of still images taken continuously.

  In FIG. 1, the image generation apparatus 10 includes an imaging unit 2 configured by an imaging element such as a CCD or a CMOS sensor. The imaging unit 2 is configured such that the aperture, focus, zoom, and the like can be controlled by the signal processing and control unit 1 described later, and can capture images corresponding to various compositions and subjects.

  The imaging unit 2 is driven and controlled by the signal processing and control unit 1 to capture a moving image of a subject and output an image signal. The signal processing and control unit 1 outputs a driving signal for the imaging element to the imaging unit 2 and captures an image signal from the imaging unit 2.

  The signal processing and control unit 1 performs predetermined signal processing such as color signal generation processing, matrix conversion processing, and other various signal processing on the input image signal. The signal processing and control unit 1 can output image information that has been encoded and compressed when recording an image signal. The signal processing and control unit 1 can supply the image signal subjected to the signal processing to the recording unit 4 for recording, and can also provide the display unit 7 for display.

  The image generation apparatus 10 is also provided with a clock unit 5 and an operation determination unit 6. The clock unit 5 generates time information used by the signal processing and control unit 1. The operation determination unit 6 generates an operation signal based on a user operation for various switches such as a release switch (not shown) provided in the image generation apparatus 10 and a shooting mode setting, and outputs the operation signal to the signal processing and control unit 1. ing. The signal processing and control unit 1 controls each unit based on the operation signal.

  In the present embodiment, the image signal of the moving image captured by the imaging unit 2 is supplied not only to the signal processing and control unit 1 but also to the face detection unit 3 and the recording unit 4. The recording unit 4 can record an image signal of the input moving image.

  The face detection unit 3 receives the image signal of the moving image from the imaging unit 2, and detects whether or not there is a shadow pattern having human face characteristics in the image for each frame (frame). The face detection unit 3 gives the detection result of the detected face area to the signal processing and control unit 1. Note that the face detection unit 3 may obtain a feature amount indicating a facial expression for the detected face region, and give it to the signal processing and control unit 1 as a detection result.

  The signal processing and control unit 1 generates information such as the image position and size of a face existing in the image based on the detection result of the face detection unit 3, and the generated image is recorded in the recording unit 4. The temporary recording unit 4b in the recording unit 4 records the corresponding frame.

  FIG. 2 is an explanatory diagram for explaining information recorded in the temporary recording unit 4b. FIG. 2 shows the person information generated by the signal processing and control unit 1.

  The numbers in () in FIG. 2 indicate X (horizontal) and Y (vertical) coordinate values when the image is divided into 10 in the horizontal and vertical directions and the lower left corner of the image is the origin. The following numerical value indicates the size of the face area when the horizontal length of the image is 10. The number of divisions used to indicate the coordinates and size is an example.

  The person information shown in FIG. 2 shows the face area detected at each timing T1, T2,. Timing T1, T2,... Corresponds to a frame of an image from the imaging unit 2, and the signal processing and control unit 1 identifies a person from the detection result of the face area for each frame or a frame of a predetermined period. .

  That is, when the face detection result is given from the face detection unit 3, the signal processing and control unit 1 assigns a person to the face area detected in the image. In the example of FIG. 2, three face regions are detected at timing T1, and the center coordinates of each face region are (1, 5), (5, 5), and (8, 2), respectively. It shows that the persons A to C are assigned respectively. The horizontal sizes of the face areas of the persons A to C are 1, 1, and 2, respectively.

  The signal processing and control unit 1 recognizes the motion of the face area based on the detection result of the face detection unit 3 to recognize which person the assigned face area corresponds to. be able to. For example, the signal processing and control unit 1 can detect the movement of the face area based on the same person by using a known tracking autofocus technique.

  Thus, at the next timing T2, the signal processing and control unit 1 does not detect the face area assigned to the person A, the face area assigned to the person B moves to the coordinates (4, 6), and is assigned to the person C. 2 is detected at a position of coordinates (8, 2), and the person information shown in FIG. 2 is generated.

  Thereafter, in the same manner, the signal processing and control unit 1 generates the person information by obtaining the center coordinates of the face area and the size of the face area at each timing for each person.

  In FIG. 2, the size is determined according to the size of the face region in the horizontal direction. However, it is obvious that the size in the vertical direction may be used or the area may be used.

  In addition, by detecting the movement of the face area, it was detected to which person the face area in the image corresponds. However, the feature amount of the face area is obtained, and each face region is obtained by comparing the feature amount. It may be detected which person corresponds to.

  In the present embodiment, the signal processing and control unit 1 includes a main subject determination unit 1b, a mount determination unit 1d, a duplication determination unit 1d, and a synthesis unit 1e. The main subject determining unit 1b uses the person information to determine a person who is a main subject among a plurality of persons having face areas that appear in the moving image.

  The main subject determination unit 1b determines the main subject based on the frequency of detection of the face area in the moving image, the position in the image and its change, the size of the face area and its change, and the like. For example, in the example of FIG. 2, the face area of the person A disappears from the image at the timing T2. Further, the face area of the person C is located in a relatively peripheral portion of the image, the position does not change, and the size is relatively small and does not change. On the other hand, the face area of the person B has a relatively large size at the timing T5 due to movement in the approximate center of the screen and a change in size. Therefore, for example, in the example of FIG. 2, the main subject determining unit 1b selects the person B as the main subject.

  The mount determination unit 1d uses the person information to determine an image (hereinafter referred to as a mount) that is the background of the composite image of the main subject. The duplication determination unit 1d determines, as a selected frame, a frame including the main subject image used for the composite image among a plurality of images including the main subject detected from the moving image.

  FIG. 3 is an explanatory diagram for explaining the processing of the mount determining unit 1c and the copy determining unit 1d, and for selection generated by the signal processing and control unit 1 based on the processing of the mount determining unit 1c and the copy determining unit 1d. Frame information is shown.

  FIG. 3 is a diagram for explaining selection of a frame to be used for a composite image when the person B in FIG. 2 is selected as the main subject. In the present embodiment, the main subject is extracted from the moving image and displayed superimposed on the background. Therefore, in order to obtain a mount as a background, it is desirable that the motion of the pattern on the mount is small and that the image be a still image as much as possible.

  Also, when considering combining many main subjects, the mount is a frame with a large free space so that many main subjects extracted from other frames or main subjects with large sizes can be combined. In other words, I will explain the idea that it is better to select a frame with the smallest size of the main subject as an example, but it may be a frame where the main subject is not shown, and if there is enough space, even a frame with a clear face Good. Of course, the first frame that is the starting point of the movement of the main subject may be determined as the mount. Further, the last frame that is the end point of the movement of the main subject may be determined as the mount. The mount determination method can be variously selected. For example, the mount determination unit 1c may determine an image to be the mount based on an operation signal from the operation determination unit 6 based on a user's mount specifying operation.

  The example of FIG. 3 shows that the mount determining unit 1c selects the frame having the smallest main subject size among the frames as the mount. In the selected mount, the horizontal size of the face area of the main subject is 1. As will be described later, on the mount, in order to arrange each subject so as not to overlap in the horizontal direction, the horizontal size of the shoulder width is set to one on each side of the face region with respect to the size of the face region of the main subject. Determine free space. In the example of FIG. 3, the empty space is 4 spaces on the left side of the main subject on the mount and 3 spaces on the right side.

  The duplication determination unit 1d determines which frame of the main subject is used for synthesis among the frames in which the face area of the main subject exists. The duplication determination unit 1d selects a frame so as to synthesize a main subject rich in change and a main subject with various patterns. The duplication determination unit 1d gives the information of the selected frame (selected frame) to the temporary recording unit 4b for recording.

  In the example of FIG. 3, when the size of the face area changes greatly between the preceding and succeeding frames, the copy determining unit 1d sets the changed frame as the selected frame. In addition, when the position of the face area on the image greatly changes between the previous and next frames, the duplication determination unit 1d sets the selected frame as the selected frame. Further, the duplication determination unit 1d may determine a facial expression from the feature amount of the face area and select a frame that has been determined to be a smile. For example, the degree of bending of a heel or mouth is quantified as a feature amount, and the higher the value, the more the smile is determined.

  The synthesizing unit 1e extracts a main subject image portion from the frame determined by the duplication determination unit 1d, creates a duplicate image, and synthesizes it on the mount determined by the mount determination unit 1c. The signal processing and control unit 1 can give the composite image generated by the synthesizing unit 1e to the display unit 7 for display and can also give the recording unit 4 for recording.

  Next, the operation of the embodiment thus configured will be described with reference to the flowcharts of FIGS. 4 to 6 and the explanatory diagram of FIG. 4 shows camera control, FIG. 5 shows a main subject determination flow, and FIG. 6 shows a selected frame determination flow. FIG. 7 shows an example of a synthesized image synthesized from a moving image.

  When the power of the image generation apparatus 10 is on, the process proceeds from step S1 in FIG. 4 to step S2, and the signal processing and control unit 1 determines whether or not the shooting mode is instructed. If the shooting mode is not instructed, the signal processing and control unit 1 determines whether or not the reproduction mode is instructed in step S3. When the reproduction mode is instructed, the signal processing and control unit 1 displays a list of thumbnails in step S4. When the user selects an image with reference to the thumbnail list, the process proceeds from step S5 to step S6, and the signal processing and control unit 1 reproduces the selected image.

  On the other hand, when the shooting mode is instructed, the signal processing and control unit 1 causes the display unit 7 to display a captured image (through image) based on the image signal from the imaging unit 2 in step S7.

  Steps S8 to S18 are a flow for creating a composite image. When a recording start operation is performed by the user, the signal processing and control unit 1 drives the imaging unit 2 based on an operation signal from the operation determination unit 6 to start capturing a moving image.

  FIG. 7 shows an example of three frames F1 to F3 of a moving image taken by the imaging unit 2. Frames F1 to F3 are taken at substantially the same angle of view. A building is shown on the upper right side of the imaging range, a tree is shown on the left side, and a road on a curve is shown in the approximate center of the imaging range. Note that the characters “background part” in the frame F1 are entered for the purpose of explaining the composite image.

  In step S10, face determination is performed. The face detection unit 3 detects a face area from each frame of the moving image and outputs the detection result to the signal processing and control unit 1. In step S11, the signal processing and control unit 1 causes the temporary recording unit 4b to record the person information generated based on the face detection result. The moving image from the imaging unit 2 is recorded in the recording unit 4, and the signal processing and control unit 1 records person information corresponding to each frame of the moving image recorded in the recording unit 4.

  In the next step S12, it is determined whether the background change is large. Since the signal processing and control unit 1 is not suitable for the synthesis of the composite image when the background changes significantly, such as when the shooting range of the moving image to be shot changes, the background change in step S13. Warning. For example, the signal processing and control unit 1 causes the display unit 7 to display a warning display indicating that the background change is large. Assuming such a composite image, a user who shoots a moving subject may shoot while confirming this.

  The signal processing and control unit 1 determines whether or not the moving image recording has ended in step S14, and if not, repeats the processing of steps S9 to S13.

  When the recording of the moving image is completed, in step S15, the signal processing and control unit 1 determines whether or not there is a subject candidate. The signal processing and control unit 1 issues a warning in step S19 when a frame whose face is detected is not included in the moving image. For example, the signal processing and control unit 1 causes the display unit 7 to display a warning display indicating that the person to be synthesized in the moving image is not photographed.

  In step S16, the main subject is determined. As shown in FIG. 5, the main subject determination unit 1b determines the main subject by the processes in steps S21 to S23. That is, the main subject determination unit 1b determines the main subject by giving priority to the person recorded from the beginning to the end in the moving image, giving priority to the person with movement, and giving priority to the person near the center of the screen. Note that the processing order of steps S21 to S23 can be changed as appropriate. Furthermore, the main subject determination unit 1b may determine the main subject in consideration of the size of the face area.

  In the example of FIG. 7, persons W1 and W2 are recorded in any of the frames F1 to F3. However, the images W2 (F1) to W2 (F3) of the person W2 are all photographed at the same position and the same size on the right side of the shooting range. On the other hand, the images W1 (F1) to W1 (F3) of the person W1 are photographed with the position and size of the face area being changed substantially at the center of the imaging range. Accordingly, in the example of FIG. 7, the person W1 is determined as the main subject.

  In the next step S17, the selected frame is determined. That is, in step S17, the mount determination unit 1c determines the mount and the duplication determination unit 1d determines the selected frame to be used for the composite image.

  FIG. 6 shows this flow. First, in step S31, the mount determining unit 1c determines a frame to be mounted. For example, the mount determining unit 1c selects a frame having a small main subject size and a large space (empty space) excluding the main subject as the mount. In the example of FIG. 7, among the backgrounds B1 to B3 of the frames F1 to F3, the background B1 that is the background of the image W1 (F1) of the person with the smallest face size of the person W1 (F1) selected as the main subject Space is the largest. The mount determining unit 1c determines, for example, the frame F1 having the background B1 as the mount.

  The mount determining unit 1c selects the mount according to the size of the face area, but may select the mount according to the size of the person including the face area.

  The next steps S32 to S39 are frame selection processing for determining from which frame an image of a person to be displayed superimposed on the mount is selected. First, in step S32, the copy determination unit 1d sets the first frame image information to Im0. Next, in step S33, the copy determining unit 1d sets Im1 as the image information of the next frame, and compares Im0 and Im1 (step S34).

  The image signal is recorded in the recording unit 4, and the duplication determination unit 1d reads the image signal from the recording unit 4 and compares Im0 and Im1 with each other. Thereby, the duplication determination unit 1d determines whether or not there has been a significant change in the position and size of the face area of the main subject or the facial expression (step S35).

  For example, the duplication determination unit 1d determines whether or not the center position of the face area has moved larger than a predetermined threshold, whether the size of the face area has changed more than a predetermined threshold, It is determined whether or not the main subject has largely changed based on whether or not the threshold has changed. Note that these determinations may be made not only from the face area but also from the image of the entire person.

  In the next step S36, the duplication determination unit 1d records, in the temporary recording unit 4, information for identifying the frame and the amount of change for the frame for which it has been determined that the main subject has changed significantly. The duplication determination unit 1d determines whether or not the determination has been made up to the last frame in step S37. If it is not determined to the last frame, in step S38, the information of Im1 is set to Im0, and steps S33 to S37 are repeated.

  When the determination up to the last frame is completed, in step S39, the copy determining unit 1d selects a predetermined number of frames in descending order of the amount of change among the frames recorded in the temporary recording unit 4b with the amount of change larger than the threshold. Select as a selected frame to be used for composition.

  When the selected frame is determined, the composition unit 1e creates a composite image in step S18 of FIG. That is, the synthesizing unit 1e generates a duplicate image by extracting the image portion of the main subject on the selected frame selected in step S39, and generates a synthesized image by superimposing the generated duplicate image on the mount image.

  FIG. 7 shows an example in which the frame F1 is selected as the mount and the frames F2 and F3 are selected as the selected frames used for duplication. That is, the synthesizing unit 1e extracts a person image W1 (F2) from the frame F2, and extracts a person image W1 (F3) from the frame F3. Then, a composite image (still image) P1 is obtained in which duplicate images based on the extracted images W1 (F2) and W1 (F3) are arranged on the mount image. On the composite image P1, main subjects extracted from each frame are arranged side by side. This composite image P1 is a still image in which various states of the main subject are converted into a single image, and the state of movement of the main subject can be instantly imagined.

  The synthesizing unit 1e gives the generated synthesized image to the recording unit 4 for recording. The synthesizing unit 1e gives the generated synthesized image to the display unit 7 for display.

  As described above, in this embodiment, a main subject is detected from each frame in a moving image, a selected frame used for composition of the main subject and a frame used as a mount are selected, and a plurality of main subjects are duplicated on the mount. Composite the images. When selecting a selected frame used for subject composition, a frame with a large change in the subject is selected. As a result, the synthesized image reproduces the movement of the subject, and it is possible to grasp the state of the subject that is alive instantly from one still synthesized image.

(Second Embodiment)
FIG. 8 is a flowchart showing the second embodiment of the present invention. In FIG. 8, the same steps as those in FIG.

  This embodiment is different from the first embodiment only in the selected frame determining method, and the hardware configuration is the same as that in FIG. In the first embodiment, as shown in steps S33 and S34 in FIG.

  On the other hand, in the present embodiment, frames at predetermined time intervals are sequentially compared. FIG. 8 shows an example in which the predetermined time is 1 second. Further, in the present embodiment, after selecting a frame for duplicating the main subject, a frame to be a mount is selected.

  In step S41, the duplication determination unit 1d sets Im1 as the image information of the frame one second after the first frame, and compares Im0 and Im1 (step S34). Thereby, the duplication determination unit 1d determines whether or not there has been a significant change in the position, size, and expression of the face area of the main subject (step S35).

  In the next step S42, the duplication determination unit 1d records, in the temporary recording unit 4, information for specifying the frame and the amount of change for the frame for which it has been determined that the main subject has changed significantly. In step S43, the copy determination unit 1d determines whether there is a frame after 1 second. If there is a frame after 1 second, in step S38, the information of Im1 is set to Im0, and steps S33 to S37 are repeated.

  If there is no frame after one second, the duplication determination unit 1d determines in step S44 that the frames recorded in the temporary recording unit 4b as having a change amount larger than the threshold value are the two in descending order of change amount. Is selected as a frame to be used for composition. Note that three or more frames may be selected as in step S39 of FIG.

  In the present embodiment, in the next step S45, a frame to be mounted is selected. That is, the copy determining unit 1d is a frame whose main subject is small in size, and changes between the facial expression of the main subject in the frame and the facial expression of the main subject in the frame selected in step S44. Select the large frame as the mount frame.

  According to the flow in FIG. 8, three images of the main subject are combined and displayed on the mount. The images of the three main subjects are relatively different in facial expression from each other, and it is possible to appreciate facial expressions rich in changes in the main subject by the composite image.

  Other operations in the present embodiment are the same as those in the first embodiment.

  As described above, in this embodiment, when selecting a frame to be used for a composite image, a frame at a predetermined time can be used. Thereby, it is possible to remarkably reduce the amount of calculation required to detect a frame in which the main subject changes greatly. In this embodiment, when selecting a frame to be the mount, not only the size of the main subject in the frame to be the mount but also the image of the other main subject to be combined with the image of the main subject of the frame to be the mount The size of the change is also taken into consideration, and a composite image richer in change than in the first embodiment can be obtained.

(Third embodiment)
FIGS. 9 to 11 relate to a third embodiment of the present invention, FIG. 9 is a flowchart showing a composite image creation process, FIG. 10 is a flowchart showing a subject composition process, and FIG. 11 is an image according to the third embodiment. It is explanatory drawing for demonstrating a synthesis | combination.

  This embodiment is different from the first embodiment only in the method of creating a composite image, and the hardware configuration is the same as that in FIG. In the first embodiment, as shown in FIG. 7, an image of a person is cut out and duplicated, and placed in an empty space on the mount. However, it is conceivable that a relatively large amount of calculation is required for the process of extracting a person's image and the process of arranging it in an empty space. Therefore, in the present embodiment, it is possible to easily perform extraction and arrangement processing of the image of the main subject used for the composite image.

  In step S51 of FIG. 9, the composition unit 1e obtains the left and right empty space sizes L and R of the main subject on the mount. FIG. 11A shows the mount 21 selected by the mount determining unit 1e. The main subject 22 is imaged at the center of the mount 21. The area excluding the image area of the main subject 22 is an empty space, and the combining unit 1e obtains the sizes L and R of the left and right empty spaces 23 and 24 of the main subject 22 among the empty spaces.

  In the next step S52, the composition unit 1e calculates the size of the left and right empty spaces. In the example of FIG. 11A, R> L.

  Next, in step S53, the composition unit 1e calculates the size of the main subject of the selected frame. FIGS. 11B and 11C show the frames 25 and 28 including the main subject to be duplicated. On the frame 25, the main subject 26 to be copied is shown in the center. Only the upper body of the main subject 26 is imaged. On the frame 28, a main subject 29 to be copied is shown in the center. In the main subjects 26 and 29, the main subject 26 is larger in size.

  In the next step S54, the combining unit 1e duplicates and combines the large main subject 26 in the large empty space 24 on the mount 21. In the next step S55, the combining unit 1e duplicates and combines the small main subject 29 in the small empty space 23 on the mount 21.

  FIG. 10 shows a specific example of the composition processing in steps S54 and S55 of FIG. In step S56 of FIG. 10, duplicate images of the main subjects 26 and 29 used for composition are created from the selected frames 25 and 28. In this case, in order to simplify the process, a duplication image is not created along the contour of the main subject 26, but a duplication range is determined based on the face regions 26a and 29a detected by the face detection unit 3. . That is, it is assumed that the size of the face area 26a, 29a (head width) is three times the horizontal size of the main subject. In addition, in order to prevent the main subject from which the lower body has not been imaged from being combined in a floating state, and to reduce the amount of duplication calculation, from the upper end of the face areas 26a and 29a to the lower end of the imaging range Are the images of the person area of the main subject, and the rectangular areas 27 and 30 surrounded by these broken lines are taken as duplicate images.

  Next, in step S57, the composition unit 1e aligns the lower end of the duplicate image with the lower end of the empty space and pastes the duplicate image on the mount. FIG. 11 (d) shows this state. The duplicate image in the area 27 is combined with the wider empty space 24 on the mount 21 with the lower end coincident with the lower end of the imaging range. The duplicate image in the region 30 is combined with the narrower empty space 23 on the mount 21 with the lower end coincident with the lower end of the imaging range.

  As described above, in the present embodiment, a duplicate image of the selected main subject can be created and synthesized with a small amount of calculation.

  Although an example has been shown in which the empty space and the size of the main subject to be arranged are obtained by the size in the horizontal direction, the size in the vertical direction may be used, or the size in the horizontal and vertical directions may be used.

  In the present embodiment, an example in which only three main subjects are combined has been described, but it is obvious that the number of main subjects to be combined can be set as appropriate.

(Fourth embodiment)
FIGS. 12 and 13 relate to the fourth embodiment of the present invention, FIG. 12 is a flowchart showing subject copy composition processing, and FIG. 13 is an explanatory diagram for explaining a duplicate image.

  The present embodiment is different from the third embodiment only in subject composition processing, and the hardware configuration is the same as that in FIG. In the third embodiment, a rectangular duplicate image is synthesized. On the other hand, in the present embodiment, a duplicate image that matches the shape of the subject can be easily created.

  In step S61, the composition unit 1e creates a duplicate image including the main subject in the selected frame. FIG. 13 shows the main subject 35 in the selected frame. The synthesizing unit 1e compares the selected frame with the previous frame, assumes that the changed portion is the image portion of the main subject 35, and sets this image portion as the duplicate image 36. The synthesizing unit 1e pastes the created duplicate image on the empty space on the mount. In this case, the vertical position of the face area of the created duplicate image is aligned and pasted (step S62).

  As described above, in the present embodiment, it is possible to easily generate a duplicate image by comparing frames and estimating the changed portion as the image portion of the main subject.

  This method can also be used to determine the free space on the mount.

  FIG. 14 is a flowchart showing a determination flow of such an empty space.

  The mount determining unit 1c compares the frame selected as the mount with the preceding and succeeding frames (step S65). The mount determining unit 1c determines that the portion changed between the mount frame and the preceding and following frames is the image portion of the main subject, and cuts out the remaining portion as an empty space (step S66). In this way, an empty space can be obtained relatively easily.

(Fifth embodiment)
FIGS. 15 to 17 relate to the fifth embodiment of the present invention, FIGS. 15 and 16 are flowcharts showing frame selection processing and composition processing, and FIG. 17 is an explanatory diagram for explaining a composite image.

  This embodiment is different from the third or fourth embodiment only in the method for selecting or synthesizing the frames to be copied, and the hardware configuration is the same as that in FIG.

  In the third or fourth embodiment, duplicate images of a large main subject and a small main subject are arranged corresponding to the size of the empty space. However, depending on the size of the empty space and the size of the duplicate image, it is conceivable that the main subjects overlap each other or protrude from the imaging range.

  FIG. 17A shows an example of a composite image in this case. Duplicated images 43 and 44 based on two main subjects are arranged on a mount 41 including one main subject 42. The sum of the horizontal sizes of the duplicate images 43 and 44 based on the two main subjects is larger than the sum of the empty spaces, and the duplicate images 43 and 44 overlap the image of the subject 42 so that a part of the image of the main subject 42 is obtained. Is missing.

  Therefore, in the flow of FIG. 15, a frame including an image of the main subject that can be arranged in the imaging range is selected according to the size of the empty space.

  In step S39 of FIG. 6 and step S44 of FIG. 8, the frame to be copied is selected without comparing with the size of the empty space. In the flowchart of FIG. 15, before selecting a frame from which a duplicate image is to be generated, the size of an empty space is obtained and a frame is selected according to the size of the empty space.

  The processing in steps S71 and S72 in FIG. 15 is the same as that in steps S51 and S52 in FIG. 9, and the size is determined by obtaining the size of the free space. The duplication determination unit 1d selects a frame including the main subject that is the source of the duplication image according to the size of the empty space. The synthesizing unit 1e generates a duplicate image based on the image of the main subject in the selected frame, and generates a synthesized image by superimposing the duplicate images generated on the mount.

  By adopting the flow of FIG. 15 as described above, it is possible to prevent the main subject images from overlapping and missing a part of the image or the main subject image from protruding from the imaging range.

  The frame selection method based on the flow of FIG. 15 may be performed in combination with the process of step S39 of FIG. 6 or step S44 of FIG. That is, before selecting in the steps of S39 and S44, the size of the empty space is obtained, and the frame having a large change amount is selected in these steps S39 and S44 from among the frames from which the duplicate image entering the empty space is obtained. You may make it do.

  On the contrary, after performing the processing of steps S39 and S44, the mount frame and the selected frame that is the origin of the duplicate image may be selected. That is, in this case, in steps S39, S44, etc., the duplication determination unit 1d selects a frame including a main subject having a large change amount as a selected frame candidate. Next, the signal processing and control unit 1 compares the size of the main subject in all the images between the images. Based on this comparison, the mount determining unit 1c selects the image of the mount so that a desired number of main subject images are arranged on the mount, and the duplication determining unit 1d selects the selected frame candidates. Select the selected scene to be used for duplication. The synthesizing unit 1e generates a duplicate image based on the main subject in the selected frame, and synthesizes it on the selected mount, thereby generating a synthesized image.

  By generating a composite image by such a procedure, there is an advantage that a duplicate image of a subject can be arranged effectively according to the size of all main subjects in all the images.

  In the flow of FIG. 16, the copy image is reduced to a size at which a duplicate image can be arranged according to the size of the empty space.

  In FIG. 16, in steps S71 and S72, the size of the empty space is obtained to determine the size. Next, the synthesis unit 1e determines the size of the main subject of the selected frame.

  In step S77, the synthesizing unit 1e performs the reduction process of the duplicate image so that the duplicate image based on the larger main subject fits in the larger empty space, and then performs the pasting. Similarly, in step S78, the synthesizing unit 1e performs a reduction process of the duplicate image so that the duplicate image based on the smaller main subject fits in the smaller empty space, and then combines them.

  FIGS. 7B and 7C show this reduction processing. In FIG. 7B, duplicate images 53 and 54 based on two main subjects are arranged on a mount 51 including one main subject 52. Since the duplicate images 53 and 54 based on the two main subjects are relatively large in size, the main subject 52 and the duplicate images 53 and 54 are displayed so as to overlap each other.

  On the other hand, FIG. 7C shows an example in which reduced duplicate images 53 ′ and 54 ′ obtained by reducing the duplicate images 53 and 54 are arranged on the mount 51. The size of the reduced duplicate images 53 ′ and 54 ′ is a size corresponding to the empty space, and the reduced duplicate images 53 ′ and 54 ′ and the subject 42 are arranged on the mount 51 without overlapping.

  As described above, in the present embodiment, the frame including the main subject is obtained corresponding to the size of the empty space, or the duplicate image based on the main subject is reduced and then arranged in the empty space. The images of the subjects can be arranged in a sufficient size without overlapping each other on the mount.

  In the above description, the example in which the duplicate image is reduced has been described. However, depending on the size of the empty space and the size of the duplicate image based on the main subject, the duplicate image may be enlarged and combined.

  Further, in this embodiment, an example in which duplicate images are arranged in two empty spaces has been described. However, three or more duplicate images may be arranged by obtaining three or more empty spaces. Is clear.

  In each of the above embodiments, the image of the main subject on the mount is used as it is. However, the copy determining unit generates a copy image based on the main subject on the mount and places the copy image at an arbitrary position on the mount. You may comprise. In this case, there is an advantage that the degree of freedom of arrangement of each duplicate image arranged on the mount is improved.

  In addition, the image generation apparatus according to the present invention does not need to have a shooting function, and may include an image input unit that captures a moving image signal instead of the imaging unit. That is, the image generation apparatus according to the present invention can be applied to a photo frame or the like on which a still image is displayed when a moving image file is input.

  In addition, the final image does not have to be limited to a still image, and if you prepare multiple images synthesized in accordance with the idea of the present invention and display them sequentially, it will not be a lively video with special effects. Needless to say.

    DESCRIPTION OF SYMBOLS 1 ... Image generation apparatus, 1b ... Main subject determination part, 1c ... Mount determination part, 1d ... Duplication determination part, 1e ... Composition part, 2 ... Imaging part, 3 ... Face detection part, 4 ... Recording part, 4b ... Temporary recording Part, 6 ... operation determination part, 7 ... display part.

Claims (9)

A main subject determination unit for determining a main subject from among persons included in a moving image having frames of a predetermined cycle;
A duplicating unit that selects a plurality of images including the main subject among the frames of the predetermined period as a selected frame, and generates a duplicate image using the image portion of the main subject from the plurality of selected frames,
A combining unit that generates a combined image by combining a plurality of the duplicate images;
Comprising
The duplication unit performs selection of the selected frame at a predetermined time different from the predetermined period after the moving image is captured. A temporary recording section;
The image generation apparatus according to claim 1, further comprising: a copy determination unit that records information on the selected frame among the frames of the predetermined period in the temporary recording unit. The image generating apparatus according to claim 2, wherein the main subject is a person, and the temporary recording unit records face position information. The image generating apparatus according to any one of claims 1 to 3, wherein the combining unit selects a frame having the predetermined cycle as a mount, and generates the combined image on the mount. The image generation apparatus according to claim 4, wherein the mount includes an image portion of the main subject. The composition unit selects the first frame as the starting point or the last frame as the end point of the movement of the main subject as the mount.
The image generation apparatus according to claim 4. A main subject determination step for determining a main subject from among persons included in a moving image having frames of a predetermined cycle;
A duplication step of selecting a plurality of images including the main subject among the frames of the predetermined cycle as selection frames, and generating duplicate images using the image portions of the main subject from the plurality of selection frames,
Generating a composite image by combining a plurality of the duplicate images;
Comprising
The duplication step performs selection of the selected frame at a predetermined time different from the predetermined period after the moving image is captured. On the computer,
A main subject determination step for determining a main subject from among persons included in a moving image having frames of a predetermined cycle;
A duplication step of selecting a plurality of images including the main subject among the frames of the predetermined cycle as selection frames, and generating duplicate images using the image portions of the main subject from the plurality of selection frames,
Generating a composite image by combining a plurality of the duplicate images;
And execute
In the duplication step, the selection of the selected frame is performed at a predetermined time different from the predetermined cycle after the moving image is captured. An imaging unit that captures a moving image composed of frames with a predetermined period;
A main subject determination unit that receives the moving image from the imaging unit and determines a main subject from persons included in the moving image;
A duplicating unit that selects a plurality of images including the main subject among the frames of the predetermined period as a selected frame, and generates a duplicate image using the image portion of the main subject from the plurality of selected frames,
A combining unit that generates a combined image by combining a plurality of the duplicate images;
Comprising
The image generation system according to claim 1, wherein the copying unit performs selection of the selected frame at a predetermined time different from the predetermined period after the moving image is captured.

Images