JP2010239459A - Image combination apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To produce a highly interesting image by giving motion to an image after combination. <P>SOLUTION: An imaging apparatus 100 comprises: a motion detecting unit 8f; and an image combining unit 8g. On the basis of motion image data recording a moving subject, the motion detecting unit 8f detects "motion information" of the subject and the image combining unit 8g produces a composite motion picture by combining a subject cut-out image and a background image for predetermined motion in the subject region of the subject cut-out image while reflecting the "motion information" detected by the motion detecting unit. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image composition apparatus and a program for synthesizing a plurality of images.

  Conventionally, a technique for generating a composite image by combining a subject image with a background image or a frame image is known (see, for example, Patent Document 1).

JP 2004-159158 A

  However, when still images are combined as in Patent Document 1, the position of the subject remains fixed at the combined position in the combined image, resulting in an uninteresting image.

  Therefore, an object of the present invention is to provide an image composition device and a program that can give a motion to a synthesized image and can generate a highly interesting image.

In order to solve the above-described problem, an image composition device according to a first aspect of the present invention provides:
Still image acquisition means for acquiring the first and second still images, detection means for detecting changes over the plurality of images of the subject commonly included in the plurality of images, and the second still image as a background, And synthesizing means for generating a synthesized moving image obtained by synthesizing the first still image reflecting the change detected by the detecting means.

According to a second aspect of the present invention, in the image composition device according to the first aspect,
The change of the subject over the plurality of images is a change per unit time of the position of the subject in the angle of view of the plurality of images, a change per unit time of a range occupied by the image of the subject in the plurality of images, It is at least one of the change per unit time of the rotation angle of the subject with respect to at least one of the horizontal direction and the vertical direction of the plurality of images.

According to a third aspect of the present invention, in the image composition device according to the first or second aspect,
The apparatus further comprises storage means for storing the changes detected by the detection means as a plurality of images, the synthesizing means reads out the plurality of images from the storage means, and uses the second still image as a background image. And generating means for generating a plurality of synthesized still images obtained by synthesizing the first still image at the position of the subject in each of the plurality of images read by the reading means.

According to a fourth aspect of the present invention, in the image synthesizing device according to the third aspect,
The generating means generates composite content information sequentially based on the change detected by the detecting means, and the first content according to the composite content information sequentially generated by the composite content generating means. And setting means for setting at least one of a still image combining position, size and shape, and rotation angle.

The invention according to claim 5 is the image composition device according to any one of claims 1 to 4,
The image acquisition device further includes a moving image or a continuous shot image including a plurality of continuous still images, and the detection unit converts the moving image and the continuous shot image acquired by the image acquisition device to the plurality of images. As described above, a change in the subject included in the plurality of images in common with the plurality of images is detected.

According to a sixth aspect of the present invention, in the image composition device according to the fifth aspect,
The image acquisition means includes an imaging means.

The invention according to claim 7 is the image composition apparatus according to any one of claims 1 to 6,
The first still image is an extracted image obtained by extracting a specific area from a predetermined still image.

The program of the invention described in claim 8 is:
With the computer, still image acquisition means for acquiring the first and second still images, detection means for detecting changes over the plurality of images of the subject included in common to the plurality of images, the second still image as a background, It is characterized by functioning as a synthesizing unit that generates a synthesized moving image obtained by synthesizing the first still image reflecting the change detected by the detecting unit.

  ADVANTAGE OF THE INVENTION According to this invention, a motion can be given to the image after the synthesis | combination of still images, and an image with high interest property can be produced | generated.

It is a block diagram which shows schematic structure of the imaging device of one Embodiment to which this invention is applied. 3 is a flowchart illustrating an example of an operation related to subject clipping processing by the imaging apparatus of FIG. 1. It is a figure which shows typically an example of the image which concerns on the to-be-photographed object cutting process of FIG. 3 is a flowchart illustrating an example of an operation related to background image generation processing by the imaging apparatus of FIG. 1. 3 is a flowchart illustrating an example of an operation related to a composite image generation process by the imaging apparatus of FIG. 1. It is a flowchart which shows an example of the operation | movement which concerns on the 1st image composition process in the composite image generation process of FIG. It is a flowchart which shows an example of the operation | movement which concerns on the 2nd image composition process in the composite image generation process of FIG. It is a figure which shows typically an example of the image synthesize | combined by the synthesized image generation process of FIG. It is a figure which shows typically an example of the moving image data which concern on the synthesized image generation process of FIG. It is a figure which shows typically an example of the to-be-moved subject extracted from the moving image data of FIG. It is a figure which shows typically an example of the synthetic | combination moving image produced | generated by the synthetic image production | generation process of FIG.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus 100 according to an embodiment to which the present invention is applied.
The imaging apparatus 100 according to the present embodiment detects the “movement information” of the moving subject O based on the motion image data in which the movement of the moving subject O (see FIG. 9A and the like) is recorded. ”, The subject cutout image P3 and the background image P4 (FIG. 8A) so that the subject region S of the subject cutout image P3 (see FIG. 8B) is operated in accordance with the movement of the moving subject O. )) To generate a composite moving image of the subject composite image P5 (see FIGS. 11A to 11D).
Specifically, as illustrated in FIG. 1, the imaging apparatus 100 includes a lens unit 1, an electronic imaging unit 2, an imaging control unit 3, an image data generation unit 4, an image memory 5, and a feature amount calculation unit. 6, a block matching unit 7, an image processing unit 8, a recording medium 9, a display control unit 10, a display unit 11, an operation input unit 12, and a CPU 13.
In addition, the imaging control unit 3, the feature amount calculation unit 6, the block matching unit 7, the image processing unit 8, and the CPU 13 are designed as, for example, a custom LSI 1A.

The lens unit 1 includes a plurality of lenses and includes a zoom lens, a focus lens, and the like.
Although not shown, the lens unit 1 includes a zoom drive unit that moves the zoom lens in the optical axis direction and a focus drive unit that moves the focus lens in the optical axis direction when imaging a subject. May be.

  The electronic imaging unit 2 is composed of an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-oxide Semiconductor), for example, and converts an optical image that has passed through various lenses of the lens unit 1 into a two-dimensional image signal. To do.

Although not shown, the imaging control unit 3 includes a timing generator, a driver, and the like. Then, the imaging control unit 3 scans and drives the electronic imaging unit 2 with a timing generator and a driver, converts the optical image into a two-dimensional image signal with the electronic imaging unit 2 every predetermined period, and the electronic imaging unit 2 Image frames are read out from the imaging area for each screen and output to the image data generation unit 4.
In addition, the imaging control unit 3 performs adjustment control of conditions when imaging a subject such as AF (automatic focusing process), AE (automatic exposure process), and AWB (automatic white balance).

The imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 configured in this way function as an imaging unit as a subject presence image P1 (see FIG. 3A) and a background image P4 (see FIG. 3A) related to image composition processing. 8 (a)).
In addition, the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 capture the subject clipped image P3 (see FIG. 3) in a state where the imaging conditions for capturing the subject presence image P1 are fixed after the subject presence image P1 is captured. The non-existing subject image P2 (see FIG. 3B) for generating 3) is captured.

The image data generation unit 4 appropriately adjusts the gain for each RGB color component with respect to the analog value signal of the image frame transferred from the electronic imaging unit 2, and then performs sample holding by a sample hold circuit (not shown). The digital signal is converted into digital data by a / D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a digital luminance signal Y and color difference signal Cb , Cr (YUV data).
The luminance signal Y and the color difference signals Cb and Cr output from the color process circuit are DMA-transferred to an image memory 5 used as a buffer memory via a DMA controller (not shown).

  The image memory 5 is composed of, for example, a DRAM or the like, and temporarily stores data processed by the feature amount calculation unit 6, the block matching unit 7, the image processing unit 8, the CPU 13, and the like.

The feature quantity calculation unit 6 performs feature extraction processing for extracting feature points from the subject non-existing image P2 with reference to the subject non-existing image P2. Specifically, the feature amount calculation unit 6 selects a predetermined number (or a predetermined number or more) of highly featured block areas (feature points) based on the YUV data of the subject non-existing image P2, and selects the relevant block. Is extracted as a template (for example, a square of 16 × 16 pixels).
Here, the feature extraction process is a process of selecting a feature having a high characteristic convenient for tracking from a large number of candidate blocks.

  The block matching unit 7 performs block matching processing for aligning the subject non-existing image P2 and the subject existing image P1. Specifically, the block matching unit 7 optimally matches where the template extracted by the feature extraction process corresponds to in the subject existing image P1, that is, the template pixel value optimally matches in the subject existing image P1. Search for a position (corresponding region). Then, an optimum offset between the subject non-existing image P2 and the subject existing image P1 having the best evaluation value (for example, sum of squared differences (SSD), sum of absolute differences (SAD), etc.) is calculated. Calculated as a template motion vector.

The image processing unit 8 includes an alignment unit 8a that performs alignment between the subject presence image P1 and the subject non-existence image P2.
The alignment unit 8a calculates a coordinate conversion formula (projection conversion matrix) of each pixel of the subject presence image P1 with respect to the subject non-existence image P2 based on the feature points extracted from the subject non-existence image P2, and the coordinate conversion formula Accordingly, the subject presence image P1 is coordinate-converted to align with the subject non-existence image P2.

  Further, the image processing unit 8 generates difference information of corresponding pixels between the subject presence image P1 and the subject non-existence image P2 aligned by the alignment unit 8a, and the subject exists based on the difference information. A subject region extraction unit 8b that extracts a subject region S including the subject from the image P1 is provided.

Further, the image processing unit 8 specifies a position of the subject region S extracted in the subject presence image P1, and a position information generation unit 8c that generates position information indicating the position of the subject region S in the subject presence image P1. It has.
Here, as the position information, for example, an alpha map can be cited. The alpha map is a weight for alpha blending the image of the subject area S with respect to a predetermined background for each pixel of the subject existing image P1. It is expressed as a value (0 ≦ α ≦ 1).

Further, the image processing unit 8 does not transmit a pixel having an alpha value of 1 out of each pixel of the subject existing image P1 to a predetermined single color image (not shown) based on the generated alpha map. In addition, a cut-out image generation unit that generates image data of the cut-out image P3 of the subject (see FIG. 3C) by combining the image of the subject with a predetermined single-color image so that the pixels with an alpha value of 0 are transmitted. 8d.
Thereby, the cut-out image generation unit 8d acquires the subject cut-out image P3 obtained by cutting out the area including the subject from the subject existing image P1 in which the background and the subject exist.

In addition, the image processing unit 8 includes a motion image acquisition unit 8e that acquires motion image data (details will be described later) recorded as a moving image including a plurality of continuous still images or continuous shot images of the movement of the moving subject O. Yes.
Specifically, the motion image acquisition unit 8e receives the motion image data instructed based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user among the motion image data recorded on the recording medium 9. Read and get.

The image processing unit 8 includes a motion detection unit 8f that detects “motion information” of the moving subject O from the motion image data acquired by the motion image acquisition unit 8e.
Specifically, the motion detection unit 8f performs predetermined image processing on the motion image data to identify the moving subject O included in the motion image data, and displays “ "Motion information" (changes over a plurality of images of a moving subject O included in a plurality of images in common) is detected. For example, when moving image data M composed of a plurality of image frames f1,... Recorded in the Motion-JPEG format is acquired as the moving image data by the moving image acquisition unit 8e, the motion detection unit 8f A smoothing process is performed on the frames f1,... To remove noise, and then a first-order differentiation process is performed by a matrix operation to extract a contour line (edge part) of the pixel set. Then, the motion detection unit 8f performs an expansion process on the extracted image to enhance edge connectivity, and then uses an isolated connected component (image set) having a smaller number of connected pixels than a predetermined number. By performing the removal process and specifying the outer peripheral contour of the moving subject O having the largest area, the position of the image region surrounded by the outer peripheral contour of the moving subject O is determined as the position of the moving subject O in each image frame f. As specified.

Then, the motion detector 8f specifies the center coordinates of the moving subject O in each image frame f based on the position of the moving subject O in each of the plurality of image frames f,... Constituting the moving image data M. By scanning in a predetermined direction (for example, the X-axis direction from the left end to the right end, the Y-axis direction from the upper end to the lower end, etc.) from a predetermined position (for example, the left end and the upper end) of each image frame f, A rectangular frame W (see FIG. 10A) surrounding the moving subject O is specified, and the size of the rectangular frame W in the X-axis (horizontal) direction and the size in the Y-axis (vertical) direction (vertical and horizontal sizes) are determined. calculate.
The motion detection unit 8f also coordinates the corners of the rectangular frames W (see FIGS. 10B to 10D) of the second and subsequent image frames f with respect to the rectangular frame W of the first image frame f1. The relative position information is sequentially calculated as the amount of movement. Furthermore, the motion detection unit 8f sequentially calculates relative range information as a scaling factor in the X-axis direction and the Y-axis direction of the rectangular frames W of the second and subsequent image frames f with respect to the rectangular frame W of the first image frame f1. Furthermore, the motion detection unit 8f sequentially calculates relative angle information as the rotation angle of the rectangular frames W of the second and subsequent image frames f with respect to the rectangular frame W of the first image frame f1.
In this way, the motion detection unit 8f detects relative position information, relative range information, and relative angle information as “motion information” per unit time of the moving subject O in the second and subsequent image frames f.

Note that the above-described specific method of moving the subject O is an example and is not limited to this, and may be arbitrarily changed as appropriate.
For example, when a specific subject is in focus in the first image frame f1, the imaging target in the AF area (focused position) may be specified as the moving subject O. For the second and subsequent image frames f, the moving subject O detected from the first image frame f1 is specified as a moving object, and the moving subject O is moved between adjacent image frames f using a predetermined moving object analysis technique. May be tracked.
Moreover, it is good also as a structure which arrange | positions the touchscreen (not shown) as the operation input part 12 on a display screen, and specifies the to-be-photographed object O according to the contact position of a touchscreen. That is, the position of the moving subject O in the image frame f displayed on the display unit 11 is input by a predetermined touch operation of the touch panel by the user, and the motion detection unit 8f is input based on the touch operation of the touch panel. The moving object O may be specified based on the position information.
Further, in order to increase the speed of each of the above processes, a low resolution image in which the resolution of each image frame f is reduced may be used.

  The image processing unit 8 includes an image composition unit 8g that synthesizes the subject cutout image P3 and the background image P4. Specifically, the image composition unit 8g acquires a user-desired background image P4 from the recording medium 9, and transmits pixels with an alpha value of 0 among the pixels of the background image P4. Is overwritten with the pixel value of the corresponding pixel of the subject cutout image P3, and among the pixels of the background image P4, the pixel with an alpha value of 0 <α <1 is a 1's complement (1-α ) Is used to generate the cropped image P3 using the one's complement (1-α) in the alpha map. A value blended with a single background color is calculated, and the value is subtracted from the subject cutout image P3, and is combined with an image obtained by cutting out the subject region S (background image × (1-α)).

The image composition unit 8g includes a composition control unit 8h that controls composition of the subject cutout image P3 with the background image P4. That is, the image composition unit 8g adjusts the composition position, size, shape, rotation angle, and the like of the subject cutout image P3 with respect to the background image P4 under the control of the composition control unit 8h, and combines the plurality of image frames g by image composition. ,... Are generated to generate a composite moving image of the subject composite image P5 including the plurality of image frames g,.
Specifically, the composition control unit 8h includes the second and subsequent images constituting the composition video based on “motion information” such as relative position information, relative range information, and relative angle information detected by the motion detection unit 8f. The composite content information of the subject clipped image P3 is sequentially generated for the frame g.
Here, as the composition content information, the center coordinates of the subject region S related to the composition position of the subject clipped image P3 generated based on the relative position information, and the composition of the subject clipped image P3 generated based on the relative range information. The magnification of the subject area S relative to the first image frame g1 related to the size, the rotation angle of the subject area S relative to the first image frame g1 related to the direction of synthesis of the subject cutout image P3 generated based on the relative angle information, and the like. Can be mentioned.
Then, the composition control unit 8h composes the first image frame g1 of the subject cutout image P3 at a desired position specified by the user by the image composition unit 8g, and then sequentially generates the second and subsequent image frames g. The composition position, size, shape, rotation angle, and the like of the subject clipped image P3 are set according to the composition content information of the subject clipped image P3, and are combined by the image composition unit 8g. For example, as shown in FIG. 10, the moving subject O of the second image frame f2 of the moving image data M is shifted to the right with respect to the first image frame f1 and extends vertically. Therefore, the second image frame g2 has a shape shifted so as to extend in the vertical direction while the composition position is shifted to the right with respect to the first image frame g1 in substantially the same manner as the subject O in which the subject region S moves. To synthesize. Further, the third image frame f3 of the moving image data M is shifted further to the right than the second image frame f2, but the size and shape are substantially the same as the first image frame f1. Therefore, the third image frame g3 is further shifted to the right side from the second image frame g2 in substantially the same manner as the subject O in which the composition position of the subject region S moves. Synthesize so that the shapes are approximately equal. Further, the nth image frame fn of the moving image data M is shifted further to the right than the third image frame f3 and is contracted in the vertical direction with respect to the first image frame f1. The n-th image frame gn is shifted further to the right than the third image frame g3 substantially in the same manner as the subject O in which the composition position of the subject region S moves, and in the vertical direction with respect to the first image frame g1. Synthesize as if shrunk.

In this way, the image composition unit 8g has at least one of the composition position, size, shape, and rotation angle of the subject cutout image P3 with respect to the background image P4 for the plurality of image frames g,. The subject image clipping g is sequentially generated by changing the image to generate the image frame g so that the moving subject clipping image P3 is caused to perform a predetermined operation based on the “motion information” detected by the motion detection unit 8f. A synthesized moving image is generated by synthesizing the image P3 and the background image P4.
The number of image frames g combined may be, for example, the same as the number of image frames f of the moving image data M, or may be a number specified in advance according to the time of the generated combined moving image. .

The recording medium 9 is constituted by, for example, a nonvolatile memory (flash memory) or the like, and the still image data of the background image P4 and the subject cutout image P3 encoded by the JPEG compression unit (not shown) of the image processing unit 8 is stored. Remember.
Further, the image data of the subject cutout image P3 is associated with the alpha map generated by the position information generation unit 8c of the image processing unit 8, and the extension of the image data of the subject cutout image P3 is set to “.jpe”. Saved.

Further, the recording medium 9 includes a plurality of image frames f generated by imaging the moving subject O by the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. ) To store the motion image data encoded in a predetermined compression format.
The motion image data is obtained by recording the movement of the moving subject O by a predetermined recording method. For example, a plurality of continuous image frames (for example, n image frames) f captured at a predetermined imaging frame rate, .., Moving image data M (see FIG. 9), continuous shot image data continuously shot at a predetermined shutter speed, and the like.
Here, as the moving image data M, for example, as illustrated in FIGS. 9A to 9D, the person's jumping motion is captured from the side and recorded in the Motion-JPEG format. 9A to 9D schematically show n consecutive image frames f1 to fn constituting the moving image data M, and these image frames f1 to fn are represented on a time axis. 9 (a) to 9 (d).

The display control unit 10 performs control for reading display image data temporarily stored in the image memory 5 and displaying the read image data on the display unit 11.
Specifically, the display control unit 10 includes a VRAM, a VRAM controller, a digital video encoder, and the like. The digital video encoder periodically reads the luminance signal Y and the color difference signals Cb and Cr read from the image memory 5 and stored in the VRAM (not shown) under the control of the CPU 13 from the VRAM via the VRAM controller. Are read out, a video signal is generated based on these data, and is output to the display unit 11.

  The display unit 11 is, for example, a liquid crystal display device, and displays an image captured by the electronic imaging unit 2 based on a video signal from the display control unit 10 on a display screen. Specifically, the display unit 11 is configured to display a live view image based on a plurality of image frames f generated by imaging an object by the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 in the imaging mode. Or a REC view image captured as the actual captured image.

  The operation input unit 12 is for performing a predetermined operation of the imaging apparatus 100. Specifically, the operation input unit 12 includes a shutter button 12a related to a subject shooting instruction, a selection determination button 12b related to a selection instruction for an imaging mode, a function, and the like, and an instruction for setting an initial composite position of the subject cutout image P3, and a zoom amount. A zoom button (not shown) relating to the adjustment instruction is provided, and a predetermined operation signal is output to the CPU 13 in accordance with the operation of these buttons.

  The CPU 13 controls each part of the imaging device 100. Specifically, the CPU 13 performs various control operations in accordance with various processing programs (not shown) for the imaging apparatus 100.

Next, the subject clipping process performed by the imaging apparatus 100 will be described with reference to FIGS.
FIG. 2 is a flowchart illustrating an example of an operation related to the subject clipping process. FIG. 3 is a diagram schematically illustrating an example of an image related to the subject clipping process.
The subject clipping process is executed when a subject clipping mode is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user. It is.

  As shown in FIG. 2, first, the CPU 13 causes the display control unit 10 to display a live view image based on a plurality of image frames generated by imaging an object by the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. Is displayed on the display screen of the display unit 11 and superimposed on the live view image, an imaging instruction message of the subject existing image P1 is displayed on the display screen of the display unit 11 (step S1).

Next, the CPU 13 determines whether or not an imaging instruction has been input based on a predetermined operation of the shutter button 12a of the operation input unit 12 by the user (step S2). If it is determined that an imaging instruction has been input (step S2; YES), the CPU 13 instructs the imaging control unit 3 to focus the focus lens, exposure conditions (shutter speed, aperture, amplification factor, etc.) and white. By adjusting conditions such as balance, an optical image of the subject existing image P1 (see FIG. 3A) is picked up by the electronic image pickup unit 2 under a predetermined condition, and the image data generation unit 4 causes the electronic image pickup unit 2 to YUV data of the image frame of the transferred subject presence image P1 is generated (step S3). The YUV data of the subject existing image P1 is temporarily stored in the image memory 5.
Further, the CPU 13 controls the imaging control unit 3 to maintain a state in which conditions such as a focus position, an exposure condition, and a white balance at the time of imaging the subject presence image P1 are fixed.

Then, the CPU 13 causes the display control unit 10 to display a live view image on the display unit 11 based on a plurality of image frames generated by imaging the subject by the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. And the image of the semi-transparent display mode of the subject presence image P1 and the imaging instruction message of the subject non-existence image P2 are superimposed on the live view image and displayed on the display screen of the display unit 11 (step S4). .
Thereafter, the CPU 13 determines whether or not an imaging instruction has been input based on a predetermined operation of the shutter button 12a of the operation input unit 12 by the user (step S5). Then, after the user moves the subject out of the angle of view or waits for the subject to move, the user adjusts the camera position so that the subject non-existing image P2 overlaps the semi-transparent image of the subject existing image P1. When it is determined that the shutter button 12a of the operation input unit 12 is operated for a predetermined time and an imaging instruction is input (step S5; YES), the CPU 13 causes the imaging control unit 3 to display the subject nonexistent image P2 (FIG. 3). The optical image (see (b)) is captured by the electronic imaging unit 2 under conditions fixed after the subject presence image P1 is captured, and the subject non-existence image P2 transferred from the electronic imaging unit 2 to the image data generation unit 4 is captured. YUV data of the subject non-existing image P2 is generated based on the image frame (step S6). The YUV data of the subject existing image P1 is temporarily stored in the image memory 5.

Next, the CPU 13 uses the YUV data of the subject non-existing image P2 temporarily stored in the image memory 5 in the feature amount calculation unit 6, the block matching unit 7, and the image processing unit 8 as a reference to YUV of the subject existing image P1. A projection transformation matrix for projectively transforming data is calculated using a predetermined image transformation model (for example, a similarity transformation model or a joint transformation model) (step S7).
Specifically, the feature amount calculation unit 6 selects a predetermined number (or a predetermined number or more) of highly featured block areas (feature points) based on the YUV data of the subject non-existing image P2, and selects the relevant block. Is extracted as a template. Then, the block matching unit 7 searches the subject existing image P1 for a position where the pixel value of the template extracted by the feature extraction process is optimally matched, and the evaluation value of the difference between the pixel values is the best. An optimal offset between the subject non-existing image P2 and the subject existing image P1 is calculated as a motion vector of the template. Then, the alignment unit 8a of the image processing unit 8 statistically calculates the entire motion vector based on the motion vectors of the plurality of templates calculated by the block matching unit 7, and performs feature point correspondence related to the motion vector. Using this, a projective transformation matrix of the subject existing image P1 is calculated.

  Next, the CPU 13 causes the alignment unit 8a to perform projective transformation of the subject presence image P1 based on the calculated projective transformation row example, whereby the YUV data of the subject presence image P1 and the YUV data of the subject non-existence image P2 Is performed (step S8).

Then, the CPU 13 causes the subject region extraction unit 8b of the image processing unit 8 to perform a process of extracting the subject region S including the subject from the subject presence image P1 (step S9).
Specifically, the subject region extraction unit 8b applies a low-pass filter to each of the YUV data of the subject presence image P1 and the YUV data of the subject non-existence image P2 to remove the high-frequency component of each image. Thereafter, the subject area extraction unit 8b calculates a difference for each corresponding pixel between the subject existing image P1 and the subject non-existing image P2 subjected to the low-pass filter, and generates a difference map. Subsequently, the subject region extraction unit 8b binarizes the dissimilarity map relating to each pixel with a predetermined threshold value, and then performs a contraction process to remove a region in which the difference is caused by fine noise or camera shake from the dissimilarity map. Do. After that, the subject region extraction unit 8b performs a labeling process to remove a region below a predetermined value or a region other than the maximum region, identifies the largest island pattern as the subject region S, and corrects the shrinkage. Expansion processing is performed.

  Next, the CPU 13 causes the position information generation unit 8c of the image processing unit 8 to generate an alpha map indicating the position of the extracted subject region S in the subject existing image P1 (step S10).

Thereafter, the CPU 13 performs a process of generating image data of a subject cutout image P3 (see FIG. 3C) obtained by combining the subject image with a predetermined single color image in the cutout image generation unit 8d of the image processing unit 8. (Step S11).
Specifically, the cut-out image generation unit 8d reads out the subject existing image P1, the single color image, and the alpha map and develops them in the image memory 5, and thereafter, for all the pixels of the subject existing image P1, the pixels having an alpha value of 0. (Α = 0) for transmission, and for pixels with an alpha value of 0 <α <1, (0 <α <1), blending with a predetermined single color is performed, and for pixels with an alpha value of 1 (α = 1) Do nothing and do not transmit light to a predetermined single color.

Thereafter, the CPU 13 associates the alpha map generated by the position information generation unit 8c of the image processing unit 8 with the image data of the subject clipped image P3 in a predetermined storage area of the recording medium 9, and stores the clipped image P3 of the subject clipped image P3. The extension of the image data is stored as one file with “.jpe” (step S12).
Thereby, the subject clipping process is completed. As a result, for example, still image data of a subject cutout image P3 in which a car is extracted as a subject from the background is generated.

Next, background image generation processing by the imaging apparatus 100 will be described with reference to FIG.
FIG. 4 is a flowchart illustrating an example of an operation related to the background image generation processing.
The background image generation process is a normal still image imaging process, and a still image is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user. This is a process executed when an imaging mode selection instruction is given.

  As shown in FIG. 4, first, the CPU 13 causes the display control unit 10 based on a plurality of image frames generated by imaging the background image P4 by the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. The live view image is displayed on the display screen of the display unit 11 (step S21).

  Next, the CPU 13 determines whether or not an imaging instruction has been input based on a predetermined operation of the shutter button 12a of the operation input unit 12 by the user (step S22). If it is determined that an imaging instruction has been input (step S22; YES), the CPU 13 instructs the imaging control unit 3 to focus the focus lens, exposure conditions (shutter speed, aperture, amplification factor, etc.) and white. By adjusting conditions such as balance, an optical image of the background image P4 (see FIG. 8B) is picked up by the electronic image pickup unit 2 under predetermined conditions (step S23).

Subsequently, the CPU 13 causes the image data generation unit 4 to generate the YUV data of the image frame of the background image P4 transferred from the electronic imaging unit 2, and then the background image is stored in a predetermined storage area of the recording medium 9. The P4 YUV data is stored as an Exif format image file (step S24).
Thus, the background image generation process is terminated. As a result, for example, still image data of the background image P4 in which the forest is recorded is generated.

Next, the composite image generation process will be described in detail with reference to FIGS.
FIG. 5 is a flowchart illustrating an example of an operation related to the composite image generation process. FIG. 6 is a flowchart showing an example of an operation related to the first image composition process in the composite image generation process, and FIG. 7 is a flowchart showing an example of an operation related to the second image composition process in the composite image generation process. is there. FIGS. 8A and 8B are diagrams schematically illustrating an example of an image that is combined in the combined image generation process. FIGS. 9A to 9D are diagrams schematically illustrating an example of moving image data related to the composite image generation process, and FIGS. 10A to 10D are moving images. It is a figure which shows typically an example of the to-be-photographed subject extracted from the data. FIGS. 11A to 11D are diagrams schematically illustrating an example of a synthesized moving image generated by the synthesized image generation process.

The composite image generation processing is executed when an image combination mode is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user. It is processing.
As shown in FIG. 5, based on a predetermined operation of the operation input unit 12 by the user, a desired background image P4 (FIG. 8 (FIG. 8)) that becomes the background of the composite image among a plurality of images recorded on the recording medium 9. When a) (see a) is selected and specified, the image processing unit 8 reads the image data of the specified background image P4 and develops it in the image memory 5 (step S31).

Next, based on a predetermined operation of the operation input unit 12 by the user, a desired subject cutout image P3 (see FIG. 8B) is selected and designated from among a plurality of images recorded on the recording medium 9. Then, the image processing unit 8 reads out the image data of the designated subject cutout image P3 and develops it in the image memory 5 (step S32).
Subsequently, the CPU 13 causes the display screen of the display unit to display a synthetic moving image generation instruction selection screen for performing a predetermined operation on the image of the subject region S of the subject cutout image P3 that is a still image, and the user selects the operation input unit 12. It is determined whether or not an instruction to generate a composite moving image has been input based on a predetermined operation of the determination button 12b (step S33).

Here, if it is determined that an instruction to generate a synthetic moving image has been input (step S33; YES), the CPU 13 selects a selection determination button of the operation input unit 12 by the user from the motion image data recorded on the recording medium 9. The motion image acquisition unit 8e is caused to read the motion image data designated based on the predetermined operation 12b (step S34). Here, for example, moving image data M (see FIGS. 9A to 9D) including a plurality of image frames f,... Recorded in the Motion-JPEG format by continuous shooting is read as motion image data. Shall.
Subsequently, the CPU 13 performs predetermined image processing on the read moving image data M to identify the moving subject O included in the moving image data M, and changes the moving subject O per unit time. Is detected as “motion information” (step S35).
Here, the “motion information” includes a change per unit time of the center position of the moving subject O at the angle of view of each image frame f constituting the moving image data M (hereinafter referred to as relative position information), a moving image. The change per unit time of the range of the image of the moving subject O in each image frame f constituting the data M (hereinafter referred to as relative range information), the horizontal direction and the vertical direction of each image frame f constituting the moving image data M Change in the rotation angle of the moving subject O per unit time (hereinafter referred to as relative angle information).
Thereafter, when the initial composite position of the subject cutout image P3 in the first image frame f1 of the composite moving image is determined based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user (step S36), the CPU 13 The image synthesizing unit 8g is caused to perform a first image synthesizing process for synthesizing the background image P4 and the subject cutout image P3 to generate a synthesized moving image (step S37).
Note that the initial composite position of the subject clipped image P3 is set so that the image of the subject region S of the subject clipped image P3 does not protrude from the background image P4 based on the “motion information” detected by the motion detector 8f. The image composition unit 8g may determine automatically.

Here, the first image composition processing will be described in detail with reference to the flowchart of FIG.
As shown in FIG. 6, the image composition unit 8g reads the alpha map stored in association with the subject cutout image P3 and develops it in the image memory 5 (step S51).
Note that when the composition position of the subject cutout image P3 in the background image P4 is determined in step S36 in FIG. 5, the background image P4 and the alpha map are misaligned and the region is out of the alpha map range. , Α = 0, so that an area where no alpha value exists is not generated.

Next, the composition control unit 8h of the image composition unit 8g designates “1” as the frame number of the first image frame g1 related to the processing object among the plurality of image frames g,. S52).
Next, the image composition unit 8g designates any one pixel (for example, the pixel in the upper left corner) of the background image P4 (step S53), and processes the pixel based on the alpha value of the alpha map. Is branched (step S54). Specifically, the image composition unit 8g, for any one of the pixels of the background image P4, for a pixel with an alpha value of 1 (step S54; α = 1), the corresponding pixel of the subject cutout image P3 The pixel value is overwritten (step S55), and for the pixel with an alpha value of 0 <α <1 (step S54; 0 <α <1), the subject area S is cut out using the one's complement (1-α). After the image (background image × (1-α)) is generated, the blended value with the single background color is calculated when the subject cutout image P3 is generated using the one's complement (1-α) in the alpha map. Then, the value is subtracted from the subject cutout image P3, and is combined with an image obtained by cutting out the subject region S (background image × (1-α)) (step S56). Step S54; α = 0), Also to monkey through the background for the image P4 without.

Subsequently, the image composition unit 8g determines whether or not all the pixels of the background image P4 have been processed (step S57).
If it is determined that all the pixels have not been processed (step S57; NO), the image composition unit 8g designates the next pixel as the processing target and moves the processing target to the pixel (step S57). (S58), the process proceeds to step S54.
By repeating the above processing until it is determined that all pixels have been processed in step S57 (step S57; YES), the image composition unit 8g composites the subject cutout image P3 and the background image P4. A first image frame g1 (see FIG. 11A) that forms a composite moving image of the subject composite image P5 is generated.

If it is determined in step S57 that all pixels have been processed (step S57; YES), the composition control unit 8h determines whether or not processing has been performed for all image frames g constituting the composite video ( Step S59).
Here, if it is determined that all image frames g have not been processed (step S59; NO), the composition control unit 8h selects the next among the plurality of image frames g,. The frame number related to the image frame g is designated by incrementing by +1 (step S60). As a result, the second image frame g2 is designated as the target of the image composition process.

Next, the composition control unit 8h uses the “motion information” such as the relative position information, the relative range information, and the relative angle information detected by the motion detection unit 8f for the second image frame g2 constituting the composition video. The composite content information such as the center coordinates of the subject area S related to the synthesis position of the subject clipped image P3, the range occupied by the subject area S with respect to the first image frame g1, and the rotation angle of the subject area S with respect to the first image frame g1 are generated. (Step S61).
Subsequently, for the second image frame g2, the composition control unit 8h determines the composition position, size, shape, rotation angle, and the like of the subject clipped image P3 according to the composition content information of the subject clipped image P3 that is sequentially generated. Is set (step S62). At this time, the composition control unit 8h determines the subject area part (α = 1) and the part other than the subject area (α in the alpha map) according to the composition position, size, shape, rotation angle, and the like of the set subject clipped image P3. = 0) is corrected so that the subject cutout image P3 and the alpha map in each image frame g are adjusted so as not to be shifted.

Next, the image composition unit 8g shifts the processing to step S53, and sequentially performs steps S54 to S58 from, for example, the upper left corner pixel of the background image P4, so that the subject cutout image P3 and the background image P4 To generate a second image frame g2 constituting a synthesized moving image of the subject synthesized image P5. Accordingly, a second image frame g2 (see FIG. 11B) in which the composition position, size, shape, rotation angle, and the like of the subject region S are adjusted with respect to the first image frame g1 is generated.
If the generation of the second image frame g2 is completed and it is determined in step S57 that all pixels have been combined (step S57; YES), the process proceeds to step S58 to perform combining control. The unit 8h determines whether or not the synthesis process has been performed for all the image frames g constituting the synthesized moving image (step S59).
By repeating the above synthesis process for all image frames g (step S59; YES), the image synthesis unit 8g repeats all the image frames g1 constituting the synthesized video of the subject synthesized image P5. To gn (see FIGS. 11A to 11D) are generated.
Thereby, the first image composition process is completed.

  As shown in FIG. 5, after that, the CPU 13 causes the display control unit 10 to display each of the synthesized moving image image data of the subject synthesized image P5 including the plurality of image frames g generated by the image synthesizing unit 8g. By switching the image frame g at a predetermined display frame rate and reproducing and displaying it on the display screen of the display unit 11, a composite moving image of the subject composite image P5 in which the automobile (subject region S) operates in a predetermined manner with the background image P4 as the background Playback is performed (step S38).

  On the other hand, if it is determined in step S33 that a synthetic moving image generation instruction has not been input (step S33; NO), based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user, a user desired After specifying and determining the synthesis position of the subject clipped image P3 (step S39), the CPU 13 synthesizes the background image P4 and the subject clipped image P3 to generate a subject composite image that is a normal still image. The image composition processing is performed by the image composition unit 8g (step S40).

Here, the second image composition processing will be described in detail with reference to FIG.
As shown in FIG. 7, the image composition unit 8g reads the alpha map stored in association with the subject cutout image P3 and develops it in the image memory 5 (step S71).
Note that when the synthesis position of the subject cutout image P3 in the background image P4 is determined in step S39 in FIG. 5, the background image P4 and the alpha map are misaligned and are out of the alpha map range. , Α = 0, so that an area where no alpha value exists is not generated.

  Next, the image composition unit 8g designates any one pixel (for example, the pixel at the upper left corner) of the background image P4 (step S72), and processes the pixel based on the alpha value of the alpha map. Is branched (step S73). Specifically, the image composition unit 8g, for any one of the pixels of the background image P4, for a pixel with an alpha value of 1 (step S73; α = 1), the corresponding pixel of the subject cutout image P3 The pixel value is overwritten (step S74), and for the pixel with an alpha value of 0 <α <1, the subject area SA is cut out using 1's complement (1-α) (step S73; 0 <α <1). After the image (background image × (1-α)) is generated, the blended value with the single background color is calculated when the subject cutout image P3 is generated using the one's complement (1-α) in the alpha map. Then, the value is subtracted from the subject cutout image P3, and is combined with an image obtained by cutting out the subject area SA (background image × (1-α)) (step S75, for pixels with an alpha value of 0 (step S73; α = 0) Anything to monkey through the background for the image P4 without.

Subsequently, the image composition unit 8g determines whether or not all the pixels of the background image P4 have been processed (step S76).
If it is determined that all the pixels have not been processed (step S76; NO), the image composition unit 8g designates the next pixel as a processing target and moves the processing target to the pixel (step S76). (S77) The process proceeds to step S73.
By repeating the above process until it is determined in step S76 that all pixels have been processed (step S76; YES), the image composition unit 8g composites the subject cutout image P3 and the background image P4. Image data of a subject composite image that is a normal still image is generated.
Thus, the second image composition process is finished.

As shown in FIG. 5, thereafter, the CPU 13 causes the display control unit 10 to combine the subject on the background image P4 based on the image data of the subject composite image generated by the image composition unit 8g ( (Not shown) is displayed on the display screen of the display unit 11 (step S38).
Thus, the composite image generation process is finished.

As described above, according to the imaging apparatus 100 of the present embodiment, the “motion information” of the moving subject O is detected by the motion detection unit 8f, and the subject region S of the subject cutout image P3 is reflected by reflecting the “motion information”. The subject cutout image P3 and the background image P4 are combined so that the subject composition image P5 is synthesized by the image composition unit 8g.
Specifically, the motion detection unit 8f detects, as “motion information”, a change of the moving subject O over the image frame f in the moving image data M acquired by the motion image acquisition unit 8e as “motion information”. Then, based on the “motion information” detected by the motion detection unit 8f, the composition control unit 8h sequentially generates composition content information of the subject clipped image P3 that constitutes each of the image frames f of the composite movie, The composition position, size, shape, rotation angle, and the like of the subject clipped image P3 in the background image P4 are set according to the sequentially generated composition content information, and the image is synthesized with the background image P4 for each image frame g. As a result, a composite moving image of the subject composite image P5 composed of a plurality of image frames g,... Is generated.
Accordingly, even in the case of image synthesis between still images, based on the “motion information” of the moving subject O detected from the motion image data, an operation substantially equivalent to the motion of the moving subject O is performed on the synthesized subject composite image P5. It is possible to generate a subject composition image P5 that can be given and is highly interesting.

  Further, since the position, size, shape, and rotation angle of the moving subject O are detected as the “movement information” of the moving subject O from the moving image data M, the subject of the subject cutout image P3 can be obtained by using the “movement information”. Not only the composition position and orientation of the region S, but also the size and shape can be dynamically changed, and even in the case of image composition between still images, a motion rich in expressiveness can be imparted.

Further, by capturing a moving subject O, moving image data M (moving image data) composed of a plurality of continuous still images is generated, and the moving image acquisition unit 8e uses the generated moving image data M as a recording medium 9. Can be obtained from.
That is, only by recording the motion image data in which the motion of the predetermined moving subject O is recorded on the recording medium 9, the motion image data is acquired from the recording medium 9 and the “motion information” is obtained based on the motion image data. The subject cutout image P3 and the background image P4 are combined to generate a subject composite image P5 so that the subject region S of the subject cutout image P3 is operated in a predetermined manner by the imaging apparatus 100 alone. Can do.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, the moving image data M recorded in the Motion-JPEG format is exemplified as the moving image data. However, the moving image data is not limited to this, and may be, for example, a continuous image including a plurality of continuous still images. . Even in this case, as in the above-described embodiment, the “movement information” per unit time of the subject O moving between successive still images is detected, and the subject cut-out image is based on the “movement information”. A predetermined operation is performed on the subject area S of P3.
Alternatively, the subject clipping process may be executed in advance, and the “motion information” may be detected using the moving image data M of the subject clipping image P3 formed by the plurality of image frames f,. . In this case, since the subject area S, which is the moving subject O, is extracted in advance, it is possible to simplify the specific processing of the moving subject O and speed up the subject clipping process.

  In the above embodiment, the Motion-JPEG format is exemplified as the recording method of the moving image data M, but the MPEG format may be used. In such a case, when the movement of the moving subject O is detected, the moving image data M in the MPEG format is decoded in advance to generate a plurality of image frames f,. Motion can be detected.

  Furthermore, in the above embodiment, the relative position information, the relative range information, and the relative angle information as “motion information” are calculated for each of the second and subsequent image frames f on the basis of the first image frame f1. However, the present invention is not limited to this, and the calculation may be performed between successive image frames f. That is, relative position information is calculated as the amount of movement of the coordinates of each corner of the rectangular frame W between successive image frames f, and the relative range information is calculated as a scaling factor of the rectangular frame W in the X-axis direction and the Y-axis direction. The relative angle information may be calculated as the rotation angle of the rectangular frame W.

  In addition, the configuration of the imaging apparatus 100 is merely an example illustrated in the above embodiment, and is not limited thereto. In other words, the image pickup apparatus is exemplified as the image composition apparatus, but is not limited thereto. For example, the subject cutout image P3, the background image P4, and the motion image data are generated by an imaging device different from the imaging device 100, and only the image data transferred from the imaging device is recorded to generate a composite image. An image composition device that executes only processing may be used.

In addition, in the above embodiment, the functions as the still image acquisition unit, the detection unit, and the synthesis control unit are controlled under the control of the CPU 13 by the cutout image generation unit 8d, the motion detection unit 8f, Although the configuration realized by driving the image composition unit 8g is not limited to this, the configuration may be realized by executing a predetermined program or the like by the CPU 13.
That is, a program including a still image acquisition processing routine, a detection processing routine, and a composition control processing routine is stored in a program memory (not shown) that stores the program. Then, the first and second still images may be acquired by the CPU 13 by a still image acquisition processing routine. Further, the detection processing routine may cause the CPU 13 to detect changes in a plurality of images of a subject included in common in the plurality of images. Further, the synthesis control processing routine may cause the CPU 13 to generate a synthesized moving image in which the first still image is synthesized by reflecting changes over a plurality of detected images with the second still image as a background.

DESCRIPTION OF SYMBOLS 100 Imaging device 1 Lens part 2 Electronic imaging part 3 Imaging control part 8 Image processing part 8d Cutout image generation part 8e Motion image acquisition part 8f Motion detection part 8h Image composition part 8g Composition control part 9 Recording medium 11 Display part 13 CPU

Claims (8)

Still image acquisition means for acquiring first and second still images;
Detection means for detecting changes over the plurality of images of a subject commonly included in the plurality of images;
Combining means for generating a combined moving image in which the first still image is combined with the second still image as a background and reflecting the change detected by the detecting means;
An image composition device comprising:   The change of the subject over the plurality of images is a change per unit time of the position of the subject in the angle of view of the plurality of images, a change per unit time of a range occupied by the image of the subject in the plurality of images, The image synthesizing apparatus according to claim 1, wherein the image synthesizing apparatus is at least one of a change per unit time in a rotation angle of the subject with respect to at least one of a horizontal direction and a vertical direction of the plurality of images. Storage means for storing changes detected by the detection means as a plurality of images;
The synthesis means includes
Reading means for reading a plurality of images from the storage means;
Generating means for generating a plurality of composite still images obtained by combining the first still images at the positions of subjects in each of a plurality of images read by the reading means with the second still image as a background image;
The image synthesizing apparatus according to claim 1 or 2, characterized by comprising: The generating means includes
Based on the change detected by the detecting means, combined content generating means for sequentially generating combined content information;
A setting unit that sets at least one of a synthesis position, a size and a shape, and a rotation angle of the first still image according to the synthesis content information sequentially generated by the synthesis content generation unit;
The image synthesizing apparatus according to claim 3, further comprising: An image acquisition means for acquiring a moving image or a continuous shot image including a plurality of continuous still images;
The detecting means detects the change over the plurality of images of the subject commonly included in the plurality of images, using the moving image and the continuous shot image acquired by the image acquiring means as the plurality of images. The image composition device according to any one of claims 1 to 4, wherein   The image synthesizing apparatus according to claim 5, wherein the image acquisition unit includes an imaging unit.   The image synthesizing apparatus according to claim 1, wherein the first still image is an extracted image obtained by extracting a specific region from a predetermined still image. Computer
Still image acquisition means for acquiring first and second still images;
Detecting means for detecting a change over the plurality of images of a subject commonly included in the plurality of images;
Combining means for generating a combined moving image in which the first still image is combined by reflecting the change detected by the detecting means with the second still image as a background;
A program characterized by functioning as

Images