JP5402148B2 - Image composition apparatus, image composition method, and program - Google Patents

Description

The present invention relates to an image composition apparatus, an image composition method, and a program for compositing a plurality of images.

  Conventionally, a technique for generating a composite image by combining a subject image with a background image or a frame image is known (see, for example, Patent Document 1).

JP 2004-159158 A

  However, by simply synthesizing a moving image as a subject image with a background still image, only ordinary moving images can be generated, resulting in an uninteresting image.

Therefore, an object of the present invention is to provide an image composition device, an image composition method, and a program that can generate a highly interesting composite image.

In order to solve the above-mentioned problem, the invention described in claim 1
Detecting means for detecting the motion of a moving object comprising a plurality of image frames; direction determining means for determining whether or not the direction of motion of the moving object detected by the detecting means is substantially equal between adjacent image frames; and the direction If the direction of movement of the animal body is determined to substantially equal the determination unit, and a synthesis control means for additively combining with the passage of reproduction time an image of the moving object, the synthesis control means, said direction If the determination means determines that the direction of motion of the moving object in any one image frame has changed with respect to the direction of motion of the moving object in the image frame prior to the one image frame, the additive synthesis has already been performed. It is an object of the present invention to provide an image composition apparatus for erasing an image of a moving object .

Invention according to claim 2,
A plurality of image frames constituting the moving image as the reproduction time elapses when the direction determining unit determines that the moving direction of the moving object is substantially equal; The image composition apparatus according to claim 1, wherein the image composition device is added and sequentially synthesized.

The invention according to claim 3,
The synthesis control means adds and synthesizes a background still image and the moving body image as the playback time elapses when the direction determining means determines that the moving direction of the moving body is substantially equal. The image synthesizing apparatus according to claim 1 or 2 is provided.

The invention according to claim 4
It further comprises a motion amount determination means for determining whether or not the amount of motion of the moving object detected by the detection means is greater than or equal to a predetermined value. The image composition apparatus according to any one of claims 1 to 3 , wherein the image of the moving object is added and synthesized as the reproduction time elapses when it is determined that the image is greater than or equal to a predetermined value. It is to provide.

The invention described in claim 5
When the direction determining unit determines that the moving direction of the moving object is not substantially equal, the synthesis control unit applies each of the plurality of image frames constituting the moving image to the background still image. The image synthesizing apparatus according to claim 3 or 4 , characterized by being synthesized.

The invention described in claim 6
With reference to the motion of the moving object detected by the detecting means, the detecting means for detecting the movement of the moving object in the moving image, the combining means for combining the still image for the background and the moving image to generate a combined moving image Combining control means for combining the still image and the moving image with the combining means so that the moving object is added and displayed at different positions of the still image as the playback time elapses, and the moving object detected by the detecting means Motion determining means for determining whether or not the directions of the movements of the moving images are substantially equal, wherein the combining control means is configured to combine the moving object in the plurality of image frames constituting the moving image with the still image. When the moving object is reproduced when the moving determination unit determines that the moving directions of the moving object are substantially equal to each other. As the time elapses, the synthesizing unit synthesizes the images so that they are added and displayed at different positions. It is determined whether or not the directions are substantially equal, and the synthesis control unit further causes the motion determination unit to move the moving object in any one image frame in an image frame before the one image frame. When it is determined that the moving object has changed in the direction of movement, the moving object already added to the still image is erased.

Invention according to claim 7,
An image composition method in an image composition device, wherein a detection step for detecting a motion of a moving object composed of a plurality of image frames and a motion direction detected by the detection step are substantially equal between adjacent image frames. A direction determining step for determining whether or not the direction of movement of the moving object is determined to be substantially equal by the direction determining step, and a synthesis control step for adding and combining the images of the moving object with the lapse of the reproduction time; In the composition control step, the direction of the motion of the moving object in any one image frame is changed with respect to the direction of the motion of the moving object in the image frame before the one image frame by the direction determination step. when it is determined that, to provide an image synthesizing method characterized by erasing the images already added synthesized moving object That.

The invention according to claim 8 provides:
A direction for determining whether the computer of the image composition apparatus detects a motion of a moving object composed of a plurality of image frames, and whether the direction of motion of the moving object detected by the detection means is substantially equal between adjacent image frames. A combination control unit configured to add and synthesize an image of the moving object as a reproduction time elapses when the moving direction of the moving object is determined to be substantially equal by the determining unit and the direction determining unit; And means for determining that the direction of movement of the moving object in any one image frame has changed with respect to the direction of movement of the moving object in an image frame prior to the one image frame by the direction determination means. Further, the present invention provides a program characterized in that an image of a moving object that has already been added and synthesized is deleted .

ADVANTAGE OF THE INVENTION According to this invention, a moving body can be synthesize | combined in a novel aspect and a highly interesting synthetic | combination image can be produced | generated.

It is a block diagram which shows schematic structure of the imaging device of one Embodiment to which this invention is applied. 3 is a flowchart illustrating an example of an operation related to subject clipping processing by the imaging apparatus of FIG. 1. FIG. 3 is a diagram schematically illustrating an example of a subject existing image according to the subject clipping process of FIG. 2. FIG. 3 is a diagram schematically illustrating an example of a subject cutout image according to the subject cutout process of FIG. 2. 3 is a flowchart illustrating an example of an operation related to background image generation processing by the imaging apparatus of FIG. 1. 3 is a flowchart illustrating an example of an operation related to a composite image generation process by the imaging apparatus of FIG. 1. It is a flowchart which shows an example of the operation | movement which concerns on the synthetic | combination determination process in the synthetic | combination image generation process of FIG. It is a flowchart which shows an example of the operation | movement which concerns on the addition synthetic | combination process in the synthetic | combination image generation process of FIG. It is a figure which shows typically an example of the to-be-photographed object synthetic | combination image image-synthesized by the addition synthetic | combination process of FIG. It is a flowchart which shows an example of the operation | movement which concerns on the normal synthetic | combination process in the synthetic | combination image generation process of FIG. It is a figure which shows typically an example of the to-be-photographed object synthetic | combination image image-combined by the normal synthetic | combination process of FIG.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus 100 according to an embodiment to which the present invention is applied.
The imaging apparatus 100 according to the present embodiment detects the movement of the subject in the subject cutout moving image M2 related to the composite image generation process, and uses the detected movement of the subject as a reference to the subject as the playback time elapses. The background image P1 and the subject cutout moving image M2 are combined so as to be added and displayed at different positions of the use image P1.
Specifically, as illustrated in FIG. 1, the imaging apparatus 100 includes a lens unit 1, an electronic imaging unit 2, an imaging control unit 3, an image data generation unit 4, an image memory 5, and a motion detection unit 6. An image processing unit 7, a recording medium 8, a display control unit 9, a display unit 10, an operation input unit 11, and a CPU 12.
In addition, the imaging control unit 3, the motion detection unit 6, the image processing unit 7, and the CPU 12 are designed as, for example, a custom LSI 1A.

The lens unit 1 includes a plurality of lenses and includes a zoom lens, a focus lens, and the like.
Although not shown, the lens unit 1 includes a zoom drive unit that moves the zoom lens in the optical axis direction and a focus drive unit that moves the focus lens in the optical axis direction when imaging a subject. May be.

  The electronic imaging unit 2 is composed of an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-oxide Semiconductor), for example, and converts an optical image that has passed through various lenses of the lens unit 1 into a two-dimensional image signal. To do.

Although not shown, the imaging control unit 3 includes a timing generator, a driver, and the like. Then, the imaging control unit 3 scans and drives the electronic imaging unit 2 with a timing generator and a driver, converts the optical image into a two-dimensional image signal with the electronic imaging unit 2 every predetermined period, and the electronic imaging unit 2 Image frames are read out from the imaging area for each screen and output to the image data generation unit 4.
In addition, the imaging control unit 3 performs adjustment control of conditions when imaging a subject such as AF (automatic focusing process), AE (automatic exposure process), and AWB (automatic white balance).

The imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 configured as described above serve as the subject presence moving image M <b> 1 related to the composite image generation process (see FIGS. 3A to 3E) as an imaging unit. ) And the background image P1 (see FIG. 9A, etc.).
In addition, the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 capture the subject clipping moving image in a state in which the imaging conditions for capturing the subject existing moving image M1 are fixed after capturing the subject existing moving image M1. A subject non-existing image (not shown) for generating M2 (see FIGS. 4A to 4E) is captured.

The image data generation unit 4 appropriately adjusts the gain for each RGB color component with respect to the analog value signal of the image frame transferred from the electronic imaging unit 2, and then performs sample holding by a sample hold circuit (not shown). The digital signal is converted into digital data by a / D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a digital luminance signal Y and color difference signal Cb , Cr (YUV data).
The luminance signal Y and the color difference signals Cb and Cr output from the color process circuit are DMA-transferred to an image memory 5 used as a buffer memory via a DMA controller (not shown).

  The image memory 5 is composed of, for example, a DRAM or the like, and temporarily stores data processed by the motion detection unit 6, the image processing unit 7, the CPU 12, and the like.

The motion detection unit 6 includes a feature amount calculation unit 6a and a block matching unit 6b.
The feature amount calculation unit 6a performs feature extraction processing for extracting feature points from the subject non-existing image with reference to the subject non-existing image. Specifically, the feature amount calculation unit 6a selects a predetermined number (or a predetermined number or more) of highly featured block regions (feature points) based on the YUV data of the subject nonexistent image, and The contents are extracted as a template (for example, a square of 16 × 16 pixels).
Here, the feature extraction process is a process of selecting a feature having a high characteristic convenient for tracking from a large number of candidate blocks.

  Further, the feature amount calculation unit 6a performs a predetermined number (or alternatively) from each image frame g constituting the moving image data of the subject cutout moving image M2 recorded in the Motion-JPEG format, for example, in the composition determination process (described later). A block area (feature point) having a high feature (a predetermined number or more) is selected, and the contents of the block are extracted as a template (for example, a square of 16 × 16 pixels).

  The block matching unit 6b performs block matching processing for aligning the subject non-existing image and the subject existing moving image M1. Specifically, the block matching unit 6b determines where the template extracted in the feature extraction process corresponds in the subject existing moving image M1, that is, the template pixel value is optimal in the subject existing moving image M1. Search for a matching position (corresponding region). Then, an optimum offset between the subject non-existing image and the subject existing moving image M1 having the best evaluation value (for example, sum of squared differences (SSD), sum of absolute differences (SAD), etc.) of the pixel value is determined. Calculated as a template motion vector.

Also, the block matching unit 6b has the template extracted by the feature extraction process from one image frame g among the plurality of image frames g,... Constituting the moving image data of the subject cutout moving image M2 one before. Where to correspond in the image frame g, that is, a position (corresponding region) where the pixel value of the template optimally matches in the previous image frame g is searched. Then, the evaluation value (for example, sum of squares of differences (SSD), sum of absolute differences (SAD), etc.) of the difference between pixel values is the optimum between the one image frame g and the previous image frame g. A correct offset is calculated as a motion vector of the template. Then, for each of the plurality of calculated motion vectors, one of the motion vectors is compared with another motion vector as an evaluation target, and it is determined whether or not to vote for the motion vector of the evaluation target. The motion vector of the entire image frame g (see FIGS. 4A to 4E) is calculated by majority vote.
In FIGS. 4A to 4E, the motion vector of the subject in the image frame g is represented by a white arrow.
Here, the feature amount calculation unit 6a and the block matching unit 6b of the motion detection unit 6 constitute detection means for detecting the motion of the moving object in the moving image.

The image processing unit 7 includes an alignment unit 7a that performs alignment between the subject presence moving image M1 and the subject non-existence image.
The alignment unit 7a calculates a coordinate conversion formula (projection conversion matrix) of each pixel of the subject presence moving image M1 with respect to the subject nonexistence image based on the feature points extracted from the subject nonexistence image, and follows the coordinate conversion formula. The subject presence moving image M1 is coordinate-converted and aligned with the subject non-existence image.
In addition, the alignment unit 7a aligns each image frame f of the subject existing moving image M1 including the plurality of image frames f,.

Further, the image processing unit 7 generates difference information of each corresponding pixel between the subject presence moving image M1 and the subject non-existence image aligned by the alignment unit 7a, and the subject exists based on the difference information. A subject area extraction unit 7b that extracts a subject area S including a subject from the moving image M1 is provided.
In addition, the subject area extraction unit 7b extracts a subject area S including a subject from each image frame f of the subject existing moving image M1 including a plurality of image frames f.

Further, the image processing unit 7 specifies the position of the subject region S extracted in the subject presence moving image M1, and generates position information indicating the position of the subject region S in the subject presence moving image M1. 7c.
Here, the position information includes, for example, an alpha map. The alpha map is a weight for alpha blending the image of the subject area S with respect to a predetermined background for each pixel of the subject existing moving image M1. This is expressed as an alpha value (0 ≦ α ≦ 1).
In addition, when the subject presence moving image M1 is a moving image including a plurality of image frames f,..., The position information generation unit 7c generates position information indicating the position of the subject region S in each image frame f.

Further, the image processing unit 7 does not transmit a pixel having an alpha value of 1 among the pixels of the subject existing moving image M1 to a predetermined single color image (not shown) based on the generated alpha map. In addition, an image of the subject cutout moving image M2 (see FIGS. 4A to 4E) is synthesized by synthesizing the subject image with a predetermined single color image so as to transmit pixels having an alpha value of 0. A cut-out image generation unit 7d for generating data is provided.
Thereby, the cut-out image generation unit 7d acquires the subject cut-out moving image M2 obtained by cutting out the area including the subject from the subject existing moving image M1 in which the background and the subject exist.
Further, the cut-out image generation unit 7d acquires an image frame f obtained by cutting out a region including the subject from each image frame f of the subject existing moving image M1 including a plurality of image frames f,. The moving image data of the subject cutout moving image M2 is generated.

In addition, the image processing unit 7 determines whether or not the direction of movement of the subject in each image frame constituting the moving image data of the subject cutout moving image M2 detected by the motion detection unit 6 is substantially equal. It has.
Specifically, the motion determination unit 7e acquires motion vectors of a plurality of image frames g,... From the motion detection unit 6, and depending on whether the magnitude of the motion vector is equal to or greater than a predetermined threshold value. It is determined whether or not the amount of movement of the subject is equal to or greater than a predetermined threshold value. Then, when it is determined that the amount of movement of the subject is equal to or greater than the predetermined threshold, the motion determination unit 7e determines whether the motion of the subject is in accordance with whether or not the motion vectors between adjacent image frames g are substantially the same. It is determined whether the directions are substantially equal. For example, when a person riding on a sled slides on the snow (the ground surface) in a predetermined direction (see FIGS. 3A to 3E), the motion vector of each image frame g has a predetermined direction (for example, , Obliquely downward to the left) (see FIGS. 4A to 4E). The direction of the motion vector need not be strictly determined. For example, the motion vector in FIGS. 4B and 4C, the motion vector in FIGS. 4D and 4E, However, if the difference is a slight angle, it may be determined that the direction is substantially equal to a predetermined one direction (for example, diagonally downward to the left).

The image processing unit 7 includes an image composition unit 7f that synthesizes the subject cutout moving image M2 and the background image P1. Specifically, the image composition unit 7f acquires a user-desired background image P1 from the recording medium 8, and transmits pixels with an alpha value of 0 among the pixels of the background image P1 to obtain an alpha value. Is overwritten with the pixel value of the corresponding pixel of the subject cutout moving image M2, and among the pixels of the background image P1, the pixel with an alpha value of 0 <α <1 is a 1's complement (1- An image (background image × (1−α)) obtained by clipping the subject area S using α) is generated, and then a subject cutout moving image M2 is generated using 1's complement (1−α) in the alpha map. At this time, a value blended with a single background color is calculated, and the value is subtracted from the subject cutout moving image M2, and is combined with an image obtained by cutting out the subject region S (background image × (1-α)).
In addition, the image composition unit 7f synthesizes a predetermined image as the background image P1 with each image frame g of the subject cutout moving image M2 composed of a plurality of image frames g,.

Further, the image composition unit 7f includes a composition control unit 7g that controls composition of each image frame g constituting the subject cutout moving image M2 with the background image P1. That is, the image composition unit 7f adds the subject to a different position as the playback time elapses with reference to the motion vector of each image frame g detected by the motion detection unit 6 under the control of the composition control unit 7g. The subject cutout moving image M2 and the background image P1 are combined so as to be displayed.
Specifically, the composition control unit 7g has the motion amount of the subject in the plurality of image frames g,... Constituting the moving image data of the subject cutout moving image M2 by the motion determination unit 7e greater than or equal to a predetermined threshold value, and When it is determined that the directions of movement are substantially equal, the plurality of subjects are combined with the still image so that the subjects in the plurality of image frames g,... Are added and displayed at different positions as the playback time elapses. Addition processing is performed to generate a composite moving image including a plurality of image frames g,.
That is, the composition control unit 7g uses the alpha map for the first image frame g1 and the still image designated as the background still image among the plurality of image frames g,... Constituting the subject cutout moving image M2. After generating the image composition unit 7f, an image h1 (see FIG. 9A) in which the subject of the first image frame g1 is synthesized with the still image is designated as a background still image. On the other hand, the image synthesizing unit 7f is generated using the alpha map of the next image frame g2 so that the subject of the next (second) image frame g2 constituting the subject cutout moving image M2 is added and displayed. Thus, an image (not shown) is generated by adding and synthesizing the subject of the next image frame g2 to the image h1 (see FIG. 9A) in which the subject of the first image frame g1 is synthesized with the still image. . By repeating the above processing for all the image frames g constituting the subject cutout moving image M2, for example, as shown in FIGS. 9A to 9D, a person slipping from the back side to the front side can be obtained. A composite moving image including a plurality of image frames h,... That is gradually superimposed (added) as the playback time elapses is generated.
Note that the subject addition display is not limited to the superimposed display, and may be any display in which the number of subjects increases as the playback time elapses. For example, a plurality of subjects are spaced at a predetermined interval. Display may be arranged.

In addition, the composition control unit 7g has the motion amount of the subject in the plurality of image frames g constituting the moving image data of the subject clipped moving image M2 by the motion determination unit 7e, or the direction of the motion is substantially equal. If it is determined that there is no image, the image combining unit 7f performs normal combining processing for combining each of the plurality of image frames g with a still image designated as a background still image.
That is, for example, as shown in FIGS. 11 (a) to 11 (d), when a person moves his / her arm up and down without moving from a predetermined position, the motion vector (not shown) is diagonally upward (FIG. 11 ( b)) to the lower right (see FIG. 11 (c)) and further to the lower left (see FIG. 11 (d)). In such a case, if the addition synthesis process is performed, the arm becomes a state of a thousand-handed kannon, resulting in an unsightly image. Therefore, each image frame g and the background image P1 are synthesized to form a synthesized moving image. The image frame i to be generated is generated individually.

The recording medium 8 is composed of, for example, a nonvolatile memory (flash memory) or the like, and stores image data of the background image P1 and the subject cutout moving image M2 encoded by the encoding unit (not shown) of the image processing unit 7. Remember.
The image data of the subject cutout moving image M2 is stored in one file in association with the alpha map generated by the position information generation unit 7c of the image processing unit 7.
Further, when the subject cutout moving image M2 is a moving image, moving image data including a plurality of continuous image frames g,... (For example, n image frames) captured at a predetermined imaging frame rate, for example. It is recorded in the Motion-JPEG format. 4 (a) to 4 (e) schematically represent n consecutive image frames g1 to gn constituting the moving image data of the subject cutout moving image M2, and these image frames g1. ˜gn is expressed as FIG. 4A to FIG. 4E along the time axis.

The display control unit 9 performs control for reading display image data temporarily stored in the image memory 5 and displaying the read image data on the display unit 10.
Specifically, the display control unit 9 includes a VRAM, a VRAM controller, a digital video encoder, and the like. The digital video encoder periodically reads the luminance signal Y and the color difference signals Cb and Cr read from the image memory 5 and stored in the VRAM (not shown) under the control of the CPU 12 from the VRAM via the VRAM controller. Are read out, a video signal is generated based on these data, and is output to the display unit 10.

  The display unit 10 is, for example, a liquid crystal display device, and displays an image captured by the electronic imaging unit 2 on the display screen based on a video signal from the display control unit 9. Specifically, the display unit 10 displays a live view image based on a plurality of image frames generated by imaging an object by the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 in the imaging mode. Or a REC view image captured as the actual captured image.

  The operation input unit 11 is for performing a predetermined operation of the imaging apparatus 100. Specifically, the operation input unit 11 includes a shutter button 11a related to a subject photographing instruction, a selection instruction 11b related to a selection instruction of an imaging mode, a function, and the like, and an instruction to set an initial composite position of the subject cutout moving image M2. A zoom button (not shown) or the like related to an amount adjustment instruction is provided, and a predetermined operation signal is output to the CPU 12 in accordance with the operation of these buttons.

  The CPU 12 controls each unit of the imaging device 100. Specifically, the CPU 12 performs various control operations in accordance with various processing programs (not shown) for the imaging apparatus 100.

Next, the subject clipping process performed by the imaging apparatus 100 will be described with reference to FIGS.
FIG. 2 is a flowchart illustrating an example of an operation related to the subject clipping process. FIGS. 3A to 3E are diagrams schematically showing an example of the subject presence moving image M1 related to the subject clipping process, and FIGS. 4A to 4E are views of the subject. It is a figure which shows typically an example of the to-be-photographed object moving image M2 which concerns on a clipping process.

The subject clipping process is executed when the subject clipping mode is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 11b of the operation input unit 11 by the user. It is.
In the subject clipping process described below, the subject is captured as a subject existing moving image M1 at a predetermined imaging frame rate to generate a moving image (subject present moving image M1) including a plurality of image frames. Assume that a moving image (subject cutout moving image M2) in which the subject region S is extracted from each image frame f of the moving image is generated.

  As shown in FIG. 2, first, the CPU 12 causes the display control unit 9 to display a live view image based on a plurality of image frames generated by imaging the subject by the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. Is displayed on the display screen of the display unit 10 and superimposed on the live view image, an imaging instruction message of the subject existing moving image M1 is displayed on the display screen of the display unit 10 (step S1).

Next, the CPU 12 determines whether an imaging instruction is input based on a predetermined operation of the shutter button 11a of the operation input unit 11 by the user (step S2). If it is determined here that an imaging instruction has been input (step S2; YES), the CPU 12 instructs the imaging control unit 3 to focus the focus lens, exposure conditions (shutter speed, aperture, amplification factor, etc.) and white. By adjusting conditions such as balance, an optical image of the subject presence moving image M1 (see FIGS. 3A to 3E) is captured by the electronic imaging unit 2 at a predetermined imaging frame rate under predetermined conditions. Then, the image data generation unit 4 generates YUV data of each image frame f of the subject existing moving image M1 transferred from the electronic imaging unit 2 (step S4). The YUV data of each image frame f of the subject existing moving image M1 is temporarily stored in the image memory 5.
Further, the CPU 12 controls the imaging control unit 3 to maintain a state in which conditions such as a focus position, an exposure condition, and a white balance at the time of imaging the subject presence moving image M1 are fixed.

Then, the CPU 12 causes the display control unit 9 to display a live view image on the display screen of the display unit 10 based on a plurality of image frames generated by imaging the subject by the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. The display unit 10 displays an image in a semi-transparent display mode of any one of the image frames f of the subject existing moving image M1 and an imaging instruction message for the subject non-existing image superimposed on the live view image. It is displayed on the screen (step S4).
Thereafter, the CPU 12 determines whether or not an imaging instruction has been input based on a predetermined operation of the shutter button 11a of the operation input unit 11 by the user (step S5). Then, the user waits for the subject to move or moves the subject outside the angle of view, and then the subject non-existing image overlaps with the translucent image of the image frame f of the subject existing moving image M1. When it is determined that the camera position has been adjusted and the shutter button 11a of the operation input unit 11 has been operated in a predetermined manner and an imaging instruction has been input (step S5; YES), the CPU 12 causes the imaging control unit 3 to An optical image of a presence image (not shown) is captured by the electronic imaging unit 2 under a fixed condition after the subject presence moving image M1 is captured, and the subject absent from the electronic imaging unit 2 is transferred to the image data generation unit 4 Based on the image frame of the image, YUV data of the non-subject image is generated (step S6). Note that the YUV data of the subject existing moving image M1 is temporarily stored in the image memory 5.

Next, the CPU 12 uses the YUV data of the subject non-existing image temporarily stored in the image memory 5 in the feature amount calculation unit 6a, the block matching unit 6b, and the image processing unit 7 as a reference. A projective transformation matrix for projectively transforming the YUV data of the image frame f is calculated using a predetermined image transformation model (for example, a similarity transformation model or a joint transformation model) (step S7).
Specifically, the feature amount calculation unit 6a selects a predetermined number (or a predetermined number or more) of highly featured block regions (feature points) based on the YUV data of the subject nonexistent image, and Extract the contents as a template. Then, the block matching unit 6b searches each image frame f of the subject existing moving image M1 for a position where the pixel value of the template extracted by the feature extraction process optimally matches, and determines the difference in pixel value. An optimal offset between the subject nonexistent image having the best evaluation value and the image frame f is calculated as a motion vector of the template. Then, the alignment unit 7a of the image processing unit 7 statistically calculates the entire motion vector based on the motion vectors of the plurality of templates calculated by the block matching unit 6b, and performs feature point correspondence related to the motion vector. The projection transformation matrix of each image frame f of the subject existing moving image M1 is calculated by using this.

  Next, the CPU 12 causes the alignment unit 7a to perform projective conversion of each image frame f of the corresponding subject existing moving image M1 based on the calculated projective conversion row example, thereby each image of the subject existing moving image M1. A process of aligning the YUV data of the frame f and the YUV data of the non-subject image is performed (step S8).

Then, the CPU 12 causes the subject region extraction unit 7b of the image processing unit 7 to perform processing for extracting the subject region S including the subject from each image frame f of the subject presence moving image M1 (step S9).
Specifically, the subject region extraction unit 7b applies a low-pass filter to each of the YUV data of each image frame f of the subject presence moving image M1 and the YUV data of the subject nonexistence image to remove high frequency components of each image. To do. Thereafter, the subject area extraction unit 7b calculates a difference for each corresponding pixel between each image frame f of the subject existing moving image M1 subjected to the low-pass filter and the subject non-existing image, and generates a difference map. . Subsequently, the subject region extraction unit 7b binarizes the dissimilarity map relating to each pixel with a predetermined threshold value, and then performs a contraction process to remove a region in which the dissimilarity is caused by fine noise or camera shake from the dissimilarity map. Do. Thereafter, the subject region extraction unit 7b performs a labeling process to remove a region below a predetermined value or a region other than the maximum region, and then specifies the largest island pattern as the subject region S and corrects the contraction. Expansion processing is performed.

  Next, the CPU 12 causes the position information generation unit 7c of the image processing unit 7 to generate an alpha map indicating the position of the extracted subject region S in the image frame f of the subject existing moving image M1 (step S10). .

Thereafter, the CPU 12 causes the cutout image generation unit 7d of the image processing unit 7 to combine the subject image in each image frame g of the subject cutout moving image M2 with a predetermined single color image M2 (FIG. 4 ( A process of generating image data of each image frame g in FIGS. 4A to 4E) is performed (step S11).
Specifically, the cut-out image generation unit 7d reads out each image frame f, single color image, and alpha map of the subject existing moving image M1 and develops them in the image memory 5, and then each image frame f of the subject existing moving image M1. For all of the pixels, alpha pixels with a zero value (α = 0) are transmitted, and pixels with an alpha value of 0 <α <1 (0 <α <1), a predetermined single color and blending are performed. For pixels with an alpha value of 1 (α = 1), nothing is done so as not to transmit a predetermined single color.

Thereafter, the CPU 12 associates the alpha map generated by the position information generation unit 7c of the image processing unit 7 with the image data of each image frame g of the subject cutout moving image M2 in a predetermined storage area of the recording medium 8. The file is stored (step S12).
Thereby, the subject clipping process is completed. As a result, for example, image data of a subject cutout moving image M2 in which a person who slides on a snow with a sled from a predetermined background is extracted as a subject is generated.

Next, background image generation processing by the imaging apparatus 100 will be described with reference to FIG.
FIG. 5 is a flowchart illustrating an example of an operation related to the background image generation processing.
The background image generation process is a normal still image imaging process, and a still image is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 11b of the operation input unit 11 by the user. This is a process executed when an imaging mode selection instruction is given.

  As shown in FIG. 5, first, the CPU 12 causes the display control unit 9 to based on a plurality of image frames generated by imaging the background image P1 by the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. The live view image is displayed on the display screen of the display unit 10 (step S21).

  Next, the CPU 12 determines whether or not an imaging instruction has been input based on a predetermined operation of the shutter button 11a of the operation input unit 11 by the user (step S22). If it is determined that an imaging instruction has been input (step S22; YES), the CPU 12 instructs the imaging control unit 3 to focus the focus lens, exposure conditions (shutter speed, aperture, amplification factor, etc.) and white. By adjusting conditions such as balance, an optical image of the background image P1 (see FIG. 9A, etc.) is picked up by the electronic image pickup unit 2 under predetermined conditions (step S23).

Subsequently, the CPU 12 causes the image data generation unit 4 to generate the YUV data of the image frame of the background image P1 transferred from the electronic imaging unit 2, and then the background image is stored in a predetermined storage area of the recording medium 8. The P1 YUV data is stored as an Exif format image file (step S24).
Thus, the background image P1 generation process is terminated. As a result, for example, still image data of the background image P1 in which the city area is recorded is generated.

Next, the composite image generation process will be described in detail with reference to FIGS.
FIG. 6 is a flowchart illustrating an example of an operation related to the composite image generation process. FIG. 7 is a flowchart illustrating an example of an operation related to the synthesis determination process. FIG. 8 is a flowchart showing an example of the operation related to the addition synthesis process, and FIGS. 9A to 9D show an example of the subject synthesis moving image M3 synthesized by the addition synthesis process. It is a figure shown typically. FIG. 10 is a flowchart showing an example of the operation related to the normal synthesis process, and FIGS. 11A to 11D show an example of the subject synthesized moving image M4 synthesized in the normal synthesis process. It is a figure shown typically.

The composite image generation process is executed when an image combination mode is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 11b of the operation input unit 11 by the user. It is processing.
As shown in FIG. 6, the CPU 12 first causes the motion detection unit 6 and the image processing unit 7 to execute a synthesis determination process for determining the content of the synthesis process (step S <b> 31).

Here, the synthesis determination process will be described in detail with reference to FIG.
As shown in FIG. 7, a desired subject cutout moving image M2 (FIGS. 4A to 4) among a plurality of images recorded on the recording medium 8 based on a predetermined operation of the operation input unit 11 by the user. When (see (e)) is selected and designated, the motion detection unit 6 reads out the image data of the designated subject cutout moving image M2 and develops it in the image memory 5 (step S41).

Next, the motion detection unit 6 designates “1” as the frame number of the first image frame g1 related to the processing object among the plurality of image frames g,... Constituting the subject cutout moving image M2 (step S42). Then, the image frame g1 having the designated frame number “1” and the alpha map stored in association with the image frame g1 are read out and developed in the image memory 5 (step S43).
Subsequently, the motion detection unit 6 identifies the subject region S using the alpha map from the read image frame g1 and extracts the subject (step S44).

Next, the motion detection unit 6 determines whether there is a subject extracted from the image frame g before the read image frame g1 (step S45). Here, since the image frame g read in step S43 is the first image frame g1, if it is determined that there is no subject of the previous image frame g (step S45; NO), the motion detection unit 6 Among the plurality of image frames g,... Constituting the cutout moving image M2, the frame number relating to the next image frame g2 to be processed is specified by incrementing by +1 (step S46). As a result, the second image frame g2 is designated as the target of the synthesis determination process.
Then, the motion detection unit 6 shifts the processing to step S43 and executes the subsequent processing. As a result, the motion detection unit 6 reads out the image frame g2 of the designated frame number “2” and the alpha map stored in association with the image frame g2 and develops them in the image memory 5 (step S43). Then, the subject region S is specified from the read image frame g2 using the alpha map, and the subject is extracted (step S44).

  When it is determined in step S45 that there is a subject of the previous image frame g (step S45; YES), the motion detection unit 6 determines the subject extracted from the second image frame g2 from the subject. A motion vector is calculated by searching where the template extracted in the feature extraction process corresponds to the subject extracted from the first image frame g1 (step S47).

Next, the motion determination unit 7e of the image processing unit 7 acquires a motion vector from the motion detection unit 6, and the amount of motion of the subject depending on whether the magnitude of the motion vector is equal to or greater than a predetermined threshold value. Is greater than or equal to a predetermined threshold value (step S48).
If it is determined that the amount of movement of the subject is equal to or greater than the predetermined threshold (step S48; YES), the motion determination unit 7e determines whether there is a motion vector of the subject in the previous image frame g. (Step S49).

In step S49, since the image frame g read in step S43 is the second image frame g2, if it is determined that there is no motion vector of the subject in the previous image frame g (step S49; NO), the motion The determination unit 7e shifts the process to step S46 and executes the subsequent processes. As a result, the third image frame g3 is designated as the target of the synthesis determination process.
After the same processing as that for the second image frame g2, the motion vector of the subject of the third image frame g3 is calculated (step S47), and then in step S49, the previous image frame g (for example, the second image frame g2) When it is determined that there is a motion vector of the subject in the second image frame g2) (step S49; YES), the motion determination unit 7e determines the subject between the second image frame g2 and the third image frame g3. It is determined whether or not the direction of movement of the subject is substantially equal depending on whether or not the motion vectors substantially match (step S50).

Here, if it is determined that the direction of movement of the subject is substantially equal (step S50; YES), the motion determination unit 7e performs a composite determination process among a plurality of image frames g,... Constituting the subject cutout moving image M2. It is determined whether or not there is a next image frame g to be subjected to (step S51). Here, if it is determined that there is a next image frame g, for example, the fourth image frame g4 (step S51; YES), the motion determination unit 7e shifts the processing to step S45, and the subsequent processing. Execute.
As a result, the fourth image frame g4 is designated as the target of the synthesis determination process.

On the other hand, if it is determined in step S48 that the amount of movement of the subject is smaller than the predetermined threshold (step S48; NO), the composition control unit 7g of the image processing unit 7 performs subject cropping video as image composition processing. A normal combining process for combining each of the plurality of image frames g,... Constituting the image M2 with the background image P1 is set (step S52).
If it is determined in step S50 that the direction of movement of the subject is not substantially equal (step S50; NO), the composition control unit 7g of the image processing unit 7 shifts the process to step S52 to perform image composition. As a process, a normal composition process is set (step S52).

If the above process is repeated until it is determined in step S51 that there is no next image frame g (step S51; NO), that is, the amount of movement of the subject for all the second and subsequent image frames g. Is equal to or greater than a predetermined threshold value and the direction of motion is substantially equal, the composition control unit 7g of the image processing unit 7 adds a plurality of subjects to the still image as a plurality of image frames h as an image composition process. ,... Are set (step S53).
As a result, the combination determination process is terminated.

As shown in FIG. 5, next, the CPU 12 branches the process according to the determination result of the synthesis determination process (step S32).
Specifically, when the addition combination process is set as a result of the combination determination process (step S32; addition combination), the CPU 12 causes the image composition unit 7f to execute the addition combination process (step S33). When the normal composition process is set as a result of the composition determination process (step S32; normal composition), the image composition unit 7f is caused to execute the normal composition process (step S34).

Here, the addition synthesis process will be described in detail with reference to FIGS.
As shown in FIG. 8, based on a predetermined operation of the operation input unit 12 by the user, a desired background image P <b> 1 that becomes the background of the composite image among a plurality of images recorded on the recording medium 8 (FIG. 9 ( When a) (see a) is selected and designated, the image processing unit 7 reads out the image data of the designated background image P1 and develops it in the image memory 5 (step S61).

  Next, the composition control unit 7g of the image composition unit 7f designates “1” as the frame number of the first image frame h1 related to the processing target among the plurality of image frames g,... Constituting the subject cutout moving image M2. Then (step S62), the image frame g1 of the designated frame number “1” and the alpha map stored in association with the image frame g1 are read and developed in the image memory 5 (step S63).

  Next, the image composition unit 7f designates any one pixel (for example, the pixel at the upper left corner) of the background image P1 (step S64), and processes the pixel based on the alpha value of the alpha map. Is branched (step S65). Specifically, the image composition unit 7f, for any one of the pixels of the background image P1, for the pixel having an alpha value of 1 (step S65; α = 1), the image frame g1 of the subject cutout moving image M2 Is overwritten with the pixel value of the corresponding pixel (step S66), and for the pixel with an alpha value of 0 <α <1 (step S65; 0 <α <1), the 1's complement (1-α) is used to determine the subject. When an image (background image × (1−α)) obtained by cutting out the region S is generated and then the subject cutout moving image M2 is generated using the one's complement (1−α) in the alpha map, a single background color is generated. And the blended value is subtracted from the image frame g1 of the subject cutout moving image M2 and synthesized with the image obtained by cutting out the subject region S (background image × (1-α)) (step S67). ), Alpha value is 0 For iodine (Step S65; α = 0), so as monkey transmitting a background image P1 without doing anything.

Subsequently, the image composition unit 7f determines whether or not all the pixels of the background image P1 have been processed (step S68).
If it is determined that all the pixels have not been processed (step S68; NO), the image composition unit 7f designates the next pixel as a processing target and moves the processing target to the pixel (step S68). (S69) The process proceeds to step S65.
By repeating the above processing until it is determined in step S68 that all pixels have been processed (step S68; YES), the image composition unit 7f allows the image frame g1 and the background image of the subject cutout moving image M2 to be processed. A first image frame h1 (see FIG. 9A) constituting a synthesized moving image of the subject synthesized moving image M3 synthesized with P1 is generated.

If it is determined in step S68 that all the pixels have been processed (step S68; YES), the composition control unit 7g determines whether all the image frames g constituting the subject cutout moving image M2 have been processed. Determine (step S70).
If it is determined that all the image frames g have not been processed (step S70; NO), the composition control unit 7g uses the first image frame h1 constituting the generated composition video as a background still image. (Step S71), among the plurality of image frames g,... Constituting the subject cutout moving image M2,..., The frame number relating to the next image frame g to be processed is incremented by 1 (step S72). ). As a result, the second image frame g2 is designated as the target of the image composition process.

Next, the image composition unit 7f shifts the process to step S63 to designate any one pixel (for example, the pixel at the upper left corner) of the image frame h1 designated as the background still image, By performing the subsequent processing (steps S64 to S69), the subject synthesized moving image obtained by synthesizing the first image frame h1 designated as the background still image and the second image frame g2 of the subject cutout moving image M2. A second image frame constituting the synthesized moving image of M3 is generated. As a result, a second image frame is generated in which the subject region S of the second image frame g2 of the subject cutout moving image M2 is added and synthesized to the first image frame h1.
If it is determined in step S68 that all the pixels have been processed by completing the generation of the second image frame (step S68; YES), the process proceeds to step S70, and the composition control unit 7g. Determines whether all image frames g constituting the subject cutout moving image M2 have been processed (step S70).
By repeating the above processing until it is determined that all image frames g have been processed (step S70; YES), the image composition unit 7f allows all image frames constituting the composite moving image of the subject composition moving image M3. h1 to hn (see FIGS. 9A to 9D) are generated.
Thereby, the addition synthesis process is terminated.

  As shown in FIG. 5, after that, the CPU 12 causes the display control unit 9 to generate the synthesized moving image M3 composed of a plurality of image frames h,... Each image frame h is switched at a predetermined display frame rate and reproduced and displayed on the display screen of the display unit 10, so that as the reproduction time elapses, the still image (background image P <b> 1) is changed to a sledge as a subject at a different position. A synthesized moving image of the subject synthesized moving image M3 on which the rider is added and displayed is reproduced (step S35).

Next, the normal synthesis process will be described in detail with reference to FIGS.
The normal synthesis process is substantially the same as the addition synthesis process except that the background image P1 specified first and all the image frames constituting the subject cutout moving image M2 are synthesized.
That is, as shown in FIG. 10, based on a predetermined operation of the operation input unit 12 by the user, a desired background image P1 (FIG. 10) that becomes the background of the composite image among a plurality of images recorded on the recording medium 8. 11 (a)) is selected and designated, the image processing unit 7 reads the image data of the designated background image P1 and develops it in the image memory 5 (step S81).

  Next, the composition control unit 7g of the image composition unit 7f designates “1” as the frame number of the first image frame g1 related to the processing object among the plurality of image frames g,... Constituting the subject cutout moving image M2. Then (step S82), the image frame g1 of the designated frame number “1” and the alpha map stored in association with the image frame g1 are read out and developed in the image memory 5 (step S83).

  Next, the image composition unit 7f designates any one pixel (for example, the pixel at the upper left corner) of the background image P1 (step S84), and processes the pixel based on the alpha value of the alpha map. Is branched (step S85). Specifically, the image composition unit 7f, for any one of the pixels of the background image P1, for the pixel having an alpha value of 1 (step S85; α = 1), the image frame g1 of the subject cutout moving image M2 Is overwritten with the pixel value of the corresponding pixel (step S86), and for the pixel whose alpha value is 0 <α <1 (step S85; 0 <α <1), the one's complement (1-α) is used to determine the subject. When an image (background image × (1−α)) obtained by cutting out the region S is generated and then the subject cutout moving image M2 is generated using the one's complement (1−α) in the alpha map, a single background color is generated. And the blended value is subtracted from the image frame g1 of the subject cutout moving image M2 and synthesized with the image obtained by cutting out the subject region S (background image × (1-α)) (step S87). ), Alpha value is 0 For iodine (Step S85; α = 0), so as monkey transmitting a background image P1 without doing anything.

Subsequently, the image composition unit 7f determines whether or not all the pixels of the background image P1 have been processed (step S88).
If it is determined that all the pixels have not been processed (step S88; NO), the image composition unit 7f designates the next pixel as the processing target and moves the processing target to the pixel (step S88). In step S89, the process proceeds to step S85.
By repeating the above processing until it is determined in step S88 that all pixels have been processed (step S88; YES), the image composition unit 7f allows the image frame g1 and the background image of the subject cutout moving image M2 to be processed. A first image frame i1 (see FIG. 11A) that forms a synthesized moving image of the subject synthesized moving image M4 synthesized with P1 is generated.

If it is determined in step S88 that all the pixels have been processed (step S88; YES), the composition control unit 7g determines whether all the image frames g constituting the subject cutout moving image M2 have been processed. Determination is made (step S90).
If it is determined that all image frames g have not been processed (step S90; NO), the composition control unit 7g performs processing among a plurality of image frames g,... Constituting the subject cutout moving image M2. The frame number related to the next image frame g as a target is designated by incrementing by +1 (step S91). As a result, the second image frame g2 is designated as the target of the image composition process.

Next, the image composition unit 7f shifts the processing to step S83, designates any one pixel (for example, the pixel in the upper left corner) of the background image P1, and performs subsequent processing (steps S84 to S84). By performing S89), the second image frame i2 (FIG. 11 (b)) that composes the synthesized moving image of the subject synthesized moving image M4 obtained by synthesizing the background image P1 and the second image frame g2 of the subject cutout moving image M2. ))).
If the generation of the second image frame i2 is completed and it is determined in step S88 that all the pixels have been processed (step S88; YES), the process proceeds to step S90, and the composition control unit In 7g, it is determined whether or not the processing has been performed for all the image frames g constituting the subject cutout moving image M2 (step S90).
By repeating the above processing until it is determined that all image frames g have been processed (step S90; YES), the image composition unit 7f allows all image frames constituting the composite moving image of the subject composite moving image M4. g1 to gn (see FIGS. 11A to 11D) are generated.
As a result, the normal synthesis process is terminated.

  As shown in FIG. 5, after that, the CPU 12 causes the display control unit 9 to generate image data of the synthesized moving image of the subject synthesized moving image M4 including the plurality of image frames i generated by the image synthesizing unit 7f. Each image frame i is switched at a predetermined display frame rate and reproduced and displayed on the display screen of the display unit 10 to reproduce a synthesized moving image of the subject synthesized moving image M4 in which the person moves his / her arm up and down with the background image P1 as the background. (Step S35).

As described above, according to the imaging apparatus 100 of the present embodiment, the movement of the subject in the subject cutout moving image M2 related to the composite image generation process is detected by the motion detection unit 6, and the detected movement of the subject is used as a reference. The background image P1 and the subject cutout moving image M2 are combined by the image combining unit 7f so that the subject is added and displayed at different positions of the background image P1 as the reproduction time elapses. Specifically, a subject composition composed of a plurality of image frames h,... Is added by adding the subject in the plurality of image frames g,. A synthesized moving image of the moving image M3 is generated.
As a result, the subject can be combined with the background image P1 in a novel manner so that the subject is added and displayed at different positions of the background image P1 as the playback time elapses. It is possible to generate a highly interesting composite image that is not simply combined with the work image P1.

Further, when it is determined by the movement determination unit 7e that the amount of movement of the subject in each image frame g of the subject cutout moving image M2 is equal to or greater than a predetermined threshold and the direction of movement of the subject is substantially equal, As the playback time elapses, the image composition unit 7f composes the images so that they are added and displayed at different positions in the background image P1, so that the subject is automatically added by analyzing the motion of the subject in the subject cutout moving image M2. Can be synthesized. As a result, the subject can be synthesized in a novel manner with the background image P1 even if the operation of the imaging apparatus 100 is not well known.
In the above-described embodiment, when the motion determination unit 7e determines that the motion direction of the subject in at least each image frame g of the subject cutout moving image M2 is substantially equal, the subject is used for the background as the playback time elapses. What is necessary is just to synthesize | combine to the image synthetic | combination part 7f so that addition display may be carried out in the different position of the image P1. However, if the amount of movement of the subject is smaller than a predetermined threshold, adding and synthesizing the subjects may increase the number of subjects to be combined, resulting in an unsightly synthesized moving image.
Accordingly, when the movement determination unit 7e determines that the amount of movement of the subject in each image frame g of the subject cutout moving image M2 is equal to or greater than a predetermined threshold and the direction of movement of the subject is substantially equal, By adding and synthesizing, it is possible to generate a synthesized moving image with good visibility.

Furthermore, when the motion determination unit 7e determines that the direction of movement of the subject in each image frame g of the subject cutout moving image M2 is not substantially equal, a plurality of image frames g, which constitute the subject cutout moving image M2, Are combined with the background image P1 by the image combining unit 7f. Therefore, when the direction of movement of the subject is different and is not suitable for addition combining, a normal image combining process is performed to obtain a plurality of image frames. A synthesized moving image of the subject synthesized moving image M4 composed of g,... can be generated.
In addition, since the addition synthesis of the subject and the normal synthesis are automatically switched by analyzing the motion of the subject in the subject cutout moving image M2, it is possible to provide the imaging apparatus 100 that is easier to operate.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, in the addition / combination process, the direction of the motion vector of the subject in any one of the plurality of image frames g,... Constituting the subject cutout moving image M2 by the motion determination unit 7e is the one image. It is determined whether or not there is a change in the direction of the motion vector of the subject in the image frame g before the frame g, and when it is determined that the change has occurred, the composition control unit 7g already adds to the still image. The displayed subject may be erased.
That is, the movement of the subject is substantially equal to the predetermined image frame g, but when the movement of the subject suddenly changes from the next image frame g, the addition display of the subject to the predetermined image frame g remains. This is to prevent the subject from being overlapped with a newly synthesized subject and becoming unsightly.
If there is no possibility that the subject addition display up to the predetermined image frame g overlaps with the newly synthesized subject even if the movement of the subject changes, for example, the movement of the subject up to the predetermined image frame g When the change is within a predetermined angle range with respect to the direction of movement of the subject, the composition control unit 7g does not necessarily need to delete the subject that has already been added to the still image.

  In the above embodiment, the Motion-JPEG format is exemplified as the recording method of the moving image data of the subject cutout moving image M2, but the MPEG-format may be used, for example. In such a case, in the synthesized image generation process, the moving image data in the MPEG format is decoded in advance to generate a plurality of image frames g,. The image P1 can be combined with the work image P1.

  In addition, the conventional template matching is performed for the detection of the motion vector, but it may be detected simply from the alpha map. In other words, the center of gravity coordinates of the pixel coordinates of the subject area having an alpha value of 1 (may be thinned out instead of all pixels) may be obtained, and the movement of the center of gravity coordinates between frames may be used as a motion vector.

In addition, the configuration of the imaging apparatus 100 is merely an example illustrated in the above embodiment, and is not limited thereto. That is, the imaging apparatus 100 is illustrated as the image composition apparatus, but is not limited thereto. For example, the image data of the subject cutout moving image M2, the background image P1, and the subject cutout moving image is generated by an image pickup device different from the image pickup device 100, and only the image data transferred from the image pickup device is recorded. Thus, an image composition apparatus that executes only the composite image generation process may be used.
Further, data relating to the composition position of the subject set based on the movement of the subject in the subject cutout moving image M2 is recorded in advance on the recording medium 8, and the display control unit 9 performs the composition recorded on the recording medium 8. Based on the data relating to the position, the still image and the moving image are synthesized so that the subject is added and displayed at different positions of the still image as the reproduction time elapses, and the synthesized moving image is reproduced on the display unit. May be.
That is, for example, the data related to the composition position of the subject is generated by an imaging device different from the imaging device 100, and the subject clipping moving image M2, the background image P1, and the composition position transferred from the imaging device are related. Only the data is recorded on the recording medium 8, and when the subject existing moving image M <b> 1 is reproduced, the display control unit 9 reproduces and displays the image while synthesizing the image based on the data relating to the synthesis position recorded on the recording medium 8. It may be a device that performs.

In addition, in the above embodiment, the functions as the synthesis unit, the detection unit, and the synthesis control unit are controlled under the control of the CPU 12 by the motion detection unit 6, the image synthesis unit 7f of the image processing unit 7, and the synthesis control unit. 7g is driven, and the functions as the image recording unit, the position recording unit, and the reproduction control unit are realized by driving the display control unit 9 under the control of the CPU 12. The present invention is not limited to this, and a configuration realized by executing a predetermined program or the like by the CPU 12 may be adopted.
That is, a program memory (not shown) that stores a program stores a program including a synthesis processing routine, a detection processing routine, and a synthesis control processing routine. Then, the synthesized moving image may be generated by synthesizing the background still image and the moving image by the synthesis processing routine. Moreover, you may make it make CPU12 detect the motion of the moving body in a moving image by a detection process routine. Further, the synthesis control processing routine causes the CPU 12 to synthesize the still image and the moving image so that the moving object is added and displayed at different positions of the still image as the playback time elapses with reference to the detected movement of the moving object. You may do it.
A program memory (not shown) that stores the program stores a program including an image recording control processing routine, a position recording control processing routine, and a reproduction control processing routine. Then, a background still image and a moving image may be recorded on the recording medium 8 by the image recording control processing routine. Alternatively, the CPU 12 may cause the recording medium 8 to record the combined position of the moving object set based on the movement of the moving object in the moving image by the position recording control processing routine. Further, based on the composite position recorded on the recording medium 8 by the reproduction control processing routine, the still image and the moving image are displayed so that the moving object is added and displayed at different positions of the still image as the reproduction time elapses. And a synthesized moving image may be reproduced.

DESCRIPTION OF SYMBOLS 100 Imaging device 1 Lens part 2 Electronic imaging part 3 Imaging control part 7 Image processing part 7d Cutout image generation part 7e Motion determination part 7f Image composition part 7g Composition control part 8 Recording medium 10 Display part 12 CPU

Claims (8)

Detecting means for detecting the movement of a moving object composed of a plurality of image frames;
Direction determination means for determining whether or not the direction of movement of the moving object detected by the detection means is substantially equal between adjacent image frames;
Synthesis control means for adding and synthesizing images of the moving object over time when the direction of movement of the moving object is determined to be substantially equal by the direction determining means;
Equipped with a,
The synthesis control means includes
When it is determined by the direction determination means that the direction of motion of the moving object in any one image frame has changed with respect to the direction of motion of the moving object in an image frame prior to the one image frame, An image synthesizing apparatus that erases an image of a moving object that has been added and synthesized . The synthesis control means includes
When the direction determining unit determines that the moving direction of the moving object is substantially equal, the plurality of image frames constituting the moving image are sequentially added and added as the playback time elapses. The image synthesizing apparatus according to claim 1, wherein the images are synthesized. The synthesis control means includes
2. A background still image and an image of the moving object are added and synthesized as the playback time elapses when the direction determining means determines that the moving direction of the moving object is substantially equal. Or the image synthesizing apparatus according to 2; A movement amount determination means for determining whether or not the amount of movement of the moving object detected by the detection means is greater than or equal to a predetermined value;
The synthesis control means includes
When the amount of movement of the moving object is determined to be equal to or greater than the predetermined value by the motion-amount determining means, according to claim 1 to 3, characterized in that to additive synthesis with the image of the moving object over playback time The image composition device according to any one of the above. The synthesis control means includes
When the direction determining means determines that the moving direction of the moving object is not substantially equal, each of the plurality of image frames constituting the moving image is combined with the background still image. The image composition device according to claim 3 or 4 . Combining means for combining a still image for background and a moving image to generate a combined moving image;
Detecting means for detecting movement of a moving object in the moving image;
Based on the motion of the moving object detected by the detecting means, the still image and the moving image are combined with the combining means so that the moving object is added and displayed at different positions of the still image as the playback time elapses. Synthesis control means
A motion determination means for determining whether or not the directions of motion of the moving object detected by the detection means are substantially equal;
With
The synthesis control unit adds moving objects in a plurality of image frames constituting the moving image to the still image by the synthesis unit to generate a synthesized moving image including a plurality of image frames, and the motion determination unit When the moving direction of the moving body is determined to be substantially equal to the above, the combining means is configured to combine and display the moving body at different positions as the playback time elapses,
The movement determination means includes
Determining whether or not the directions of movement of moving objects between adjacent image frames among the plurality of image frames constituting the moving image are substantially equal;
The synthesis control means further includes
When it is determined by the motion determination means that the direction of motion of the moving object in any one image frame has changed with respect to the direction of motion of the moving object in an image frame prior to the one image frame, An image composition apparatus for erasing a moving object added to and displayed on the still image. An image composition method in an image composition apparatus,
  A detection step of detecting a motion of a moving object composed of a plurality of image frames;
  A direction determination step for determining whether or not the direction of movement of the moving object detected by the detection step is substantially equal between adjacent image frames;
  A synthesis control step of adding and synthesizing the images of the moving objects as the playback time elapses when it is determined by the direction determining step that the moving directions of the moving objects are substantially equal;
  Including
  The synthesis control step includes
  When it is determined by the direction determination step that the direction of motion of the moving object in any one image frame has changed with respect to the direction of motion of the moving object in the image frame prior to the one image frame, An image synthesizing method comprising erasing an additively synthesized moving object image. The computer of the image synthesizer
Detecting means for detecting the motion of a moving object comprising a plurality of image frames;
Direction determining means for determining whether or not the direction of motion of the moving object detected by the detecting means is substantially equal between adjacent image frames;
Synthesis control means for adding and synthesizing images of the moving object over time when the direction of movement of the moving object is determined to be substantially equal by the direction determining means;
Function as
The synthesis control means includes
When it is determined by the direction determination means that the direction of motion of the moving object in any one image frame has changed with respect to the direction of motion of the moving object in an image frame prior to the one image frame, A program characterized by erasing an additively synthesized moving object image .

Images