JP2011030244A - Image processing apparatus, image processing method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To achieve such an operation transition image as to accurately represent a sufficient operation representation and an operation range of a moving subject using a number of frame images and also further to use the operation transition image as an effective moving image. <P>SOLUTION: An image processing apparatus synthesizes a plurality of frame image data items input to produce an operation transition image. Therefore, a synthesis processing section is provided to produce synthesized image data, in which the background layers of frame image data items are disposed side by side in order in a prescribed direction, by synthesizing the latest frame image data input with existent synthetic image data in response to input of frame image data separated into a moving subject layer and the background layer. In such a case, the synthesis processing section performs the synthesis in such a state that the moving subject layer has portions overlapping the corresponding background layer and a neighboring background layer, and preferentially synthesizes the moving subject layer prior to the background layer adjacent to the corresponding background layer. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly to a technique for obtaining a still image or a moving image that expresses a motion transition of a subject.

Japanese Patent No. 3449993 JP 2004-180259 A JP 2007-259477 A JP 2002-359777 A

For example, an “motion transition image” as shown in FIG. 32 is used for applications such as sports formation confirmation.
The motion transition image can easily recognize the motion transition by cutting out a person or an object (moving subject) at a certain characteristic time (key frame) and arranging them spatially. This is characterized in that even if the moving subject does not move spatially, an image moved spatially is intentionally generated. Generally, this is realized by drawing a picture, but it may also be realized by manually cutting out and arranging photographs.

Conventionally, when such an operation transition image is automatically generated, a key frame image to be processed is extracted from a plurality of frame images obtained by video camera video or continuous shooting by a camera. Further, a method of cutting out the target subject portion in the key frame and arranging it is performed.
However, the conventional motion transition image has a problem that it is difficult to achieve both expansion of the motion range and sufficient expression of the motion transition.

For example, FIG. 32 is an example in which motion transition images are generated using key frames FK1 to FK20 extracted from moving image data obtained by capturing a person who performs a golf swing with a video camera.
When a large number of key frame images are synthesized as one composite image as shown in FIG. 32, naturally, one cut-out width wa when the key frame images are arranged must be narrowed.
Then, the moving range of the moving subject cannot be covered, and a part of the moving subject that should be the target is cut off. For example, as can be seen from the images in FIG. 32, depending on the timing of the swing process, the golf club has an image lacking the tip portion. This is not visually desirable, and is not appropriate in consideration of uses such as form check.

Here, in order to prevent a part of a moving subject (for example, a swinging person and a golf club) from being lost, it is necessary to widen the cutout width.
FIG. 33 shows an example in which the cutout width from one key frame image is widened as shown as cutout width wb. In this case, the number of keyframe images that can be arranged as one image is reduced. For example, FIG. 33 shows an operation transition image generated by extracting eight key frames FK1, FK4, FK7, FK10, FK11, FK14, FK17, and FK20 from the 20 key frames FK1 to FK20 of FIG. .
In this case, looking at each key frame image, it is possible to prevent a part of the moving subject (swinging person and golf club) from being lost. However, since the number of key frames that can be arranged on one image is reduced, it is difficult to obtain an operation transition image that smoothly expresses movement. In other words, the operation transition as a form check cannot be expressed sufficiently finely.

  As shown in FIGS. 32 and 33, if the cutout width is increased so as to cover the entire motion range of the moving subject, the number of key frames existing in the final image to be generated is reduced, and the motion transition is not expressible. . Conversely, if the cutout width is narrowed, the number of key frames can be increased and the moving subject can be expressed finely in time, but the moving range of the moving subject cannot be covered, and a part of the moving subject that is supposed to be cut out. There is an inconvenience.

  In addition, such a motion transition image is conventionally a still image that represents a motion transition by simply arranging a plurality of key frames and combining the images, but the motion transition image has a higher visual effect. It is also demanded to realize.

  Therefore, an object of the present invention is to realize a motion transition image that enables a sufficient motion expression using a large number of frame images as a motion transition image and that can accurately represent the motion range of a moving subject. . A further object of the present invention is to realize a motion transition image as an effective moving image.

The image processing apparatus according to the present invention includes a layer separation unit that separates a moving subject layer and a background layer in an image from frame image data, and the latest input that is input to the existing composite image data according to the input of the frame image data. Image synthesis processing is performed to synthesize the frame image data to generate synthesized image data in which moving subject images of each frame image data are sequentially arranged in a predetermined direction. The moving subject layer of the latest input frame image data is set to the first with respect to the synthesis target area that is a part of the existing synthesized image data, which is an overlapping portion of the image data image and the existing synthesized image data image. Priority is given to the moving subject layer of the frame image data related to the previous image composition processing, and the latest input frame is given priority over the second. Giving priority to a background layer of the image data to a third, and a synthesis processing unit that performs the layer image combining process a background layer of the frame image data according to the previous image combining process in a state of priority in the fourth.
In addition, the image processing apparatus further includes an image output unit that outputs combined image data obtained as a result of image combining processing performed on the predetermined number of frame image data by the combining processing unit as motion transition still image data.
The image processing apparatus further includes an input image selection unit that selects a key frame to be used for generating a composite image from sequentially input frame image data, and the composition processing unit is responsive to an input of frame image data selected as a key frame. By performing image composition processing, motion transition still image data is generated.
Further, the image processing apparatus further includes a composite image holding unit that holds the composite image data generated by the image composite processing according to the input of the frame image data in the composite processing unit as the existing composite image data used in the next image composite processing. .

In addition, the image processing apparatus further includes an image output unit that continuously outputs the composite image data generated by the composite processing unit every time frame image data is input as motion transition video data.
In this case, the input image selection unit that selects a key frame to be used for generating a composite image from the frame image data that is sequentially input, and the composite processing unit according to the input of the frame image data that is the key frame. The image processing apparatus further includes a composite image holding unit that holds the composite image data generated by the image composition process as the existing composite image data used in the next image composition process.
Further, the composition processing unit performs image composition processing according to the input of each frame image data to generate composite image data regardless of whether the frame is selected as a key frame, thereby generating motion transition moving image data. Is generated.

  The composition processing unit gives first priority to the moving subject layer of the latest frame image data that has been input for a predetermined area of the existing composite image data, and the motion of the frame image data related to the previous image composition processing. The subject layer is given priority over the second, the background layer of the latest input frame image data is given priority over the third, and the background layer of frame image data related to the previous image composition processing is given priority over the fourth. I do.

In addition, the synthesis processing unit is input according to the position of the pixel to be processed when performing image synthesis processing for synthesizing the latest input frame image data on a predetermined area of existing synthesized image data. Processing to apply extracted pixels from the latest frame image data, processing to apply pixels of the moving subject layer of the latest input frame image data, processing to apply extracted pixels from the previous and previous composite image data, and input The layer image composition processing using the latest frame image data and the previous and previous composite image data is selectively executed.
In this case, in the layer image composition processing, the moving subject layer pixel of the latest input frame image data is given first priority, and the moving subject layer pixel of the frame image data related to the previous composite image data is second. The pixel of the background layer of the latest input frame image data is prioritized and the third priority is given.
Alternatively, in the layer image composition process, the moving subject layer pixel of the latest input frame image data is given priority first, and the moving subject layer pixel of each frame image data related to the plurality of composite image data before the previous time. Are given priority in order from the 2nd to the nth, and the pixels of the background layer of the latest inputted frame image data are given priority to the (n + 1) th.

Further, the image processing apparatus of the present invention inputs the frame image data into the existing composite image data according to the input of the frame image data and the layer separation unit that separates the moving subject layer and the background layer in the image. The image composition processing for combining the latest frame image data is performed to generate composite image data in which the moving subject images of each frame image data are sequentially arranged in a predetermined direction. The moving subject layer of the latest input frame image data is the first to the synthesis target area that is a part of the existing synthesized image data, which is an overlapping part of the frame image data image and the existing synthesized image data image. 1 is prioritized, and the moving subject layer of the frame image data related to the previous image composition processing is prioritized second and the latest input A composition processing unit for performing a layer image composition process so that the background layer of the frame image data is given priority over the third and the background layer of the frame image data related to the previous image composition process is given priority over the fourth; Prepare.
The image processing apparatus performs image composition processing every time frame image data is input, and generates and outputs composite image data, whereby moving subject images of each frame image data are sequentially arranged in a predetermined direction. In addition to executing the moving image display, the moving subject image of the frame image data used as the key frame is continuously displayed on the moving image display, and the moving subject image of the frame image data not used as the key frame is displayed as a moving image. The image composition process is performed so that the image is temporarily displayed above.

According to the image processing method of the present invention, for frame image data, a layer separation step for separating a moving subject layer and a background layer in the image, and the latest input to the existing composite image data according to the input of the frame image data. The image composition process for synthesizing the frame image data is performed to generate composite image data in which the moving subject images of each frame image data are sequentially arranged in a predetermined direction. The moving subject layer of the latest input frame image data is set to the first with respect to the synthesis target area that is a part of the existing synthesized image data, which is an overlapping portion of the image data image and the existing synthesized image data image. , The moving subject layer of the frame image data related to the previous image composition process is given priority over the second, and the latest input frame is input. Giving priority to a background layer of the over-time image data to a third, and a synthesis processing step of performing the layer image combining process a background layer of the frame image data according to the previous image combining process in a state of priority in the fourth.
Further, the image processing method of the present invention is inputted to the existing composite image data according to the layer separation step for separating the moving subject layer and the background layer in the image and the input of the frame image data. An image composition process for compositing the latest frame image data is performed to generate composite image data in which moving subject images of each frame image data are sequentially arranged in a predetermined direction. The moving subject layer of the latest input frame image data is added to the synthesis target area that is a part of the existing synthesized image data, which is an overlapping part of the frame image data image and the existing synthesized image data image. Priority is given to 1 and the moving subject layer of the frame image data related to the previous image composition processing is given priority to 2 and the latest input Giving priority to a background layer of the frame image data to a third, and a synthesis processing step of performing the layer image combining process a background layer of the frame image data according to the previous image combining process in a state of priority in the fourth.
The image processing method performs image composition processing every time frame image data is input, and generates and outputs composite image data, so that moving subject images of each frame image data are sequentially arranged in a predetermined direction. In addition to executing the moving image display, the moving subject image of the frame image data used as the key frame is continuously displayed on the moving image display, and the moving subject image of the frame image data not used as the key frame is displayed as a moving image. The image composition processing is performed so that the image is temporarily displayed above.
The program of the present invention is a program that causes an arithmetic processing device to execute these image processing methods.

In other words, according to the present invention, the image transition process is performed on the plurality of input frame image data to generate the motion transition image. Note that the frame image data is one of each frame constituting a moving image or, for example, one image of each still image captured continuously. That is, the image data constituting one image is shown widely.
In the present invention, when an image transition process is performed on a plurality of input frame image data to generate a motion transition image, a process of separating a moving subject layer and a background layer in the image is performed on the frame image data. Then, at each time point when the frame image data is input, the latest input frame image data is combined with the existing combined image data at that time point. At this time, a large number of moving subject images can be arranged by reducing the width of one frame. In this case, the moving subject of the latest frame image data and the moving subject of the previous frame image data are used. Overlapping parts occur. Therefore, the moving subject layer of the latest input frame image data is given priority first, and the layer image composition processing is given with priority given to the moving subject layer of frame image data related to the previous image composition processing second. To. As a result, a composite image is generated in a state where a part of the moving subject overlaps the moving subject in the previous frame.
This method is also applied to moving images. In the case of a moving image, it is only necessary to output the composition processing result at each time point as moving image data (motion transition moving image) while performing image composition processing on each frame image data that is sequentially input.

According to the present invention, in automatic generation of motion transition images, it is possible to achieve both accurate representation of moving subjects by expanding the range of motion and expression of detailed motion transitions in the time direction by arranging a large number of moving subjects. can do.
Further, by outputting the composite image data at each time point as a motion transition video, it is possible to realize a moving image that has a high visual effect and can easily capture the motion transition.

1 is a block diagram of an image processing apparatus according to an embodiment of the present invention. It is explanatory drawing of the operation | movement transition still image of embodiment. It is explanatory drawing of the production | generation process of the operation | movement transition still image of embodiment. It is explanatory drawing of the layer separation process of embodiment. It is explanatory drawing of the example of an area | region setting of the input image of embodiment. It is explanatory drawing of the transition operation | movement effective area | region and residual background area | region of embodiment. It is explanatory drawing of the layer synthetic | combination process of 4 layers of embodiment. It is explanatory drawing of the synthetic | combination process in the 1st production | generation process example of embodiment. It is a flowchart of the 1st example of a production | generation process of embodiment. It is a flow chart of layer separation processing of the 1st generation processing example of an embodiment. It is explanatory drawing of the motion transition moving image of embodiment. It is explanatory drawing of the motion transition moving image of embodiment. It is explanatory drawing of the motion transition moving image of embodiment. It is explanatory drawing of the motion transition moving image of embodiment. It is explanatory drawing of the motion transition moving image of embodiment. It is explanatory drawing of the motion transition moving image of embodiment. It is explanatory drawing of the motion transition moving image of embodiment. It is explanatory drawing of the synthetic | combination process in the operation | movement transition moving image and 2nd production | generation process example of embodiment. It is explanatory drawing of the operation | movement transition moving image and lower stage copy process of embodiment. It is explanatory drawing of the last frame of the operation | movement transition moving image of embodiment. It is a flowchart of the 2nd example of a production | generation process of embodiment. It is a flowchart of the process determination process in the 2nd generation process example of an embodiment. It is explanatory drawing of the image data used for a synthetic | combination process in the 2nd production | generation example of embodiment. It is explanatory drawing of process P2, P3 of the 2nd production | generation process example of embodiment. It is explanatory drawing of process P4 of the 2nd production | generation process example of embodiment. It is explanatory drawing of the process P5 of the 2nd production | generation process example of embodiment. It is explanatory drawing of process P6 of the 2nd production | generation process example of embodiment. It is explanatory drawing of the binary and multi-value moving subject information of embodiment. It is explanatory drawing of the binary and multi-value moving subject information of embodiment. It is a flowchart of a process in the case of applying the 1st production | generation process example of embodiment to a motion transition moving image. It is a flowchart of a process in the case of applying the 2nd production | generation process example of embodiment to an operation | movement transition still image. It is explanatory drawing of the conventional synthesized image. It is explanatory drawing of the conventional synthesized image.

Hereinafter, embodiments of the present invention will be described in the following order.
[1. Configuration of image processing apparatus]
[2. First Generation Processing Example of Motion Transition Image]
[3. Second Generation Processing Example of Motion Transition Image]
[4. Moving subject information]
[5. Application of first generation processing example to movie]
[6. Application of second generation processing example to still image]
[7. program]

[1. Configuration of image processing apparatus]

FIG. 1 shows a configuration example of an image processing apparatus 1 according to the embodiment.
In FIG. 1, the configuration of the image processing apparatus 1 of this example includes an input image selection unit 2, a layer processing unit 3, an image output unit 4, a composite image update holding unit 5, an image input unit 10, a moving subject information generation unit 11, A moving subject information input unit 12 and an output device 40 are shown.
These parts are not necessarily limited to the components in the same housing. In particular, the image input unit 10, the moving subject information generation unit 11, the moving subject information input unit 12, and the output device 40 can be considered as separate components. Further, only one of the moving subject information generation unit 11 and the moving subject information input unit 12 may be provided.
Each of the input image selection unit 2, the layer processing unit 3, the image output unit 4, and the composite image update holding unit 5 may be configured as a hardware block. However, in an arithmetic processing unit such as a microcomputer, software It can also be realized as a functional block realized by a program.
The image processing apparatus 1 may be provided inside a device such as an imaging device (such as a video camera) or a video reproduction device, or may be configured as a single image processing dedicated device. Furthermore, it may be configured to execute an image processing function realized by cooperation of software and hardware in a personal computer or the like.

The image input unit 10 inputs a frame image data group that is a source of a motion transition image (still image or moving image) to be generated.
The frame image data is one of each frame constituting a moving image or, for example, one image of each still image captured continuously. That is, the term “frame image data” is used to widely indicate the image data constituting one image.
The image input unit 10 corresponds to a lens system, a light receiving element, and a series of imaging system parts that obtain an image signal by performing image signal processing if the image processing apparatus 1 is mounted on the imaging apparatus. Become.
If a captured image obtained by an external imaging device is captured, the image input unit 10 may be a reception processing system for image signals transferred, transmitted, downloaded, etc. from an external device. For example, a tuner unit for broadcast waves, an external device interface unit such as a USB (Universal Serial Bus) interface, a wired or wireless network communication unit, and the like are assumed.
In addition, when an image signal such as captured image data is recorded on a recording medium such as a memory card (solid memory) or an optical disk, the image input unit 10 plays a reproduction system part or a reproduction system part that reproduces the recording medium Realized as a playback program.

Depending on the image input unit 10, each frame image data as a moving image or a plurality of frame image data obtained by still image continuous shooting or the like is input, and the image input unit 10 inputs these frame image data into an input image selection unit. 2 and the moving subject information generation unit 11.
The image input unit 10 may perform preprocessing. When a moving image file is received, it is expanded into each frame, and when an interlaced image is received, it is converted into a form necessary for processing such as progressive processing. In addition, enlargement / reduction may be performed.

The moving subject information generation unit 11 generates moving subject information using the original image group (frame image data group) received from the image input unit 10 and other external information (depth information, etc.).
The moving subject information is information that can determine which part is a moving subject and which part is a background in at least input frame image data.
The moving subject information generation unit 11 generates moving subject information corresponding to each input frame image data, and supplies it to the input image selection unit 2.

The moving subject information may be expressed as a moving subject image. In this case, whether each pixel is a moving subject or a background is expressed as a pixel value. In addition, information indicating the location of each subject may be expressed by an equation or a vector.
In addition to the separation of the moving subject and the background, the moving subject information generation unit 11 separates each of the moving subjects when there are a plurality of moving subjects, and may separate them according to depth. In this case, when the moving subject information is represented by an image, it is not a binary image of the moving subject or the background, but a multi-value image (or multi-channel image) representing what depth each pixel is at. This will be described later with reference to FIGS.

The moving subject information may not be generated in the image processing apparatus 1 of this example, and moving subject information generated by another device or program can be received. The moving subject information input unit 12 is shown as a part that receives moving subject information for each frame image data input by the image input unit 10 from the outside. The moving subject information input unit 12 inputs moving subject information corresponding to each input frame image data from the outside, and supplies it to the input image selection unit 2.
Therefore, at least one of the moving subject information generation unit 11 and the moving subject information input unit 12 may be provided. However, the moving subject information generated by the moving subject information generation unit 11 and the moving subject information input by the moving subject information input unit 12 can be used together. In this sense, the moving subject information generation unit 11 and the moving subject information are also included. It is also useful that both information input units 12 are provided.

The input image selection unit 2 performs processing for generating a motion transition image for the frame image data sequentially supplied from the image input unit 10. That is, necessary image data used for the composition processing in the layer processing unit 3 is selected, and an appropriate combination of images is output to the layer processing unit 3. Further, the input image selection unit 2 outputs information necessary for the synthesis process such as moving subject information corresponding to the frame image data output to the layer processing unit 3 and coordinate information for synthesis to the layer processing unit 3.
In FIG. 1, a key frame determination unit 21 and a coordinate calculation unit 22 are shown in the input image selection unit 2.
The key frame determination unit 21 performs key frame determination processing and selection of frame image data to be output to the layer processing unit 3 based on the key frame determination processing. The coordinate calculation unit 22 performs coordinate calculation for the synthesis process.

A key frame is a plurality of images with different time axes that are left in the same still image as a motion transition locus in a final output image (a final frame image in the case of a moving image) as a motion transition image. For example, it is generally assumed that frames captured at appropriate intervals per unit time in a group of continuous frame image data are assumed to be key frames. There is no need, and the interval may be changed to a suitable interval for visually recognizing the movement of the imaging target.
The key frame determination unit 21 performs a process of selecting frame image data to be a key frame from the frame image data sequentially supplied from the image input unit 10. As will be described later, when the motion transition still image is generated, only the selected key frame is used for the processing of the layer processing unit 3. When generating a motion transition video, the input all frame image data is used for the processing of the layer processing unit 3, but the combined processing in the layer processing unit 3 differs depending on whether it is a key frame or not. become.

The layer processing unit 3 includes a layer separation unit 31, a layer processing unit 32, and a layer synthesis unit 33, and performs a motion transition image generation process.
The layer separation unit 31 performs layer separation of the input frame image data using the moving subject information. A layer refers to each frame image data separated by a moving subject portion and a background portion. The layer separation unit 31 generates an image of each layer by separating the corresponding range in which layer synthesis is performed, for example, based on moving subject information, an input image and a previous key frame image into a background and a moving subject. If there are a plurality of moving subjects and their depths are known, the layers are separated into the number of moving subjects + the number of backgrounds.
The layer processing unit 32 performs processing such as cutting, scaling, and coordinate movement of each separated layer. In other words, various processes are performed on the separated layers to process them into a form that can be combined. The processing performed by the layer processing unit 32 is often performed with geometric operations such as “enlargement / reduction”, “rotation”, and “translation”. However, image processing such as enhancement of an operation part may be performed. Enlargement / reduction is determined by the number of key frames, the output image size, and the like.
The layer composition unit 33 performs composition processing using the processed layer and the previous composite image. That is, the output image is combined based on the image processed by the layer processing unit 32, the previous combined image, moving subject information, and the like. The composition here refers to determining which layer of pixels to reflect in the output image based on the moving subject information and generating the output image. There are cases where the pixel is selected from pixels of a single layer, but there are also cases where a plurality of layers of pixels are mixed and output.
The processing of these layer processing units 3 will be described later.

The image output unit 4 outputs the synthesized image synthesized by the layer processing unit 3 to the output device 40. The output device 40 refers to various devices such as a monitor device and a storage device that can be an output destination of an operation transition image. The specific output destination of the operation transition image by the image output unit 4 differs depending on the system, such as display output, development to a memory, writing to an external storage medium such as a hard disk, flash memory, or network destination.
When generating a motion transition still image, the image output unit 4 outputs one frame of image data as a motion transition still image that is completed when the composition processing for the frame image data that has been made into a predetermined number of key frames is completed. Is output to the output device 40.
When generating a motion transition moving image, the image output unit 4 outputs a combined image obtained by combining processing sequentially performed for each frame image data input as image data of one frame constituting each moving image. It will output to 40.
In addition, the image output unit 4 generates the current composite image data at the time of processing the next frame image data at the time of processing for the next frame image data at the time of motion transition still image generation in which all the frame image data to be combined is a key frame. In order to be able to use it as image data, it is output to the composite image update holding unit 5 to be updated and stored.
On the other hand, at the time of motion transition video generation, the image output unit 4 uses the current composite image data as the previous previous composite image data at the next processing when the frame image data related to the current processing is a key frame. In order to be able to use it, it outputs to the synthetic image update holding | maintenance part 5, and it updates and memorize | stores it.

The composite image update holding unit 5 holds the composite image data at each time point in the process of generating the motion transition still image.
For example, when generating a motion transition still image, the composite image at each time point output from the image output unit 4 is updated and held. For example, the previous synthetic image data required in the synthesis process is updated and held as the previous time, the last time, or the like. Also, key frame information at the time of generating each composite image, moving subject information of the key frame, and the like are stored and held.
When the motion transition moving image is generated, the composite image data generated in the key frame composition processing is retained while being updated. Also in this case, key frame information at the time of generating each composite image, moving subject information of the key frame, and the like are stored and held.

[2. First Generation Processing Example of Motion Transition Image]

Hereinafter, a first generation processing example will be described as an example of a motion transition image generation processing mainly executed by the input image selection unit 2, the layer processing unit 3, the image output unit 4, and the composite image update holding unit 5.
Here, the case of generating a motion transition still image as shown in FIG. 2 is taken as an example, the concept of processing for the first generation processing example will be described with reference to FIG. 2 to FIG. A simple processing example will be described.

First, the motion transition still image generated by the image processing apparatus 1 of this example will be described with reference to FIG.
For example, the motion transition still image as shown in FIG. 2 is generated based on a plurality of frame image data obtained by performing moving image capturing or still image continuous shooting with a person performing a golf swing as a subject.
The motion transition still image of FIG. 2 uses a relatively large number of frame image data as in FIG. 32 described above, and represents images at a large number of points in a continuous golf swing period. In addition, as shown in FIG. 32, for example, the tip of the golf club is not lost, and the entire moving subject (person and golf club) of each frame is expressed while partially overlapping the left and right images. ing. In other words, the motion transition image is an image in which sufficient motion expression is possible using a large number of frame images and the motion range of the moving subject is appropriately expressed. Such an operation transition image is realized by the processing described below.

FIG. 3 shows an image of the operation transition still image generation process.
FIG. 3A schematically shows image data input to the input image selection unit 2. For example, it is moving image data, and F1, F2, F3,... In the figure indicate one frame image data.
The input image selection unit 2 selects a key frame from temporally continuous frames as such a moving image. For example, assuming that every fifth frame is a key frame, frames F1, F6, F11, F16, F21,. FIG. 3B shows the image contents of each key frame in a graphic form.

As will be described later, when generating a motion transition video, frame image data other than key frames (for example, F2, F3, etc.) are also used for the synthesis process. However, when generating a motion transition still image, only the key frame is synthesized. Used for processing.
In this case, the input image selection unit 2 extracts only this key frame and transfers it to the layer processing unit 3.
The layer processing unit 3 sequentially performs a composition process every time the images of the key frames F1, F6, F11, F16, F21,. Thus, the motion transition still image as shown in FIG. 3C is finally generated. At this time, the layer composition processing is performed so that the moving subject image of the key frame to be synthesized this time is overlapped with the previously synthesized images.

The processing of the layer processing unit 3 that performs such image composition is as follows.
The layer separating unit 31 separates the input range and the previous key frame image into the background and the moving subject based on the moving subject information, and generates an image of each layer.
FIG. 4A shows an input image (one frame image data) and moving subject information about the frame image data.
As described above, the input image selection unit 2 uses the frame image data (for example, F1) as a key frame among the frame image data F1, F2, F3... Sequentially supplied from the image input unit 10 as the layer processing unit 3. Output to.
The input image selection unit 2 is supplied with moving subject information corresponding to each frame image data from the moving subject information generation unit 11 or the moving subject information input unit 12. When the input image selection unit 2 supplies frame image data as key frames to the layer processing unit 3, the input image selection unit 2 also supplies moving subject information corresponding to the frame image data to the layer processing unit 3.

  The upper and lower parts of FIG. 4A show the frame image data and the moving subject information provided to the layer processing unit 3 in this way. Here, it is assumed that the moving subject information is expressed by binary values. That is, the information is “1” for a pixel that is a moving subject, “0” for a background pixel other than the moving subject, and the like. In the lower part of FIG. 4A, the white portion represents “1”, that is, a moving subject, and the black portion represents “0”, that is, a background.

The layer separation unit 31 separates the input frame image data by using such moving subject information.
FIG. 4B shows the moving subject layer and the background layer after separation. The moving subject layer in the upper part of FIG. 4B is an image obtained by extracting the moving subject portion (that is, only the moving subject information = “1” pixel) from the input image of FIG. 4A.
The lower background layer in FIG. 4B is an image obtained by extracting the background portion (that is, only the moving subject information = “0” pixel) from the input image in FIG.

In addition, assuming that there are a plurality of moving subjects, multi-value moving subject information of three or more values can be used. This will be described later with reference to FIGS. 28 and 29, but the moving subject information generation unit 11 or the moving subject information input unit 12 has different values depending on the depths of the moving subjects (the front-rear relationship in the depth direction on the image). The moving subject information is generated (or acquired).
In such a case, the layer separation unit 31 can separate the layer into the number of moving subjects + the number of backgrounds.

As described above, the layer processing unit 32 performs various processes on the layers separated into layers and processes them into a form that can be combined.
The motion transition image is usually performed by cutting out a part of the input frame image data rather than the entire region.
Furthermore, the range in which the moving subject is effective is different from the background area left in the final image.
The width and height of the image as the input frame image data (Wimg, Himg), the width and height of the transition operation effective area (Wmov, Hmov), and the width and height of the remaining background area (Wback, Hback) ), For example, a region mode as shown in FIGS. 5A, 5B, and 5C is assumed. 5A, 5B, and 5C are examples, and the present invention is not limited to these.
The transition motion effective area shown in FIGS. 5A, 5B, and 5C is a range in which a main moving subject is cut out. The remaining background area is an area used as a background image in the synthesis process.

These transition motion effective region values (Wmov, Hmov) and remaining background region values (Wback, Hback) are the transition direction of the subject, that is, the direction in which the image progresses in time when expressed as a motion transition image. Depending on the width, the width and height are the same.
When moving each moving subject image in the horizontal direction as in the motion transition image shown in FIG. 2, the height (Hmov, Hback) is the same as that of the original frame image data as in the example of FIG. In many cases, it is preferable to set the same value as the height (Himg).
When moving each moving subject image in the vertical direction, the width (Wmov, Wback) is set to the same value as the width (Wimg) of the original frame image data as in the example of FIG. Often preferred.
When a movement target is a person and standing, such as in a sports form check, the transition is often horizontal, and when a judo sleeping technique is a target, the possibility of vertical transition is high. As described above, the transition direction and the size of each image associated therewith strongly depend on the imaging target.

FIG. 6 shows a setting example of the transition motion effective area and the remaining background area according to the example of FIG. 5B for the actual frame image data.
FIG. 6A shows the original input frame image data. For example, it is assumed that the size of the frame image data is VGA (640 × 480). It is assumed that the centers of the transition motion effective area and the remaining background area coincide with the center of the frame image data that is the input image.
The transition operation effective area is an area having a width Wmov = 320 and a height Hmov = 480 as shown in FIG.
The remaining background area is an area having a width Wback = 160 and a height Hback = 480 as shown in FIG.
Of course, this is merely an example, and in practice, an appropriate setting may be made according to the size and movement of the subject, and an image used for the composition process may be cut out.

For example, the layer compositing unit 33 receives the input image processed by the layer processing unit 32 (frame image data set as the current key frame), the previous key frame image, moving subject information, and existing composite image data at that time. Originally, the output image is synthesized.
Note that the previous key frame image is an image that was previously input from the input image selection unit 2 and used for the composition processing, but this seems to be stored in the layer processing unit 3 until the current processing time point. Alternatively, it may be stored in the composite image update holding unit 5 and read out from the composite image update holding unit 5 together with the previous composite image data during the synthesis process.

The concept at the time of composition of the input image (current key frame) and the previous key frame will be described with reference to FIG.
By the processing of the layer separation unit 31 described above, the input current key frame is separated into the moving subject layer and the remaining background layer.
The previous key frame is also separated into a moving subject layer and a remaining background layer.
FIG. 7 shows the composition of these four layers.
As shown in FIG. 7A, the moving subject layer of the latest input frame image data (current key frame) is given first priority.
Also, the moving subject layer of the frame image data (previous key frame) related to the previous combining process is given second priority.
Further, the remaining background layer of the latest input frame image data (the current key frame) is given the third priority.
Further, the remaining background layer of the frame image data (previous key frame) related to the previous combining process is given priority over the fourth.
Then, based on these priorities, the four layer images are combined to obtain a combined image as shown in FIG.
Then, by combining such a composite image with a predetermined area of existing composite image data at that time, one moving subject image is newly added to the composite image using a number of key frames.

For example, FIG. 8 shows a composite state at a certain point in time until the motion transition still image is finally generated. In other words, this is the time when the composition process for the key frame Fx input this time is performed.
It is assumed that the transition operation effective area of the key frame Fx input this time and the range of the existing combined image data combined at the time of the previous key frame input are the ranges shown in the drawing.
When the image of the current key frame Fx is added by the composition process, the pixels in the transition operation effective region of the current key frame Fx may be directly copied onto the new composite image data for the region B in the figure.
For the area C, the existing composite image data at that time, that is, the pixel of the composite image data at the time after the composite processing up to the previous key frame may be copied as it is onto the new composite image data.
In this case, the A area becomes a problem. That is, in the area A, the moving subject FxD of the current key frame Fx and the image of the moving subject F (x−1) D of the previous key frame partially overlap. In this area A, it is necessary to synthesize the layers in the priority order as shown in FIG.

That is, the layer synthesizing unit 33 uses the current key frame Fx and the previous key frame, and generates a synthesized image by performing layer synthesis of FIG. As the B area, the portion corresponding to the B area of the current key frame Fx is used as it is. As a result, a composite image of the A + B region is obtained from the current and previous key frame images. Then, such a composite image of the A + B region is read out from the composite image update holding unit 5 and is combined with the existing composite image data at this time. That is, the process of pasting the A + B area generated this time is performed after the C area of the existing composite image data.
Thus, the composite image data in the state shown in FIG. 8 is obtained.
This composite image data is stored in the composite image update holding unit 5 and is used in the same manner in the composite processing relating to the next key frame F (x + 1).

  As described above, the layer composition shown in FIG. 7 is performed in the area A that is an overlapping part in the composition, so that a part of the moving subject FxD is not lost and is added to the synthesized image data so far. . Therefore, as described with reference to FIG. 2, even when a large number of key frames are used, a motion transition image in which the motion of the moving subject is appropriately expressed is generated.

A specific processing example of the image processing apparatus 1 will be described with reference to FIGS. 9 and 10.
FIG. 9 shows processing executed by the input image selection unit 2, the layer processing unit 3, the image output unit 4, and the composite image update holding unit 5 when generating a motion transition still image.

Steps F101 to F104 are processing of the input image selection unit 2.
As described above, temporally continuous frame image data is supplied from the image input unit 10 to the input image selection unit 2. Each time one piece of frame image data is supplied, the input image selection unit 2 responds to the frame image data and the frame image data from the moving subject information generation unit 11 (or the moving subject information input unit 12) in step F102. A process for capturing moving subject information to be performed is performed.
Note that the determination in step F101 is a determination of whether a series of frame image data for generating a motion transition still image has been taken, that is, whether the supply from the image input unit 10 has been completed. That is, the process advances from step F101 to F102 until the input of a predetermined number of key frames to be reflected as the motion transition still image is completed.

  When the input image selection unit 2 fetches the frame image data and the moving subject information in step F102, the input image selection unit 2 selects whether or not the frame image data is used as a key frame in step F103. For example, when key frame selection is performed as shown in FIG. 3, every fifth frame of first, sixth, eleventh,...

  If the input frame image data is a key frame, the input image selection unit 2 advances the process to Step F104. If the input frame image data is not a key frame, the input image selection unit 2 returns the process to step F101 and performs the process of inputting the next frame image data. That is, in the operation transition still image generation process, only the key frame is used for the composition process.

When the frame image data captured in step F102 is a key frame, the input image selection unit 2 performs a process for using the frame image data for the synthesis process. For this reason, coordinate calculation is performed in step F104. This coordinate calculation is to calculate the pixel position at which the current frame image data is reflected in the composite image. That is, this is a pixel range calculation process to which the current frame image data is applied in the final synthesized image size (motion transition still image size).
The input image selection unit 2 outputs the coordinates calculated in step F104 to the layer processing unit 3 together with the current frame image data (current key frame image) and moving subject information.

In the layer processing unit 3, first, in step F105, the layer separation unit 31 receives the input frame image data (current key frame image) and moving subject information, and the moving subject of the previous key frame image and previous key frame image. Divide into 4 layers using information. That is, as described in FIG. 4, the moving subject information is used to divide the moving subject layer and the background layer, and as shown in FIG. 7A, the moving subject layer of the current key frame, the moving subject layer of the previous key frame, the current key Divide into the frame background layer and the previous keyframe background layer.
For the moving subject layer of the previous key frame and the background layer of the previous key frame, the layer separation result performed in step F105 when the previous key frame is input can be stored for use in this processing. That's fine. Alternatively, the previously input key frame image data and moving subject information may be stored for use in the current process.

For the four-layer division, each layer image shown in FIG. 7A may be created and stored in the memory area. Here, an example in which layer separation is performed by creating “layer separation information” will be described. To do.
This layer separation information is a pixel indicating which layer is finally effective as a portion where the transition operation effective area of the previous key frame image and the current key frame image overlaps, that is, each pixel in the composition target area described in FIG. It is expressed by value.

This process is shown in FIG. The process of FIG. 10 is a process of determining one by one for all the pixels in the current key frame transition operation effective area.
First, in step F200, the layer processing unit 3 sets the first pixel of the current key frame transition operation effective area as a determination target. Then, the determination process of steps F201 to F210 is performed for the determination target pixel.
In step F211, the determination process in steps F201 to F210 is repeated while the determination target is changed to the next pixel in step F212 until the determination process is completed for all pixels in the current key frame transition operation effective region.

In step F <b> 201, the layer processing unit 3 determines, for one determination target pixel, whether the pixel is a pixel in the composition target area.
If it is not within the compositing target area, the process proceeds to step F202, and the layer processing unit 3 sets the determination target pixel in the current key frame as a pixel to be copied to the compositing result image (layer composite image). For example, the pixel value “0” is stored in the layer separation information for the determination target pixel. Then, the process proceeds to Step F211.
For example, for the pixel corresponding to the area B in FIG. 8, the process from step F201 to step F202 is performed.

If the determination target pixel is within the compositing target area, the layer processing unit 3 proceeds to step F203 to determine whether the determination target pixel is a pixel in the moving subject area of the current key frame. If the determination target pixel is a pixel in the moving subject area of the current key frame, the pixel value “1” is stored in the layer separation information for the determination target pixel in step F204. Then, the process proceeds to Step F211. The pixel value “1” is a value indicating “first priority” in FIG.
For example, in the area A in FIG. 8 (within the composition target area), the process from step F203 to F204 is performed on the pixels in the moving subject area of the current key frame, for example, the pixels at the tip of the golf club. Become.

If it is determined in step F203 that the pixel to be determined is not a pixel in the moving subject area of the current key frame, the layer processing unit 3 corresponds to the moving subject area in the previous key frame in step F205. It is determined whether or not to do. If this is the case, the layer processing unit 3 proceeds to step F206, and stores the pixel value “2” in the layer separation information for the determination target pixel. Then, the process proceeds to Step F211. The pixel value “2” is a value indicating “second priority” in FIG.
For example, in the area A (in the composition target area) in FIG. 8, the pixels in the moving subject area of the previous key frame, for example, the pixels constituting the moving subject F (x−1) D, etc. Processing will be performed.

If it is determined in step F205 that the pixel to be determined is not a pixel in the moving subject area of the previous key frame, the layer processing unit 3 corresponds to the background area of the current key frame in step F207. It is determined whether or not. If this is the case, the layer processing unit 3 proceeds to step F208, and stores the pixel value “3” in the layer separation information for the pixel to be determined. Then, the process proceeds to Step F211. The pixel value “3” is a value indicating “third priority” in FIG.
If it is determined in step F207 that the pixel to be determined is not a pixel in the background region of the current key frame, the layer processing unit 3 corresponds to the background region in the previous key frame in step F209. It is determined whether or not. If this is the case, the layer processing unit 3 proceeds to step F210, and stores the pixel value “4” in the layer separation information for the pixel to be determined. Then, the process proceeds to Step F211. The pixel value “4” is a value indicating “fourth priority” in FIG.

The above processing is performed until it is determined in step F211 that all pixels in the current key frame transition operation effective region have been determined.
The four layers as shown in FIG. 7A in the composition target area are “background layer of the previous key frame image (fourth priority)” “remaining background layer of the current key frame (third priority) in order from the bottom. “The moving subject layer of the previous key frame image (second priority)” and “the moving subject layer of the current key frame (first priority)”. These are overlapped in order from the bottom, and the one viewed from the top is the output image. That is, when the upper layer is given priority and there is a valid pixel in the layer above the self layer, it does not matter whether the pixel of the self layer is valid or invalid, or any pixel value.

By performing the processing of FIG. 10, the layer separation information holds the effective layer number for each pixel in the composition target area.
For example, if a certain pixel value of the layer separation information is “1”, it is indicated that a pixel extracted from the moving subject layer of the current key frame image that is the first priority should be used for the pixel position in the result image. .
Further, for example, if a certain pixel value of the layer separation information is “2”, it is indicated that a pixel extracted from the moving subject layer of the previous key frame image having the second priority is used.
That is, each layer described in FIG. 7A is expressed by the layer separation information.

  Subsequently, the layer processing unit 3 performs processing by the layer processing unit 32 as Step F106 of FIG. That is, processing necessary for layer composition is performed. Note that as this processing, enlargement / reduction, rotation of the image, or the like is performed according to the size of the input image and the size that reflects the input image on the resultant image. In particular, when the pixels of the input image are reflected as they are in the synthesis result, the enlargement / reduction or rotation processing may not be necessary.

  Subsequently, in step F107, the layer processing unit 3 performs layer composition to generate a layer composite image at the current key frame and the previous key frame. Note that the layer composite image referred to here is an image of the A + B region in FIG.

First, for the pixel determined to have the pixel value “0” in step F202 in FIG. 10, that is, the pixel in the area B in FIG. 8, the pixel extracted from the current key frame is copied to the layer composite image as it is. Apply.
Layer synthesis is performed for each pixel in the synthesis target area (A area).
In this case, the layer separation information is referred to for each pixel, and the corresponding pixel is extracted from the current key frame and applied to the layer composite image at the pixel position where the pixel value is “1”. If the pixel value is “2”, the corresponding pixel is extracted from the previous key frame and applied to the layer composite image.
At the pixel position with the pixel value “3”, the corresponding pixel is extracted from the current key frame and applied to the layer composite image. If the pixel value is “4”, the corresponding pixel is extracted from the previous key frame and applied to the layer composite image.
As described above, information indicating which pixels of the four layers are valid is stored in the layer separation information for each pixel position. Therefore, for each pixel position, by extracting and pasting a pixel from the current key frame or the previous key frame according to the pixel value of the layer separation information, the composition target region in FIG. 8 is synthesized according to the priority order of FIG. Will be done.

Through the above processing, a layer composite image corresponding to the A + B region in FIG. 8 is generated. In this layer composite image, as shown in FIG. 8, the moving subject FxD of the current key frame Fx and the previous key frame are moved. While the image of the subject F (x−1) D partially overlaps, the moving subject FxD of the current key frame Fx is expressed without being missing.
In the layer compositing process, since the moving subject layer of the current key frame has the first priority, the portion where the moving subject image overlaps this time and the previous time is always the current subject, that is, the moving subject image of the latest frame at that time. Will be given priority.

Subsequently, in step F108, the layer processing unit 3 combines the layer composite image with the existing previous composite image at that time. Since the previous previous composite image data is stored in the composite image update holding unit 5, the layer processing unit 3 adds the layer composite image generated in step F107 to the previous composite image data read from the composite image update holding unit 5. Is synthesized.
In the example of FIG. 8, the existing combined image data is in a state where the key frame images up to the C area + A area are combined at this point. A layer composite image in the A + B area is copied to such composite image data. The A area is overwritten by the layer composite image. As a result, composite image data in a state as shown in FIG. 8 is generated.

The current composite image data generated in this way is transferred from the layer processing unit 3 to the image output unit 4 in step F109. The image output unit 4 captures this composite image data and then supplies it to the composite image update holding unit 5.
In step F110, the composite image update holding unit 5 stores and holds the supplied composite image data as existing composite image data used for the next composite processing. That is, the content of the existing composite image data is updated to the composite image data generated this time.
Then, the process returns to step F101, and the process for the input of the next frame image data is started.

In the example of FIG. 9, layer composition is performed in step F107, and the layer composite image is combined with existing composite image data in step F108. However, the processing when the first key frame is input is particularly Needless to say, the synthesis process is unnecessary. Of course, there is no existing composite image data at this point.
In this case, in steps F107 to F108, each pixel of the image of the first key frame is copied to the upper right portion of the composite image data having a size as shown in FIG. Thus, it is stored in the composite image update holding unit 5.
After the second key frame is input, the previous key frame and existing composite image data exist at that time, so the same processing as described above is performed.

  In addition to the processing in steps F107 and F108, various types of composite image data generation processing are possible. For example, the pixels in the area B are copied first into the existing composite image data, and then a layer composite image for the area A is generated, and the layer composite image is stored in the area A of the existing composite image data. You may make it overwrite copy.

The above-described processing of steps F101 to F110 is repeated for each key frame input, so that composite image data in the state shown in FIG. 2 is finally obtained.
At this point, it is determined in step F101 that all frames have been captured. Then, the process proceeds to step F111, and the image output unit 4 outputs the composite image data held at that time (that is, the composite image data in the state of FIG. 2) to the output device 40 as motion transition still image data.
As a result, one piece of motion transition still image data is generated and output by the image processing apparatus 1 of this example.

As described above, according to the present embodiment, in automatic generation of a motion transition image, accurate representation of a moving subject by expansion of the operable range and fine motion transition in the time direction by arranging a large number of moving subjects. It is possible to achieve both expressions.
In particular, by layer synthesis, each moving subject image can be accurately represented, for example, without missing the tip of a golf club, and a large number of moving subject images can be arranged, and motion transitions can be expressed at fine time intervals.
This makes it possible to easily and very easily confirm the formation of one's own sports, complex movements of wild animals, and natural phenomena in the laboratory. Moreover, since manual work such as cutting and pasting of image data is not required, the burden on the user is very light and useful.

The image processing apparatus 1 according to the present embodiment may be any apparatus that can acquire a plurality of frame image data captured by an imaging apparatus that can capture a continuous still image or a moving image. For example, it may be built in the imaging device, or may be built in a playback device that plays back a plurality of captured frame image data. Further, it may be incorporated in a device that receives / transfers input a plurality of captured frame image data.
Therefore, the present invention can be widely applied to, for example, a mobile phone, a PDA (Personal Digital Assistant), a personal computer, a video playback device, a video editing device, and the like.

[3. Second Generation Processing Example of Motion Transition Image]

Next, a second generation processing example will be described as an example of a motion transition image generation processing executed mainly by the input image selection unit 2, the layer processing unit 3, the image output unit 4, and the composite image update holding unit 5. Go.
Here, a case where a motion transition moving image as shown in FIGS. 11 to 20 is generated is taken as an example, and a specific processing example is described with reference to FIGS.
Note that the first generation processing example described above and the second generation processing example below are examples in which the synthesis processing method is different, and the difference in whether the generated motion transition image is a still image or a moving image is different. Absent.
As will be described later with reference to FIGS. 30 and 31, a motion transition video can be generated along the first generation processing example, or a motion transition still image can be generated along the second generation processing example described below. You can also

First, the motion transition moving image generated by the image processing apparatus 1 of this example will be described.
In the case of the above-described still image, only the key frame is used, but in the case of a moving image, other than the key frame is used. However, the key frame is finally reflected in the composite image, and the other than the key frame is temporarily included in the output image during the moving image process.
Note that whether each frame image data is a key frame is based on the example of FIG.

FIG. 11 shows composite image data when the first frame image data (for example, key frame) F1 is input. In FIG. 11 to FIG. 17, (K) is attached to the frame image data that is a key frame.
Next, FIG. 12 shows composite image data when second frame image data F2 that is not a key frame (hereinafter, frame image data that is not a key frame is referred to as “non-key frame”) is input. In this case, the non-key frame F2 is synthesized in a pixel range shifted by a predetermined amount with respect to the key frame F1. As in the first generation processing example described above, the moving subject image of the latest frame is always given priority for the portion where the moving subjects overlap.
For the synthesis, a method of synthesizing the input frame image data with the existing synthesized image data is adopted. The existing composite image data at the time of synthesizing the non-key frame F2 is composite image data as shown in FIG.

Next, the composite image data when the third non-key frame F3 is input is shown in FIG. In this case, the non-key frame F3 is synthesized in a pixel range shifted by a predetermined amount with respect to the non-key frame F2 of FIG.
Here, the composition processing adopts a method of synthesizing the input frame image data with the existing composite image data as described above, but the composite image data when the non-key frame is input is the existing composite image. Not used as data.
Therefore, even when this non-key frame F3 is input, the existing composite image data remains the composite image data of FIG.
That is, the previous non-key frame F2 is not used for composition, and the non-key frame F3 is synthesized with the existing composite image data at that time. The image to be expressed.

  Although not shown, when the fourth non-key frame F4 is input, the composite image data of the key frame F1 + the non-key frame F4 is generated in the same manner, and when the fifth non-key frame F5 is input. Similarly, composite image data of key frame F1 + non-key frame F5 is generated. That is, in any case, the frame image data input to the existing combined image data at that time is combined.

Similarly, when the key frame F6 is input, it is synthesized with the existing synthesized image data.
FIG. 14 shows composite image data when the second key frame F6 is input. The key frame F6 is combined with the existing composite image data (in this case, only the key frame F1).
However, at this time, the composite image data of FIG. 14 is also stored as existing composite image data.
As the existing composite image data used for composition, not only the composite image data at the time of the previous key frame input (previous composite image data), but also the composite image data at the time of the previous key frame input (previous composite image data) ) May be used.
For example, in a specific processing example to be described later, since the previous composite image data and the previous composite image data are used, the description will be given here also using the previous composite image data and the previous composite image data.
Note that the existing combined image data is used for combining each pixel with priority as in the above-described layer combining, but in that sense, the extent to which the overlapping portions of the moving subjects are in the range Accordingly, the use range of the composite image data that goes back in the past may be set. For example, an example is also conceivable in which the combined image data before and after the combined image data before and after the combined image data is used for the combining process.

  After the composite image data shown in FIG. 14 is generated, the previous composite image data (the composite image shown in FIG. 14) and the previous composite image data (the composite image shown in FIG. 11) are held as the existing composite image data. Used in the synthesis process.

Next, the composite image data when the non-key frame F7 is input is shown in FIG. Also in this case, by synthesizing the image of the non-key frame F7 with the existing synthesized image data, a synthesized image is generated by the images of the key frames F1 and F6 and the image of the non-key frame F7 as shown in the figure.
Further, FIG. 16 shows composite image data when the non-key frame F8 is input. Also in this case, by synthesizing the image of the non-key frame F8 with the existing synthesized image data, a synthesized image is generated by the images of the key frames F1 and F6 and the image of the non-key frame F8 as shown in the figure.
The same applies when non-key frames F9 and F10 are input thereafter.

FIG. 17 shows the composite image data when the third key frame F11 is input. The key frame F6 is combined with the existing composite image data.
However, at this time, the composite image data of FIG. 17 is also stored as existing composite image data. At this time, as the existing composite image data, the previous composite image data is updated to the composite image of FIG. 17 and the previous composite image data is updated to the composite image of FIG.

As shown in FIGS. 11 to 17, input frame image data (key frame and non-key frame) are sequentially combined, and combined image data as illustrated is obtained at each time point. For example, these combined image data are output as one frame constituting each moving image. Further, similarly, composite image data is generated for each input of frame image data, and moving image output proceeds as shown in FIGS.
That is, the subject image of each frame including the non-key frame is synthesized while leaving the subject image of each key frame, and the synthesized image data is output at each time point.
As a result, a moving image representing a motion transition such as a golf swing, that is, a motion transition moving image is output.

Taking a process for generating such a motion transition video as an example, a second generation process example of a motion transition image will be described with reference to FIG.
FIG. 21 shows processing executed by the input image selection unit 2, layer processing unit 3, image output unit 4, and composite image update holding unit 5 when generating a motion transition video.
Steps F301 to F305 are processing of the input image selection unit 2.
As described above, temporally continuous frame image data is supplied from the image input unit 10 to the input image selection unit 2. Each time one piece of frame image data is supplied, the input image selection unit 2 responds to the frame image data and the frame image data from the moving subject information generation unit 11 (or the moving subject information input unit 12) in step F302. A process for capturing moving subject information to be performed is performed.
Note that the determination in step F301 is a determination of whether a series of frame image data for generating a motion transition moving image has been taken, that is, whether the supply from the image input unit 10 has ended. That is, the process proceeds from step F301 to F302 until the input of a predetermined number of frame image data to be reflected as the motion transition moving image is completed.

  When the input image selection unit 2 fetches the frame image data and moving subject information in step F302, in step F303, the input image selection unit 2 selects whether or not to use the frame image data as a key frame. For example, when key frame selection is performed as shown in FIG. 3, every fifth frame of first, sixth, eleventh,...

When the key frame determination is performed on the input frame image data, the input image selection unit 2 takes in the existing previous synthesized image data and the previous synthesized image data from the synthesized image update holding unit 5 at step F304.
Note that there is no previous synthesized image data and no previous synthesized image data until the synthesis process is performed on the frame image data set as the first key frame. Further, there is no previous combined image data until the combining process is performed on the frame image data set as the second key frame. In step F304, if the previous composite image data and the previous composite image data were stored in the composite image update holding unit 5 at that time, the processing is to capture them.

In step F <b> 305, the input image selection unit 2 performs pixel position coordinate calculation processing for reflecting the current frame image data (key frame or non-key frame) in the composite image. That is, it is a calculation process of a pixel range to which the current frame image data is applied in the final synthesized image size (size of motion transition moving image).
The input image selection unit 2 outputs the coordinates calculated in step F305, the current frame image data and moving subject information, the previous combined image data, and the previous combined image data to the layer processing unit 3.

The layer processing unit 3 performs process determination processing in step F306.
The process determination is a process for determining what type of synthesis process is to be executed for each pixel (or for each column or for each row) within a range where the synthesis process is performed in order to perform an efficient synthesis process. is there.
Note that, as shown in FIGS. 11 to 20, when generating a motion transition video in which the subject image transitions in the horizontal direction, the process determination process may be performed for each pixel or column. On the other hand, although not shown, when generating a motion transition moving image in which the subject image transitions in the vertical direction, the process determination process may be performed for each pixel or for each row. The following description is based on an example in which process determination is performed for each column in accordance with the examples of FIGS.

The process determination process performed in step F306 is shown in FIG.
In this example, the current frame image data, the previous synthesized image data, and the previous synthesized image data are used for the synthesis process. When two existing composite image data are used, the number of processes to be determined is six (processes P1 to P6). If the number of combined image data to be used increases, such as the combined image data, the number of processes increases.
Also, in the composition process, the previous key frame range information and moving subject information included in the previous composite image data, and the previous key frame range information and moving subject information included in the previous composite image data are used. These may be stored in the composite image update holding unit 5 along with the previous composite image data and the previous composite image data, and read in step F302.

The process determination process in FIG. 22 is performed for each column of interest (may be a row as described above).
Each column in which the process is determined is, for example, each column in the range of A1 to A7 at the time of FIG.
Within this range, process determination is performed with each pixel row (in the case of FIG. 18, the upper pixel row in the composite image data) as a target column.
First, in step F401, the layer processing unit 3 sets one column as a target column and acquires the pixel position (coordinate position in the composite image data).
In step F402, it is determined whether or not the position of the target column is within the range of the previous combined image data. The range of the previous combined image data is the range from the A3 area to the A7 area in FIG.
If the target column is not within the range of the previous combined image data, the layer processing unit 3 proceeds to step F403, and determines whether or not the target column is within the current transition operation effective region. The transition operation effective area this time is the transition operation effective area (see FIG. 6B) of the frame image data (key frame or non-key frame) input this time, and the areas A2 to A5 in FIG. Range.

If the layer processing unit 3 determines in step F403 that the column of interest is not within the current transition operation valid area, the layer processing unit 3 determines that the process P1 is applied to the column of interest. This is a case where the column of interest falls within the A1 region of FIG.
If the layer processing unit 3 determines in step F403 that the column of interest is within the current transition operation effective area, the layer processing unit 3 determines that the process P2 is applied to the column of interest. This is a case where the target column is in the area A2 in FIG.

  If it is determined in step F402 that the column of interest is within the range of the previous composite image data, the layer processing unit 3 proceeds to step F404 and determines whether or not the column of attention is within the current transition motion effective region. . If it is determined in step F404 that the column of interest is not within the current transition operation effective area, it is determined that the process P3 is applied to the column of interest. This is a case where the column of interest falls within the A6 and A7 regions of FIG.

If it is determined in step F404 that the target column is within the current transition motion effective region, the layer processing unit 3 proceeds to step F405, and determines whether the target column is within the range of the subsequent region. As shown in FIG. 6 (b), the rear region is the rear side in the transition motion effective region with respect to the traveling direction A of the motion transition image (when moving from left to right), as shown in FIG. This is an area not included in the remaining background area of c). In the case of an image in which the motion transition shifts from right to left, it is the right region in the transition motion effective region. Further, in the case of an image whose motion transition is shifted from top to bottom, it is an upper region in the transition motion effective region.
In the example of FIG. 18, the column in the A5 area is determined to be in the rear area.
If it is determined in step F405 that the target column corresponds to the rear region, the layer processing unit 3 determines that the process P4 is applied to the target column. This is a case where the target column is in the area A5 in FIG.

If it is determined in step F405 that it is not the range of the rear region, the layer processing unit 3 proceeds to step F406, and determines whether or not the column of interest is within the range of the combined image data two times before. The range of the last-time synthesized image data is the range from the A4 area to the A7 area in FIG.
When it is determined that the column of interest is not within the range of the combined image data last time, the layer processing unit 3 determines that the process P5 is applied to the column of interest. This is a case where the target column is in the area A3 in FIG.
If it is determined in step F406 that the column of interest is within the range of the previous combined image data, the layer processing unit 3 determines that the process P6 is applied to the column of interest. This is a case where the target column is in the area A6 in FIG.

The layer processing unit 3 performs the composition process of step F307 every time the above-described process determination of FIG. 22 is performed for a certain target column in step F306 of FIG. Step F307 is a process of combining the pixels of the target column as a combined result image according to the process determination result, and layer separation, processing, and combining are performed as necessary.
Steps F308 and F309 will be described later, but the processes of steps F306 to F309 are repeated until it is determined in step F310 that the process has been completed for the current frame image data input this time. In other words, if the process determination process is performed in units of columns, the process determination is repeated until the process determination and the synthesis process are performed for all columns.

Each of the synthesis processes for the target column according to the process determination result will be described.
First, FIG. 23 shows image data used for the composition process.
FIG. 23A shows the transition operation effective area of the input image (frame image data) input this time in step F302.
FIG. 23B shows the previous combined image data existing at this time.
FIG. 23 (c) shows pre-existing composite image data existing at this time.
Note that these are examples of the input image, the previous combined image data, and the previous combined image data at the time of FIG.

The process P1 is a process that does not perform drawing.
Therefore, when it is determined that the column of interest is the process P1, in step F307, the combining process of reflecting the pixel in the combining result is not performed.
The columns to which the process P1 is applied are the columns of the A1 area in FIG. 18, and therefore, nothing is drawn in the A1 area.

The process P2 is a process of copying pixel data from the current frame image data transition operation effective area.
The columns to which the process P2 is applied are the columns in the area A2 in FIG. This A2 area is an area that does not overlap with the previous combined image data (and the previous combined image data). Therefore, the pixels of the current input image may be reflected as they are.
As shown in FIG. 24A, the pixels corresponding to the A2 area are pixels in each column indicated by hatched portions in the input image transition operation effective area.
The pixels in each column are copied as they are as the pixels in the area A2 in FIG.

Process P3 is a process of copying pixel data from the previous combined image data.
The columns to which the process P3 is applied are the columns A6 and A7 in FIG. The A6 area and the A7 area are areas that do not overlap with the transition operation effective area of the frame image data input this time. Therefore, the pixels of the previous composite image data may be reflected as they are.
As shown in FIG. 24B, the pixels corresponding to the A6 area and the A7 area are pixels in each column indicated by hatching in the previous synthesized image data.
The pixels in each column of the previous composite image data are copied as they are as the pixels in the A6 area and A7 area of FIG.

Process P4 is a process of copying pixel data of the moving subject image (moving subject layer) of the current frame image data onto the previous synthesized image data.
The columns to which the process P4 is applied are the columns in the area A5 in FIG. This A5 area is the subsequent area of the frame image data input this time. In this area, only the moving subject of the current frame image data is displayed so as to overlap the previous synthesized image data.
The upper part of FIG. 25 shows the moving subject image of the input image. This is a moving subject layer after the layer separation described with reference to FIG. 4 is performed on the input frame image data. That is, the layer processing unit 3 performs layer separation using the moving subject information on the frame image data input this time by the function of the layer separation unit 31, and generates an image of only the moving subject shown (image without a background). .
In the moving subject image in FIG. 25, the hatched portion corresponds to the rear area. Of the pixels in each column corresponding to the subsequent region, only the pixels constituting the moving subject image are transferred to the portion corresponding to the A5 region (hatched portion) of the previous synthesized image data shown in the lower part of FIG.
In this way, the state in which the moving subject pixel in the rear area of the current frame image data is transferred onto the previous composite image data is reflected as the composite image data in the area A5 in FIG.

Process P5 is a process of combining three layers of the moving subject image of the frame image data input this time, the moving subject image of the previous key frame, and the background image of the frame image data input this time.
The columns to which the process P5 is applied are the columns in the area A3 in FIG. This A3 area is a part where the transition operation effective area of the frame image data input this time overlaps with the previous synthesized image data. Therefore, the three layers are synthesized in the priority order shown in FIG.
FIG. 26A shows a moving subject image (moving subject layer) of the input image. FIG. 26C shows a background image (remaining background layer) of the input image.
The layer processing unit 3 performs the layer separation described with reference to FIG. 4 on the input frame image data by the function of the layer separation unit 31, and obtains these moving subject images and background images.
FIG. 26B shows a key frame when the previous combined image data is generated, that is, a moving subject image (moving subject layer) of the previous key frame. Although the previous key frame image is included in the previous synthesized image data, the moving subject image of the previous key frame can be extracted using moving subject information corresponding to the previous key frame.
In FIGS. 26 (a), (b), and (c), a portion corresponding to the area A3 on the combined image in FIG. 18 is hatched.
In the compositing process of process P5, the moving subject image of the input image is given the first priority and the moving subject image of the previous key frame as illustrated in FIG. Are combined with the second priority and the background image of the input image as the third priority.
That is, the highest priority is given to the pixels constituting the moving subject in the shaded area in FIG. In a portion other than the pixels constituting the moving subject in the hatched portion in FIG. 26A, if there is a pixel constituting the moving subject in the shaded portion in FIG. 26B, the pixel is applied. Pixels constituting the background image in FIG. 26C are applied to the other pixel portions.
The state in which the pixels of the three layers are combined in this way is reflected as the combined image data in the area A3 in FIG.

The process P6 includes four layers of the moving subject image of the frame image data input this time, the moving subject image of the previous key frame, the moving subject image of the previous combined image data, and the background image of the frame image data input this time. Is a process to synthesize.
The column to which the process P6 is applied is each column in the A4 area of FIG. The A4 region is a portion where the transition operation effective region of the frame image data input this time overlaps with the previous combined image data and the previous combined image data. Therefore, the four layers are synthesized with the priority shown in FIG.
FIG. 27A shows a moving subject image (moving subject layer) of the input image. FIG. 27D shows a background image (remaining background layer) of the input image.
FIG. 27B shows a key frame when the previous combined image data is generated, that is, a moving subject image (moving subject layer) of the previous key frame.
FIG. 27C shows a key frame when the synthesized image data is generated two times before, that is, a moving subject image (moving subject layer) of the key frame two times before.
The layer processing unit 3 performs the layer separation described with reference to FIG. 4 on the input frame image data by the function of the layer separation unit 31, and obtains these moving subject images and background images. Also, the moving subject image of the previous key frame and the moving subject image of the previous key frame are extracted from the previous synthesized image data and the previous synthesized image data using the moving subject information.
In FIGS. 27A, 27B, 27C, and 27D, a portion corresponding to the area A4 on the combined image in FIG. 18 is hatched.
In the compositing process of process P6, the moving subject image of the input image is given the first priority and the previous key frame of the input image as shown in FIG. 27 (a) (b) (c) (d). Layer synthesis is performed with the moving subject image as the second priority, the moving subject image of the key frame two times before the third priority, and the background image of the input image as the fourth priority.
That is, the highest priority is given to the pixels constituting the moving subject in the shaded area in FIG. In the portion other than the pixels constituting the moving subject in the shaded portion in FIG. 27A, if there is a pixel constituting the moving subject in the shaded portion in FIG. 27B, that pixel is applied. Further, if there is no pixel constituting the moving subject in FIGS. 27A and 27B and there is a pixel constituting the moving subject in the hatched portion of FIG. 27C, that pixel is applied. Pixels constituting the background image in FIG. 27D are applied to the other pixel portions.
The state in which the pixels of the four layers are combined in this way is reflected as the combined image data of the area A4 in FIG.

The synthesis process of the processes P1 to P6 is performed as described above.
In step F307 in FIG. 21, one of the processes P1 to P6 is executed as the process of the target column in accordance with the process determination result in step F306.
Note that the processing in steps F306 to F309 may be repeated not for each column but for each pixel.
Then, when any of the processes is performed for all the columns (all pixels) to be combined, for example, combined image data as shown in FIG. 18 is completed.
At that time, the processing in FIG. 21 proceeds from step F310 to F311.

In step F <b> 311, the layer processing unit 3 supplies the composite image data to the image output unit 4. The image output unit 4 outputs the supplied composite image data to the output device 40 as one frame constituting the motion transition video.
Note that the image output unit 4 combines one image according to the time required for the synthesis process, the input interval of the frame image data from the image input unit 10, or the like to express a moving image that moves slowly. The image data may be continuously output as a plurality of frames constituting the motion transition moving image.

In step F312, the process branches depending on whether or not the frame image data processed this time is a key frame. That is, the process branches depending on the determination result in step F303.
If the frame image data input this time is a key frame, in step F313, the combined image data generated this time is stored in the combined image update holding unit 5 in order to be used as the previous combined image data in the subsequent processing. Process to save.
The composite image update holding unit 5 uses the previous composite image data held at that time as the previous composite image data, and newly holds the composite image data supplied from the current image output unit 4 as the previous composite image data. The moving subject information of the key frame is also stored along with the information.

Then, the process returns to step F301 to proceed to the next frame image data processing.
In the process in which the processes in steps F301 to F313 described above are repeated for each input of frame image data, motion transition moving images as shown in FIGS. 11 to 20 are output.
That is, every time frame image data is input and image composition is performed, the composite image data is output as one frame constituting a moving image in step F311. For example, if the output device 40 is a monitor display device, the operation transition A video will be displayed.
For example, if the output device 40 is an image recording device, each frame constituting the motion transition moving image is recorded as moving image data.

  If it is determined in step F301 that all frames have been captured, the process proceeds to step F314, where the composition process is terminated, and the image output unit 4 terminates the output of the composite image data. Alternatively, when the output device 40 is a monitor display device, for example, the composite image data of the final frame as shown in FIG. 20 may be continuously output. In that case, on the monitor display device, the motion transition moving image of FIGS. 11 to 20 is completed and the last image of FIG. 20 is continued.

As described above, according to the present embodiment, it is possible to generate and output a motion transition moving image. And this motion transition moving image has a high visual effect by outputting the composite image data at each time point as a moving image, and the motion transition can be easily understood.
In particular, in the second generation processing example, efficient processing is realized by performing synthesis processing based on the above process determination. In other words, the composition processing in the processes P5 and P6 involving the composition of layers of three layers and four layers is a processing with a relatively large processing load, but these are executed only in the necessary pixel region. An area that does not require layer composition can be handled by simple copy processing such as processes P1, P2, and P3. As a result, the processing efficiency can be improved and the processing load can be reduced.
Of course, in the case of the second generation processing example, as in the case of the motion transition still image described in the first generation processing example, an accurate expression of the moving subject by expanding the operable range of the subject, and a large number of motions. It is possible to realize both a precise expression of motion transition in the time direction by arranging the subjects. For example, the golf club can be accurately represented without missing the tip of the golf club, and a large number of moving subject images can be arranged, and the movement transition can be expressed at fine intervals in time.
Further, the image processing apparatus 1 that generates and outputs such a motion transition moving image is applicable to various types of devices, as in the first generation processing example.

Here, the processing of steps F308 and F309 in FIG. 21 will be described.
The copy process in step F309 is a process of copying the last moving subject in the upper stage as the first moving subject in the lower stage, for example, as shown in FIG.
As shown in FIG. 19, a predetermined range at the upper right end of the composite image data (when shifting from left to right) is set as the lower copy range.
Then, the layer processing unit 3 performs process determination, and in step F308, for each column combined with the combined image data, whether the column (or pixel) corresponds to the lower copy range (that is, the uppermost last region). Determine whether or not. If it falls under the lower copy range, in step F309, the pixels in that column are copied to the corresponding position in the lower row.
By performing this process, as shown in FIG. 19, when the image in the lower copy range in the upper stage is synthesized, the same image is also synthesized in the lower stage at the same time.
On the display of the motion transition video, the same image appears simultaneously in the last moving subject in the upper part of FIG. 19 and the first moving subject in the lower part.

By performing such processing, the moving subject in the lower copy range can be accurately expressed without being partly interrupted. That is, depending on the state of the subject, the position of the last image in the upper stage may be interrupted, but such a subject is also appropriately displayed in the lower stage. As a result, the upper-stage operation transition and the lower-stage operation transition are easily transferred.

[4. Moving subject information]

As described above, in the first and second generation processing examples, processing for separating the frame image data into the moving subject layer and the remaining background layer is performed as the layer separation processing. It has been described that the moving subject information from the moving subject information generation unit 11 or the moving subject information input unit 12 is used in the layer separation.
In each of the above examples, the moving subject information is a binary example as shown in FIG. 4A, but multi-valued moving subject information of three or more values may be used.
For example, the moving subject information generation unit 11 or the moving subject information input unit 12 generates (or acquires) moving subject information having different values according to the depths of a plurality of moving subjects (the front-rear relationship in the depth direction on the image). You may do it.
In such a case, the layer separation unit 31 can separate the layer into the number of moving subjects + the number of backgrounds.

For example, FIG. 28 shows an example in which two persons (two moving subject images) exist in the frame image data.
FIG. 28A shows an example in which moving subject information is binary, and FIG. 28B shows an example in which moving subject information is multivalued (for example, ternary).
FIG. 28A shows an example in which the background is “0” and the moving subject is “1”. In this case, the two moving subjects are both “1”.
On the other hand, as shown in FIG. 28B, when the multivalue is used, the background can be expressed as “0”, one moving subject A as “1”, the other moving subject B as “2”, and the like.

Here, FIG. 29 shows a state in which two moving subjects overlap in the front-rear direction on the screen.
In this case, in the case of binary as shown in FIG. 29A, the depth of the two moving subjects cannot be distinguished on the moving subject information.
However, if it is multi-valued as shown in FIG. 29B, the moving subject information is information that can also determine the front-rear relationship of a plurality of moving subjects.

If multivalued moving subject information is used in this way, layer separation can be performed for each of a plurality of moving subject images. Then, it is possible to perform composition by making the priority order of one moving subject different from the other moving subject, or to generate a motion transition image by extracting only one moving subject.

[5. Application of first generation processing example to movie]

As described above, the motion transition moving images as shown in FIGS. 11 to 20 can also be generated by the first generation processing example.
FIG. 30 shows a processing example in the case of generating and outputting a motion transition video in accordance with the first generation processing example shown in FIG.
In addition, the same step number is attached | subjected about the process same as FIG. 9, and duplication description is avoided. In FIG. 30, steps F103A, F109A, F110A, F111A, and F112A, which are step numbers given “A”, are different from those in FIG.

Hereinafter, differences from the processing of FIG. 9 will be described.
In the process of FIG. 9, in step F103, the input image selection unit 2 selects only the key frame and uses it for the image composition process. However, in the case of generating the motion transition moving image, as described in the second generation processing example, both key frames and non-key frames are used for the synthesis processing. For this reason, step F103A is a process for determining whether the input frame image data is a key frame or a non-key frame (the same process as step F303 in FIG. 21).

  In steps F104 to F108 in FIG. 30, the input frame image data is combined. In step F109A, the image output unit 4 converts the generated combined image data into one frame constituting the motion transition video. The data is output to the output device 40 (the same processing as step F311 in FIG. 21).

  Steps F112A and F110A are processes for updating the previous composite image data. That is, this is processing having the same meaning as in steps F312 and F313 in FIG. 21. If the frame image data input this time is a key frame, the previous combined image data held in the combined image update holding unit 5 is updated. It is processing.

In the process in which the processes in steps F301 to F110 are repeated every time frame image data is input, the combined image data is output as one frame constituting a moving image in step F109A every time image combining is performed. Accordingly, for example, the motion transition moving image as illustrated in FIGS. 11 to 20 is output to the output device 40.
For example, when the output device 40 is a monitor display device, an operation transition moving image is displayed. For example, if the output device 40 is an image recording device, each frame constituting the motion transition moving image is recorded as moving image data.

If it is determined in step F101 that all frames have been captured, the process proceeds to step F111A, where the composition process is terminated, and the image output unit 4 terminates the output of the composite image data. Alternatively, when the output device 40 is a monitor display device, for example, the composite image data of the final frame as shown in FIG. 20 may be continuously output.
According to the processing of FIG. 30 described above, the motion transition moving image is generated and output in the first generation processing example.

[6. Application of second generation processing example to still image]

A motion transition still image can be generated and output in accordance with the second generation processing example.
FIG. 31 shows a processing example in the case of generating and outputting a motion transition still image according to the second generation processing example shown in FIG.
Note that the same steps as those in FIG. 21 are denoted by the same step numbers to avoid redundant description. In FIG. 31, steps F303A, F311A, F313A, and F314A, which are step numbers given “A”, are different processes from those in FIG.

Hereinafter, differences from the processing of FIG. 21 will be described.
In the case of generating the motion transition moving image in FIG. 21, both the key frame and the non-key frame are used for the synthesis process as described above. However, in the case of generating a motion transition still image, as described in the first generation processing example, the input image selection unit 2 selects only the key frame and uses it for the image composition processing.
Therefore, also in FIG. 31, the input image selection unit 2 selects only the key frame in step F303A and supplies it to the layer processing unit 3 (the same processing as step F103 in FIG. 9).
In steps F304 to F310, the composition processing described with reference to FIG. 21 is performed.

In step F311A, the image output unit 4 holds the generated composite image data (processing similar to step F109 in FIG. 9).
In step F313A, the image output unit 4 supplies the composite image data to the composite image update holding unit 5, and the composite image update holding unit 5 updates the previous composite image data and the previous composite image data (step F110 in FIG. 9). Processing with the same meaning).

The above-described processing of steps F301 to F313A is repeated for each key frame input, so that composite image data in the state shown in FIG. 2 is finally obtained.
At this time, the process proceeds from step F301 to step F314A, and the image output unit 4 outputs the composite image data held at that time (that is, the composite image data in the state of FIG. 2) as motion transition still image data. Output to.
With such a process of FIG. 31, one motion transition still image data is generated and output.

[7. program]

Although the above embodiment has been described as the image processing apparatus 1, the image processing apparatus 1 of the present invention can be applied to various devices that perform image processing in addition to a dedicated apparatus that generates a motion transition image. For example, an image reproducing device, an imaging device, a communication device, an image recording device, a game machine, a video editing machine, and the like are assumed.
Furthermore, it is naturally assumed that the image processing apparatus 1 is realized in a general-purpose personal computer or other information processing apparatus.
For example, by providing as an image processing application software a program for causing an arithmetic processing unit to execute the operation of each processing block in FIGS. 9 and 10, FIG. 21 and FIG. 22, FIG. Image processing can be realized.

That is, the program that realizes the image processing of the image processing apparatus 1 is a program that causes the arithmetic processing apparatus to execute the next layer separation step and composition processing step.
In the layer separation step, the moving subject layer and the background layer in the image are separated from the frame image data.
In the compositing process step, the moving subject layer of the latest frame image data that has been input is given first priority over a partial area of the existing composite image data at the time when the frame image data is input, Image composition processing including layer image composition processing in which the moving subject layer of the frame image data related to the processing is given second priority is performed.

  With such a program, the present invention can be executed in a personal computer, a cellular phone, a PDA (Personal Digital Assistant), and other various information processing apparatuses using image data.

Such a program can be recorded in advance in an HDD as a recording medium built in a device such as a personal computer, a ROM in a microcomputer having a CPU, a flash memory, or the like.
Alternatively, temporarily or permanently on a removable recording medium such as a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (Magnet optical) disk, DVD, Blu-ray disk, magnetic disk, semiconductor memory, memory card, etc. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.
In addition to installing the program from a removable recording medium to a personal computer or the like, the program can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

  DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 2 Input image selection part, 3 Layer processing part, 4 Image output part, 5 Composite image update holding | maintenance part, 10 Image input part, 11 Moving subject information generation part, 12 Moving subject information input part, 21 Key frame Determination unit, 22 coordinate calculation unit, 31 layer separation unit, 32 layer processing unit, 33 layer composition unit, 40 output device

Claims (10)

For frame image data, a layer separation unit that separates the moving subject layer and the background layer in the image,
In response to the input of the frame image data, an image composition process for synthesizing the latest input frame image data is performed on the existing composite image data, and the moving subject images of each frame image data are sequentially arranged in a predetermined direction. In addition to generating composite image data to be processed, in the image composition processing, composition that is a part of the existing composite image data that is an overlapping portion of the image of the latest frame image data and the image of the existing composite image data For the target area, the moving subject layer of the latest frame image data input is given priority first, and the moving subject layer of the frame image data related to the previous image composition processing is given priority second, and the latest input The background layer of frame image data has priority over the third, and the background layer of frame image data related to the previous image composition processing has priority over the fourth A synthesis processing unit that performs the layer image combining process,
An image processing apparatus comprising:   The image processing according to claim 1, further comprising: an image output unit configured to output combined image data obtained as a result of image combining processing performed on the predetermined number of frame image data by the combining processing unit as motion transition still image data. apparatus. An input image selection unit for selecting a key frame used for generating a composite image from sequentially input frame image data;
The image processing apparatus according to claim 2, wherein the composition processing unit generates motion transition still image data by performing image composition processing in response to input of frame image data selected as a key frame.   The composite image holding part which hold | maintains the composite image data produced | generated by the image composite process according to the input of frame image data in the said composite process part as said existing composite image data used by the next image composite process. The image processing apparatus according to 3.   The image processing apparatus according to claim 1, further comprising: an image output unit that continuously outputs the composite image data generated by the composite processing unit every time frame image data is input as motion transition moving image data. An input image selection unit that selects key frames used for generating a composite image from sequentially input frame image data;
A composite image in which the composite image data generated by the image composite processing according to the input of the frame image data that has been set as the key frame is held as the existing composite image data used in the next image composite processing in the composite processing unit. A holding part;
The image processing apparatus according to claim 5, further comprising:   Regardless of whether the frame is selected as a key frame, the above composition processing unit generates image data by performing image composition processing according to the input of each frame image data, thereby generating motion transition video data The image processing apparatus according to claim 6. For frame image data, a layer separation unit that separates the moving subject layer and the background layer in the image,
In response to the input of the frame image data, an image composition process for synthesizing the latest input frame image data is performed on the existing composite image data, and the moving subject images of each frame image data are sequentially arranged in a predetermined direction. In addition to generating composite image data to be processed, in the image composition processing, composition that is a part of the existing composite image data, which is an overlapping part of the image of the latest frame image data and the image of the existing composite image data For the target area, the moving subject layer of the latest frame image data input has priority first, and the moving subject layer of the frame image data related to the previous image composition processing has priority second, and the latest input The background layer of frame image data has priority over third, and the background layer of frame image data related to the previous image composition processing has priority over fourth. As appears Te, a synthesis processing unit that performs the layer image combining process,
An image processing apparatus comprising: For frame image data, a layer separation step for separating the moving subject layer and the background layer in the image,
In response to the input of the frame image data, an image composition process for synthesizing the latest input frame image data is performed on the existing composite image data, and the moving subject images of each frame image data are sequentially arranged in a predetermined direction. In addition to generating composite image data to be processed, in the image composition processing, composition that is a part of the existing composite image data, which is an overlapping portion of the image of the latest frame image data and the image of the existing composite image data For the target area, the moving subject layer of the latest frame image data input is given priority first, and the moving subject layer of the frame image data related to the previous image composition processing is given priority second, and the latest input In the state where the background layer of the frame image data has priority over the third, and the background layer of the frame image data related to the previous image composition processing has priority over the fourth And combining processing step of performing the ear image synthesis processing,
An image processing method. For frame image data, a layer separation step for separating the moving subject layer and the background layer in the image,
In response to the input of the frame image data, an image composition process for synthesizing the latest input frame image data is performed on the existing composite image data, and the moving subject images of each frame image data are sequentially arranged in a predetermined direction. In addition to generating composite image data to be processed, in the image composition processing, composition that is a part of the existing composite image data, which is an overlapping portion of the image of the latest frame image data and the image of the existing composite image data For the target area, the moving subject layer of the latest frame image data input is given priority first, and the moving subject layer of the frame image data related to the previous image composition processing is given priority second, and the latest input In the state where the background layer of the frame image data has priority over the third, and the background layer of the frame image data related to the previous image composition processing has priority over the fourth And combining processing step of performing the ear image synthesis processing,
A program that causes an arithmetic processing unit to execute.

Images