JP2003304444A - Apparatus and method for compositing image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image composition apparatus and an image composition method which enable an effective representation of the movement of a subject accompanied with movement in the direction of a depth. <P>SOLUTION: This image composition apparatus 100 performs image composition of the subject accompanied with its movement in the direction of a depth. The apparatus 100 is provided with an image memory 2 for storing image data, an image extraction portion 3 which reads the image data out of the image memory 2 and extracts an image of the subject, and an image composition portion 4 which performs image composition on the basis of the image data stored in the image memory 2 and/or the extracted image extracted by the extraction means. The composition portion 4 performs upside lapping processing which laps extracted images above successively when the images are ones before a reference image determined separately, and performs underside lapping processing which laps extracted images below successively when the images are ones after the reference image. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus and an image synthesizing method for synthesizing images, and more particularly to image synthesizing used in television program production, movie production and the like.

[0002]

2. Description of the Related Art Conventionally, slow-motion reproduction has been widely used in sports broadcasting and the like in order to express the movement of a player who is a subject in a manner easily understandable to a viewer. Also,
A technique called multi-motion is also used, in which the movement of the player is expressed like a stroboscopic photograph, and the still images are sequentially overwritten on the background image to synthesize them.

[0003]

In sports broadcasting, slow-motion reproduction that slowly expresses the movement of a fast player is easy and effective for the viewer. On the other hand, there is a drawback in that it is difficult to capture the trajectory of the player and the transition of the form due to the slow movement.

[0004] Multi-motion has an advantage that the process of movement can be easily understood by the afterimage effect because the images of still players are superimposed and synthesized one after another. However, because images are sequentially superimposed according to the time and history, when displaying a baseball or golf swing, etc., there is no significant movement in the position of the subject, so even if there is a scene of interest such as a hitting point, it will be overwritten. , There was a drawback that you could not see this attention scene.
In addition, the display by multi-motion has a drawback in terms of visual expression that it is flat and cannot give a stereoscopic effect to the viewer.

The present invention has been made to solve such a problem, and provides an image synthesizing apparatus and an image synthesizing method capable of effectively expressing a motion of a subject accompanied by a motion in the depth direction. The purpose is to do.

[0006]

An image synthesizing apparatus according to the present invention is an image synthesizing apparatus for synthesizing an image of a subject accompanied by a movement in a depth direction, the image memory storing image data, and the image memory. Read more image data,
Image extracting means for extracting an image of a subject, image combining means for performing image combination based on the image data stored in the image memory and / or the extracted image extracted by the image extracting means, and a reference for determining a reference image Image determining means, and the image synthesizing means performs an overlay process in which an extracted image is sequentially superimposed on an image before the reference image determined by the reference image determining means, and after the reference image. The image is subjected to a lower superposition process in which the extracted images are sequentially superposed downward. With such a configuration, since the reference image is displayed on the top, if the attention scene is selected as the reference image, the attention scene remains without being overwritten and the progress can be effectively confirmed. it can.

In a preferred embodiment, the lower superimposing process combines the first composite image generated by the upper superimposing process with an image after the reference image, and the first composite image is combined with the first composite image. This is realized by overwriting the composite image obtained by extracting the subject of the composite image.

Further, the lower overlay processing is performed by removing a portion of the image obtained by extracting the subject of the first composite image generated by the upper overlay processing from the image obtained by extracting the subject of the image after the reference image. It may be realized by synthesizing the second synthesized image described above with the first synthesized image.

Furthermore, in the lower superimposing process, the object of the image after the reference image is extracted by the image extracting means so that the extraction range of the object of the first composite image generated by the upper superimposing process is not extracted. It may be realized by combining the extracted image thus extracted with the first combined image.

The reference image determining means may determine the reference image based on the information for specifying the reference image input by the input means.

The reference image determining means may determine the reference image based on the voice information detected by the voice detecting means. With such a configuration, the reference image can be automatically determined, and the combining process can be easily performed.

The reference image determining means may determine the reference image based on the position information of all or part of the subject detected by the position detecting means. With such a configuration, the reference image can be automatically determined, and the combining process can be easily performed. Further, according to such means, it is possible to automatically select, as the reference image, an image that specifies the state in which all or part of the subject is closest to the shooting position.

Further, the reference image determining means may determine the reference image based on the position information of all or a part of the subject detected by the depth detecting means. With such a configuration, the reference image can be automatically determined, and the combining process can be easily performed. Further, according to such means, it is possible to automatically select, as the reference image, an image that specifies the state in which all or part of the subject is closest to the shooting position.

In particular, it is preferable that the reference image determining means selects, as the reference image, an image that specifies the state in which all or part of the subject is closest to the photographing position. With such a configuration, the displayed image is visually three-dimensional, and thus a powerful expression effect can be obtained.

Another image synthesizing apparatus according to the present invention is an image synthesizing apparatus for synthesizing an image of a subject that moves in the depth direction, and stores an image memory for storing image data and image data from the image memory. An image extracting means for reading out and extracting an image of the subject, an image synthesizing means for synthesizing an image based on the image data stored in the image memory and / or the extracted image extracted by the image extracting means, and a reference image are determined. And a reference image determining means for performing the lower overlay processing in which the extracted images are sequentially overlaid on the image before the reference image determined by the reference image determining means. The subsequent image is subjected to an overlay process in which the extracted images are sequentially overlaid. With such a configuration, since the reference image is displayed on the top, if the attention scene is selected as the reference image, the attention scene remains without being overwritten and the progress can be effectively confirmed. it can.
In particular, this is an effective process when the attention scene is at the back.

On the other hand, the image synthesizing method according to the present invention is an image synthesizing method for synthesizing an image of a subject that moves in the depth direction, and includes a step of determining a reference image and a step of determining the reference image before the determined reference image. The method has a step of performing an upper superimposing process of sequentially superimposing the extracted images on the upper side of the image, and a step of performing a lower superimposing process of sequentially superimposing the extracted images on the lower side of the image after the reference image. is there. With such a method, the reference image is displayed at the top, so if the attention scene is selected as the reference image, the attention scene remains without being overwritten and the progress can be effectively confirmed. it can.

Another image synthesizing method according to the present invention is an image synthesizing method for synthesizing an image of a subject that moves in the depth direction, which comprises a step of determining a reference image and an image before the reference image. On the other hand, the method has a step of performing a lower superimposing process in which the extracted images are sequentially superposed downward, and a step of performing an upper superimposing process in which the extracted images are sequentially superposed on an image after the determined reference image. Is. With such a method, the reference image is displayed at the top, so if the attention scene is selected as the reference image, the attention scene remains without being overwritten and the progress can be effectively confirmed. it can. In particular, this is an effective process when the attention scene is at the back.

An image synthesizing program according to the present invention is an image synthesizing program for synthesizing an image of a subject with movement in the depth direction, and a step of determining a reference image to a computer, and the determined reference image. A step of performing an upper superimposing process in which the extracted images are sequentially superposed on the previous image, and a step of performing a lower superimposing process in which the extracted images are sequentially superposed on the subsequent image of the reference image. It is what is executed. By executing such a program on the computer, the reference image is displayed at the top, so if you select the scene of interest as the reference image, the scene of interest will not be overwritten and will remain until the end, and the process will be effective. Can be confirmed.

Another image synthesizing program according to the present invention is an image synthesizing program for synthesizing an image of a subject accompanied by a movement in the depth direction.
A step of determining a reference image, a step of performing a superimposing process of sequentially superimposing an extracted image on a lower side of an image before the reference image, and a step of sequentially increasing an upper side of an image after the determined reference image. The step of performing the superimposing process in which the extracted images are superposed on is executed. By executing such a program on a computer,
Since the reference image is displayed at the top, if the attention scene is selected as the reference image, the attention scene remains to the end without being overwritten, and its progress can be effectively confirmed. In particular, this is an effective process when the attention scene is at the back.

[0020]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiment 1 of the Invention FIG. 1 shows the configuration of an image composition apparatus according to the first embodiment of the present invention. The image synthesizing apparatus 100 is connected to the photographing camera 1 by a cable or a wireless line.

The photographing camera 1 photographs a moving subject and generates an image data signal. In this example, it is mainly used for photographing a subject having a movement in the depth direction, that is, on a line connecting the subject from the photographing camera 1 and in a direction in which they are separated from each other. For example, a swinging motion of a baseball batter, a pitching motion of a pitcher, a swinging motion of golf, a shooting motion of soccer, a rotating motion of figure skating, a throwing motion of judo, and a cornering motion of a car in a car race. The photographing camera 1 inputs the generated image data signal to the image synthesizing apparatus 100 via a cable. The image data signal from the photographing camera 1 may be temporarily stored in a storage medium such as a magnetic tape, and the image synthesizing apparatus 100 may read the image data signal from this storage medium.

The image synthesizing apparatus 100 includes, for example, a CPU,
The hardware is configured by a dedicated computer having a ROM, a RAM, a hard disk, a display, a keyboard, and the like. It is also possible to configure the image synthesizing device 100 by installing an application program in a so-called general-purpose computer. further,
The image synthesizing apparatus 100 includes a buffer memory,
Composite image 5 with background, composite image 6 without background, extracted image 7
To store.

The image synthesizing apparatus 100, as shown in FIG. 1, includes an image memory 2, an image extracting section 3, and an image synthesizing section 4.

The image memory 2 is a storage means for accumulating image data signals input from the photographing camera 1. In the image memory 2, the image data to be subjected to the image composition processing according to the present invention is stored frame by frame in association with time information. The number of frames of this image data is appropriately selected according to the speed and the operation time of the subject to be image-synthesized.

The image extracting section 3 inputs image data from the image memory 2, extracts an image of a subject, and separates it from a background image. For example, in the case of an image of a swing motion of a baseball batter, only the part of the batter that is the subject is extracted. The extraction of the image information by the image extraction unit 3 can be realized by the operator designating the extracted image on the screen, or can be automatically generated by the following method.

(1) Image processing is performed by calculating the difference from the background.

(2) The position of the extraction target is detected by using the detector to extract the specific portion in the image.

(3) A portion other than a specific color is extracted using the color information.

The image extracted by the image extraction unit 3 is
The extracted image 7 is stored in the buffer memory.

The image synthesizing unit 4 inputs the image data stored in the image memory 2 and the extracted image data generated by the image extracting unit 3, and performs an image synthesizing process. For example,
The image synthesizing unit 4 performs a process of superposing the extracted image 7 input from the image extracting unit 3 on the image input from the image memory 2 to generate a synthetic image 5 with a background.
Further, the image synthesizing unit 4 can also perform a process of superimposing a plurality of extracted images input from the image extracting unit 3 to generate a synthetic image 6 having no background.

Here, an example of a method of synthesizing the image B located after the synthesized image A will be described. First, the extracted image of the image B is superposed on the synthetic image A having the background. After that, the composite image A having no background is further superposed on the superposed composite image. By such a combining process, the image B can be displayed so as to be located below the image A.

FIG. 2 shows an image before combining and an image after combining in order to explain the image combining processing of the image combining apparatus according to the first embodiment of the present invention. Images 2a to 2f are images before combining, and image 20 is an image after combining.

Images 2a to 2f are images in which swing motions of a baseball batter are photographed at regular intervals and are simply displayed. The image 2a is arranged in time series starting from the first, and the last is the image 2f. The background of each image is shaded. In the present invention, a reference image is determined, and a combining process for an image and a reference image temporally earlier than the reference image and a combining process for an image temporally later than the reference image are changed. It is characterized by In the first embodiment of the present invention, an image and a reference image temporally before the reference image are subjected to an overlay process on the subject, and an image temporally after the reference image is overlayed on the subject. Perform processing. There are various methods for determining the reference image, which will be described in detail in the fourth and subsequent embodiments of the invention.

In the example shown in FIG. 2, the image 2c which is the scene of interest showing the hitting point is used as the reference image. In selecting the reference image, when preferentially selecting an image that is desired to be prevented from being overwritten by another image in this way, the subject or a part of the subject (bat There are various cases such as the case of selecting an image in which () etc. are located.

In the first embodiment of the present invention, the reference image 2
The superimposing process is performed on the subjects of the image 2a, the image 2b, and the reference image 2c before c. More specifically, the image of the batter of image 2a (including the bat, the same applies hereinafter) is overlaid so that the image of the batter of image 2b is on top. Further, the image of the batter of the image 2c is overlaid on this composite image so that it is on the top. Then, the subject of the image 2d, the image 2e, and the image 2f after the reference image 2c is subjected to the lower superposition processing. More specifically, the image 2 is added to the composite image on which the images 2a to 2c are overlaid.
The images of the batters in d are overlaid so that they face down. further,
The image of the batter of the image 2e is superimposed on the composite image so that the image is on the bottom. Finally, for that composite image,
The image of the batter in the image 2f is overlaid so as to face downward. The composite image generated in this way is the image 20.

In the image 20, the reference image having the target scene is displayed at the top, so that the target scene remains without being overwritten and the progress can be confirmed. When the scene of interest is on the side of the camera for shooting, the displayed image is visually three-dimensional, so that a powerful expression effect can be obtained.

Next, the image synthesizing process will be specifically described with reference to FIG. The image example is the same as that shown in FIG.

First, the images 2a, 2b,
2c is read out and the images 2a, 2
An image 21 is generated by synthesizing b and 2c so that they are superposed in order. Since this synthesizing method is a well-known method that has been performed in the conventional multi-motion, description thereof will be omitted. Next, the image extracting unit 3 reads the image 2d from the image memory 2, extracts only the image of the subject, and generates the extracted image 22. The extracted image 22 is an image without a background.

Next, the composite image 21 and the image 22 are subjected to a composition process 23 to obtain an image 24. In addition, the image extraction unit 3
Generates an extracted image 25 by extracting only the image of the subject from the composite image 21. This extracted image 25 is an image without a background. Then, the combined image 24 and the extracted image 25 are combined to generate a combined image 27. Similarly, for this composite image 27, image 2e and image 2
By performing a synthesis process on f, the synthesized image 2 shown in FIG.
You can get 0.

Second Embodiment of the Invention FIG. 4 shows the configuration of the image composition apparatus according to the second embodiment of the present invention. The photographing camera 31 is the photographing camera 1 shown in FIG. 1, the image memory 32 is the image memory 2 shown in FIG.
The image extracting unit 3 shown in FIG. 1 and the image synthesizing unit 34 have the same functions as the image synthesizing unit 4 shown in FIG. 1, respectively.

The image synthesizing apparatus 100 according to the second embodiment of the present invention includes an extraction area memory 36 as shown in FIG.

The image synthesizing unit 34 according to the second embodiment of the present invention superimposes the extracted image 37 extracted by the image extracting unit 33 on the image data stored in the image memory 32 to synthesize the image with the background. The image 35 is generated. Further, the image synthesizing unit 34 appropriately fetches and stores information indicating the range of the image extracted by the image extracting unit 33 in the extraction area memory 36.

Here, an example of a method of synthesizing the image B located after the synthesized image A will be described. First, the range portion of the extracted image of the composite image A stored in the extraction area memory 36 is removed from the extracted image 37 of the image B. Then, the composite image A is superimposed on this image. By such a combining process, the image B can be displayed so as to be located below the image A.

Next, with reference to FIG. 5, the image combining processing according to the second embodiment of the present invention will be specifically described. The image example is the same as that shown in FIG.

First, the images 2a and 2 are read from the image memory 32.
b, 2c are read out, and the image combining unit 34
An image 41 is generated by synthesizing a, 2b, and 2c so that they are sequentially overlaid. Next, the image extracting unit 33 extracts only the image of the subject from the image 41, generates the extracted image 44, and stores it in the extraction area memory 36. The extracted image 44 is an image without a background.

Further, the image 2d is read out from the image memory 32, only the image of the subject is extracted by the image extracting section 33, the extracted image 43 is generated and stored in the extraction area memory 36. The extracted image 43 is an image without a background.

Next, the extraction image 43 and the extraction image 44 are read from the extraction area memory 36, and the image synthesizing unit 34 performs a process of removing the extraction image 44 from the extraction image 43. An image 46 is obtained by this processing. continue,
By combining the image 46 and the image 41, a combined image 42 is obtained. Similarly, by synthesizing the image 2e and the image 2f with respect to the synthetic image 42,
The composite image 20 shown in FIG. 2 can be obtained.

Third Embodiment of the Invention FIG. 6 shows the configuration of an image synthesizing apparatus according to the third embodiment of the present invention. The photographing camera 51 is the photographing camera 1 shown in FIG. 1, the image memory 52 is the image memory 2 shown in FIG.
The image extracting section 3 and the image synthesizing section 54 shown in FIG. 1 have the same functions as the image synthesizing section 4 shown in FIG.

The image synthesizing apparatus 100 according to the third embodiment of the present invention includes an extraction area memory 56 as shown in FIG.

The image synthesizing unit 54 according to the third embodiment of the present invention superimposes the extracted image 57 extracted by the image extracting unit 53 on the image data stored in the image memory 52 to synthesize the image with the background. The image 55 is generated. Further, the image synthesizing unit 54 appropriately fetches and stores information indicating the range of the image extracted by the image extracting unit 53 in the extraction area memory 56.

Here, an example of a method of synthesizing the image B that is behind the synthesized image A will be described. First, when the image extraction unit 53 generates an extracted image of the image B, the composite image A stored in the extraction area memory 56
The extraction range of is processed so as not to be extracted. Then, the composite image A is superimposed on this image. By such a combining process, the image B can be displayed so as to be located below the image A.

Next, the image synthesizing process according to the third embodiment of the present invention will be specifically described with reference to FIG. The image example is the same as that shown in FIG.

First, the images 2a and 2 are read from the image memory 52.
b, 2c are read out, and the image combining unit 54
An image 61 is generated by synthesizing a, 2b, and 2c so that they are sequentially overlaid. Next, the image extracting unit 53 extracts only the image of the subject from the image 61, generates the extracted image 62, and stores the information of the extraction range in the extraction area memory 56.

Further, the image 2d is read from the image memory 52, and only the image of the subject is extracted by the image extracting section 53. At this time, based on the extraction range information of the extracted image 62 stored in the extraction area memory 56, this range is extracted. Image is processed so as not to be extracted. In this way, the extracted image 6
5 is generated. This extracted image 65 is an image without a background.

Next, the image combining section 54 combines 66 the extracted image 65 and the combined image 61 to obtain a combined image 67. Similarly, the composite image 20 shown in FIG. 2 can be obtained by combining the composite image 67 with the images 2e and 2f.

Fourth Embodiment of the Invention Fourth Embodiment of the Invention
Relates to the process of determining the reference image. As shown in FIG. 8, in the fourth embodiment of the invention, the reference image determination device 73 is connected to the image synthesis device 72. The reference image determining device 73 may be incorporated in the image synthesizing device 72. That is, the reference image determination device 7
3 can be realized as one function of the image synthesizing device 72.

The photographing camera 71 corresponds to the photographing camera 1 shown in FIG.
2 corresponds to the image compositing apparatus 100 shown in FIG.

The reference image determining device 73 includes a keyboard 7
4, the microphone 75 and the camera 76 can be connected. The reference image determining device 73 according to the fourth embodiment of the present invention is described as being capable of determining the reference image according to various modes, but the present invention is not limited to this, and the reference image can be determined according to one or any combination of modes. The image may be determined.

The reference image determining device 73 is the image synthesizing device 7
If it is configured separately from 2, CPU, RO
It is composed of a control circuit such as M and RAM, a memory, a display and the like. An algorithm for determining a reference image and a software program for performing processing are installed in the memory.

The keyboard 74 is used as an input means for the operator to directly input the determination of the reference image while checking the image on the screen. As this input means,
In addition to the keyboard 74, for example, a pointing device such as a mouse or a touch pad is included. At this time,
The reference image determination device 73 extracts information specifying the reference image based on an input signal from an input unit such as a keyboard 74 and outputs the information to the image synthesis device 72.

The microphone 75 is used as a voice detecting means for detecting a voice generated by the motion of the subject. For example, in the case of a swing motion of a baseball batter, a hitting sound generated when the ball collides with a bat is detected. At this time, the reference image determination device 73 inputs the voice information detected by the microphone 75, and first extracts a sound to be specified, for example, a hitting sound from the voice information. The extraction of the sound can be realized by, for example, setting the data of the strength and frequency of the sound to be specified in advance and comparing the data with the input voice information. When the sound to be specified is extracted from the voice information, the time information when the sound is detected is generated. Based on this time information, the time when the sound is detected from the image memory of the image synthesizing device 72, or an image captured before or after a certain period of time after that is determined as the reference image. The algorithm for specifying the reference image based on the time information is configured to be preset by the operator.

The camera 76 is a camera provided at a position different from the photographing camera 81, and is a camera for deriving the time information of the reference image by photographing the motion of the subject. For example, in the case of a swing motion of a baseball batter, a camera attached to the ceiling of the dome stadium is applicable. Since this camera can photograph the batter from above, the position of the bat can be clearly photographed. Then, the time information when the bat passes through the area of the hitting point is detected. The reference image is determined based on this time information. The algorithm for specifying the reference image based on the time information is configured to be preset by the operator.

Fifth Embodiment of the Invention In the fifth embodiment of the present invention, as shown in FIG. 9, the position detection sensor 83 is connected to the image synthesizing device 82. Here, the photographing camera 81 corresponds to the photographing camera 1 shown in FIG. 1, and the image synthesizing device 82 corresponds to the image synthesizing device 100 shown in FIG.

The position detection sensor 83 acquires position information of a moving subject. The image composition device 82 acquires the position information of the subject from the position detection sensor 83, and determines the order in which the extracted images are superimposed so that the position of the subject and the composite image do not conflict. The position detection sensor 83 is, for example, an optical sensor and is installed at a predetermined position.
For example, when shooting a golf swing motion, a golf club is installed at a position where it hits the ball. Based on the detection signal of the position detection sensor 83, it is possible to obtain time information when the golf club passes the position where the golf club hits the ball. The reference image is determined based on this time information. The algorithm for specifying the reference image based on the time information is configured to be preset by the operator.

Sixth Embodiment of the Invention In the sixth embodiment of the present invention, as shown in FIG. 10, a depth detection device 93 is connected to the image synthesizing device 92. The depth detecting device 93 may be provided in the image synthesizing device 92. here,
The photographing camera 91 corresponds to the photographing camera 1 shown in FIG. 1, and the image synthesizing device 92 corresponds to the image synthesizing device 100 shown in FIG.

The depth detection device 93 obtains position information in a depth direction of a specific portion of an operating subject, that is, in a line direction connecting the photographing camera 91 and the subject. For example, in the case of a swinging motion of a baseball batter, the position information is the tip portion of the bat. This positional information can be detected by, for example, Colo.
r Digiclops (manufactured by Point Gray) 3D image capturing device can be used.

The image synthesizing device 92 detects, based on the position information in the depth direction detected by the depth detecting device 93, the time information at which the specific location and the photographing camera 91 are located closest to each other, and the specific location at this position. The image in which the position is located is taken as the reference image. Then, the images are combined by the method as described in the above embodiments of the invention so that the reference image is displayed at the top.

When a part of the subject does not move in the depth direction but all of the subject moves in the depth direction, the position information of the entire subject in the depth direction is detected,
The reference image may be determined based on this position information. Specifically, an image in which all of the subjects are closest to the camera 91 is used as a reference image. Then, the images are combined by the method as described in the above embodiments of the invention so that the reference image is displayed at the top.

In this way, the photographed image of the specific portion of the subject or the state in which the whole of the subject is positioned closest to the photographing camera is used as the reference image, and the reference image is displayed at the top. By doing so, the subject can be displayed three-dimensionally.

Incidentally, the photographing camera 91 and the depth detecting device 93
May be implemented by a single camera.

Seventh Embodiment of the Invention In the image synthesizing device according to the seventh embodiment of the present invention, a depth detecting device 93 as shown in FIG. 10 is connected to the image synthesizing device 92 as in the sixth embodiment of the present invention. This depth detection device 93
May be provided in the image synthesizing device 92. In the seventh embodiment of the present invention, in particular, the motion of the subject detected by the depth detection device 93 is further analyzed, and the position of the specific position at the start and end of the motion is compared with the position of the specific position at the reference time. By doing so, the composition process is changed.

FIG. 11 shows a processing flow of the image synthesizing apparatus according to the seventh embodiment of the present invention. First, the subject is photographed by the photographing camera 91 (S101). Next, the image synthesizing device inputs image data from the photographing camera 91 and stores it in the image memory (S102). A reference image is determined based on the image data stored in this image memory (S103). The reference image may be automatically determined using the depth detection device 93 described in the sixth embodiment of the invention, or may be determined by the method described in the fourth and fifth embodiments of the invention. Good.

Next, whether the position of the specific portion at the start of the operation and at the end of the operation is deeper than the position of the specific portion at the reference time when the reference image is picked up, that is, the photographing camera 91.
Depth detection device 93
(S104).

As a result of the determination, if it is determined that the position of the specific portion at the start and end of the operation is behind the position of the specific portion at the reference time when the reference image is captured, up to the reference image. , And upper overlay processing, and after the reference image, lower overlay processing is performed (S105).

As a result of the determination, when it is determined that the position of the specific portion at the start and end of the operation is not deeper than the position of the specific portion at the reference time when the reference image is captured, up to the reference image, Lower overlay processing is performed, and after the reference image, upper overlay processing is performed (S106).

For example, in the case of a swinging motion of a baseball batter, in the case of shooting from the front as in the case shown in FIG. 2, if a specific portion is a bat, step S105 is performed. On the other hand, when the image is taken from behind the batter, step S106 is performed.

In this example, the case where the depth detection device 93 automatically determines is explained, but step S1
The operator may be able to set either the processing of 05 or the processing of step S106 using a keyboard or the like.

Other Embodiments. It should be noted that the image synthesizing apparatus described above does not necessarily have to be configured by a single computer, but may be configured by a plurality of computers. Further, the image synthesizing device, the photographing camera, and the reference image determining device may be configured by a single device.
For example, the image synthesizing device or the reference image determining device may be incorporated in a home or business camera that captures a moving image. Further, the image synthesizing device may be incorporated in a home digital video reproducing device or a game machine. It is advisable to install this image synthesizing device together with a photographing camera in a so-called batting center, golf driving range, game center, or the like.

Further, an image synthesizing device is provided on the Internet server, image data is transmitted from the client computer via the Internet, and the image synthesizing device according to the present invention performs the image synthesizing process according to the present invention. You may make it transmit. At this time, the reference image may be designated by the client computer.

Although some examples of determining the reference image have been described, it is not always necessary to determine the reference image itself, and information related to the reference image may be determined. For example, the reference image can be indirectly determined by determining the reference time.

In the above example, the background image is incorporated in the finally obtained image, but it is not always necessary to incorporate it. Further, other images may be incorporated instead of the background image at the time of shooting.

It is possible to easily change the reference image by displaying the synthesized image created by the image synthesizing device according to the present invention and designating the subject to be the reference image among the subjects on the screen with a pointer or the like. It may be.

In the above example, when the image synthesizing device is constituted by a general-purpose computer, an application program for causing the general-purpose computer to function as the image synthesizing device is installed. This application program can be stored in various types of storage media, and can be transmitted via a communication medium. Here, the storage medium is, for example, a flexible disk, a hard disk, a magnetic disk, a magneto-optical disk, or a CD-RO.
M, DVD, ROM cartridge, RAM memory cartridge with battery backup, flash memory cartridge, non-volatile RAM cartridge, etc. are included. The communication medium includes a wired communication medium such as a telephone line, a wireless communication medium such as a microwave line, and the Internet.

[0084]

As described above, according to the present invention, it is possible to provide an image synthesizing apparatus and an image synthesizing method capable of effectively expressing a motion of a subject accompanied by a motion in the depth direction.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an image synthesizing apparatus according to the present invention.

FIG. 2 is a diagram showing an image before composition and an image after composition.

FIG. 3 is a diagram showing a synthesizing process by the image synthesizing apparatus according to the present invention.

FIG. 4 is a block diagram showing a configuration of an image synthesizing apparatus according to the present invention.

FIG. 5 is a diagram showing a synthesizing process by the image synthesizing apparatus according to the present invention.

FIG. 6 is a block diagram showing a configuration of an image synthesizing apparatus according to the present invention.

FIG. 7 is a diagram showing a compositing process by the image compositing apparatus according to the present invention.

FIG. 8 is a block diagram showing a configuration of an image synthesizing apparatus according to the present invention.

FIG. 9 is a block diagram showing a configuration of an image synthesizing apparatus according to the present invention.

FIG. 10 is a block diagram showing a configuration of an image synthesizing apparatus according to the present invention.

FIG. 11 is a flowchart showing a compositing process by the image compositing apparatus according to the present invention.

[Explanation of symbols]

1 Camera for photographing 100 Image synthesizer 2 image memory 3 image extractor 4 image synthesizer 73 Reference image determination device

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Hiroyuki Okamoto             1-4-6 Shimoida, Kofu City, Yamanashi Prefecture             2nd floor, Inc., Inc. (72) Inventor Daiichiro Kato             2-2-1 Jinnan 2-chome, Shibuya-ku, Tokyo             Sending Association Broadcast Center (72) Inventor Hiroyasu Masuda             2-2-1 Jinnan 2-chome, Shibuya-ku, Tokyo             Sending Association Broadcast Center (72) Inventor Makoto Ikeda             2-2-1 Jinnan 2-chome, Shibuya-ku, Tokyo             Sending Association Broadcast Center (72) Inventor Yutaka Iwai             2-2-1 Jinnan 2-chome, Shibuya-ku, Tokyo             Sending Association Broadcast Center (72) Inventor Kenichi Hashimoto             2-2-1 Jinnan 2-chome, Shibuya-ku, Tokyo             Sending Association Broadcast Center (72) Inventor Hiroyuki Tamaki             2-2-1 Jinnan 2-chome, Shibuya-ku, Tokyo             Sending Association Broadcast Center F term (reference) 5B057 AA20 BA02 BA11 CA12 CA16                       CB12 CB16 CC01 CE08                 5C023 AA06 AA10 AA16 BA01 BA11                       CA01 DA04 EA03

Claims (14)

[Claims] 1. An image synthesizing device for synthesizing an image of a subject with movement in the depth direction, comprising: an image memory for storing image data; and an image for reading the image data from the image memory and extracting an image of the subject. An extracting unit, an image combining unit that performs image combining based on the image data stored in the image memory and / or the extracted image extracted by the image extracting unit; and a reference image determining unit that determines a reference image, The image synthesizing unit performs an upper superimposing process in which the extracted image is sequentially superposed on the image before the reference image determined by the reference image determining unit, and sequentially on the images after the reference image. An image synthesizing device that performs lower overlay processing in which extracted images are overlaid below. 2. The lower superimposing process combines the first composite image generated by the upper superimposing process with an image after the reference image, and the subject of the first composite image is compared with the composite image. The image synthesizing apparatus according to claim 1, which is realized by overwriting the extracted synthetic image. 3. The lower superposition processing is generated by removing a portion of the image obtained by extracting the subject of the first composite image generated by the upper superposition processing from the image obtained by extracting the subject of the image after the reference image. The second composite image is
The image synthesizing apparatus according to claim 1, wherein the image synthesizing device is realized by synthesizing the image into the synthetic image. 4. The subject of the image after the reference image is extracted by the image extracting means in the lower superimposing process so that the subject extraction range of the first composite image generated by the upper superimposing process is not extracted. The method is realized by synthesizing an extracted image with the first synthetic image.
The image synthesizing apparatus described. 5. The reference image determining means determines the reference image based on the information for specifying the reference image input by the input means.
Alternatively, the image synthesizing apparatus according to item 4. 6. The image synthesizing apparatus according to claim 1, wherein the reference image determining means determines the reference image based on the voice information detected by the voice detecting means. 7. The reference image determining means determines the reference image based on the position information of all or a part of the subject detected by the position detecting means.
The image synthesizing device according to 2, 3, or 4. 8. The reference image determining means determines the reference image based on position information of all or a part of the subject detected by the depth detecting means.
The image synthesizing device according to 2, 3, or 4. 9. The reference image determining means selects, as a reference image, an image that specifies a state in which all or part of a subject is closest to a shooting position.
Alternatively, the image synthesizing apparatus according to item 4. 10. An image synthesizing device for synthesizing an image of a subject accompanied by movement in the depth direction, comprising: an image memory for storing image data; and an image for reading the image data from the image memory and extracting an image of the subject. An extracting unit, an image combining unit that performs image combining based on the image data stored in the image memory and / or the extracted image extracted by the image extracting unit; and a reference image determining unit that determines a reference image, The image synthesizing unit performs a lower superimposing process in which the extracted images are sequentially superposed downward on the image before the reference image determined by the reference image determining unit, and for the image after the reference image, An image synthesizing device that performs an overlay process in which extracted images are sequentially overlaid. 11. An image synthesizing method for synthesizing an image of a subject accompanied by a movement in the depth direction, which comprises a step of determining a reference image, and a step of sequentially extracting the reference image before the determined reference image upward. An image synthesizing method comprising: a step of performing an upper superimposing process in which images are superposed, and a step of performing a lower superimposing process in which extracted images are sequentially superposed on an image after the reference image. 12. An image synthesizing method for synthesizing an image of a subject with movement in the depth direction, the method comprising: determining a reference image; and extracting images sequentially below the image before the reference image. An image synthesizing method comprising: a step of performing a superimposing lower superimposing process; and a step of performing an upper superimposing process of sequentially superimposing an extracted image on an image after a determined reference image. 13. An image synthesizing program for synthesizing an image of a subject accompanied by movement in the depth direction, the image synthesizing program causing a computer to determine a reference image, and determining the reference image. Performing an upper superimposing process in which the extracted image is sequentially superposed on the previous image, and a lower superimposing process in which the extracted image is sequentially superposed in the lower image on the subsequent image of the reference image. Image synthesis program to let you. 14. An image synthesizing program for synthesizing an image of a subject with movement in the depth direction, the image synthesizing program causing a computer to determine a reference image; Executes a step of performing a lower overlay process in which the extracted images are sequentially overlaid on the lower image, and a step of performing an upper overlay process in which the extracted images are sequentially overlaid on the image after the determined reference image. Image synthesis program to let you.