JPH07212653A - Picture processing unit - Google Patents

Abstract

PURPOSE:To synthesize a CG picture and a real pickup picture in real time synchronously between the CG picture and the real pickup picture by generating the CG picture in matching with the real pickup picture. CONSTITUTION:The processing unit is provided with an image pickup position, detection section 102 detecting a position direction of an image pickup section 101 outputting real pickup picture element data 108 and a pickup condition 109 picking up an object, an image generating section 103 receiving the image pickup condition and the position direction to generate an image based on the shape of a 3-dimension geometrical model and attribute data, a real pickup video image delay section 104 delaying the real pickup picture element data 108 by a processing time of the image generating section 103, a synthesis key signal generating section 105 outputting a synthesis key signal 112 and real pickup picture element data 116 corresponding to the synthesis key signal used to discriminate whether or not an area is a synthesis object area based on a key being a discrimination reference of the given synthesis object area, and a picture synthesis section 106 synthesizing the real pickup picture element data 116 and the generated picture element data 111 based on a synthesis key signal 112 and outputting the synthesis picture element data 113.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for synthesizing a CG image and a photographed image, and particularly, for generating a CG image matched with the photographed image in real time, and generating the CG image and the photographed image. The present invention relates to an image processing apparatus that synchronizes and synthesizes.

[0002]

2. Description of the Related Art In recent years, with the development of CG technology, CG images have come to be used in various places. In particular, a technique for synthesizing a CG image and a photographed image is often used in television programs and commercials (Japanese Patent Laid-Open No. 4-372079, etc.). As an example of the conventional image processing apparatus, an image processing apparatus called "chroma key" for performing image synthesis will be described below.

Chromakey image composition is often used in television broadcasting, and is an image composition technology in which one of two images is embedded in the other image. FIG. 9 shows an image processing apparatus using chroma key image composition. The key signal generation unit 901 performs a detection process on the key detection target image 905 with a specific color or signal level as a key 909, which is a region 903 where a key is detected or a region 904 where a key is not detected. When the key signal generator 901 detects the key 909, the key signal 908 is generated. Image synthesizer 9
When the key signal 908 is detected in 02, the corresponding pixel is replaced with the pixel of the replacement target image 906, and the composite image 907 is output. By making one image a real image and the other image a CG image, a composite image of the real image and the CG image is output.

As another conventional image processing apparatus, an image processing apparatus called "Z buffer method" for performing image synthesis by depth value comparison will be described with reference to FIG. An image processing apparatus using the Z buffer method has at least two or more input images, and for one input image, a frame buffer 1001 for temporarily taking in pixel data and a depth data corresponding to the pixel data are temporarily taken. The depth buffer 1002 that captures the input image is configured by the image synthesis unit 1003 that captures the output of the frame buffer and the output of the depth buffer of each input image. The comparison circuit 1005 selects the input image having the smallest depth value in the image composition unit 1003, and the output of the selector 1004 is switched to the input image on the side selected by the comparison circuit 1005, and the pixel data thereof is output. A composite image 1006 in which hidden surface removal processing has been performed on the above input image is completed. Note that FIG.
Indicates that the darker the depth buffer 1002 is, the larger the depth value is. A composite image of the real image and the CG image is output by using at least one real image and at least one CG image as input images.

[0005]

However, in the above-mentioned conventional configuration, when the real image and the CG image matched with the real image are combined in real time, the CG image is generated with respect to the real image for the CG image generation time. There is a problem that the image becomes out of sync with the delay. This is shown in FIG. The CG image having a correlation with the live-action image is, for example, the zoom, focus, and iris of the camera (hereinafter referred to as image-pickup condition), the shooting position, and the line-of-sight direction of the camera (hereinafter referred to as image-pickup position data) when the real-shot image is shot It is a CG image generated based on the above.

It is therefore an object of the present invention to provide an image processing apparatus for synthesizing a real shot image and a CG image matched with the real shot image in real time.

[0007]

According to a first aspect of the present invention, there is provided an image pickup section for picking up an image of a subject and outputting real shot pixel data and shooting conditions.
An imaging position detection unit that detects the position direction of the imaging unit in synchronization with the imaging unit, and a three-dimensional geometry with the imaging conditions output from the imaging unit and the position direction output from the imaging position detection unit as inputs. An image generation unit that generates an image based on the shape and attribute data of the shape model and outputs generated pixel data;
A live-action video delay unit that receives the live-action pixel data output of the imaging unit as an input and delays the output by a processing time of the image generation unit;
A synthetic key signal for identifying whether the region is a synthesis target region on the basis of a key which is an identification reference of the synthesis target region provided with the output of the live-action image delay unit as input, and real-image pixel data corresponding to the synthesis key signal. A synthetic key signal generator that outputs
An image synthesizing unit that inputs the synthetic key signal generating unit output and the image generating unit output, synthesizes the image data of the real shot pixel data and the generated pixel data based on the synthetic key signal, and outputs synthetic pixel data; It is characterized by comprising an image output unit which receives a partial output as an input and outputs it as a video image.

According to a second aspect of the present invention, an image pickup section for picking up an image of a subject and outputting real shot pixel data and shooting conditions, an image pickup position detecting section for detecting the position direction of the image pickup section in synchronization with the image pickup section, and the image pickup section. Based on the image condition output from the image capturing unit and the position direction output from the image capturing position detecting unit as input, the shape and attribute data of the three-dimensional geometric shape model and the given key that is the identification reference of the synthesis target region An image generation unit that generates an image and outputs generated pixel data and a generated key signal that identifies whether the generated pixel data is a combination target region, and a process of the image generation unit that receives the actual pixel data output of the imaging unit as an input The photographed image delay unit that delays and outputs the time, the photographed image delay unit output and the image generation unit output are input, and the photographed pixel data and the generated pixel data are imaged based on the generation key signal. An image synthesizing unit for outputting a form composite pixel data, is characterized by comprising an image output unit for outputting a video as input the image synthesizing unit output.

According to a third aspect of the present invention, an image pickup section for photographing a subject and outputting the actually photographed pixel data and photographing conditions, and measuring the position of the subject photographed by the image pickup section and outputting the actually photographed depth value in synchronization with the image pickup section. Depth information detection unit, an imaging position detection unit that detects the position direction of the imaging unit in synchronization with the imaging unit, shooting conditions output from the imaging unit, and a position direction output from the imaging position detection unit An image generation unit that generates an image based on the shape and attribute data of the three-dimensional geometric shape model and outputs the generated pixel data and the generated depth value of the generated pixel data, and the actual pixel data output of the imaging unit And a depth information delay unit that outputs the image information after delaying the processing time of the image generation unit, and a depth information delay unit that inputs the output of the depth information detection unit and outputs the image after delaying the processing time of the image generation unit. Inputting the output of the photographed image delay unit, the output of the image generation unit, and the output of the depth information delay unit, the synthesized image data of the photographed pixel data and the generated pixel data is synthesized based on the photographed depth value and the generated depth value. Is provided, and an image output section that receives the combined image data output from the image combining section as an input and outputs as an image.

According to a fourth aspect of the present invention, an image pickup section for picking up an image of a subject and outputting real shot pixel data and shooting conditions, a position of the subject shot by the image pickup section is measured, and a real shot depth value is output in synchronization with the image pickup section. Depth information detection unit, an imaging position detection unit that detects the position direction of the imaging unit in synchronization with the imaging unit, shooting conditions output from the imaging unit, and a position direction output from the imaging position detection unit An image generation unit that generates an image based on the shape and attribute data of the three-dimensional geometric shape model and outputs the generated pixel data and the generated depth value of the generated pixel data, and the actual pixel data output of the imaging unit And a depth information delay unit that outputs the image information after delaying the processing time of the image generation unit, and a depth information delay unit that inputs the output of the depth information detection unit and outputs the image after delaying the processing time of the image generation unit. A synthetic key signal for identifying whether the region is a synthesis target region on the basis of a key which is an identification reference of the synthesis target region provided with the output of the live-action image delay unit as input, and real-image pixel data corresponding to the synthesis key signal. Based on the synthetic key signal, the actual shooting depth value, and the generated depth value, with the synthetic key signal generation unit for outputting, the synthetic key signal generation unit output, the image generation unit output, and the depth information delay unit output as inputs. An image synthesizing unit for synthesizing the image of the photographed pixel data and the generated pixel data and outputting synthesized pixel data,
An image output unit that receives the combined image data output from the image combining unit as an input and outputs the image as a video is provided.

According to a fifth aspect of the present invention, an image pickup section for picking up an image of a subject and outputting real shot pixel data and shooting conditions, a position of the subject shot by the image pickup section is measured, and a real shot depth value is output in synchronization with the image pickup section. Depth information detection unit, an imaging position detection unit that detects the position direction of the imaging unit in synchronization with the imaging unit, shooting conditions output from the imaging unit, and a position direction output from the imaging position detection unit Is input and the shape and attribute data of the three-dimensional geometric shape model and an image are generated based on the given key that is the identification criterion of the synthesis target area, and whether the generated pixel data and the generated pixel data are the synthesis target area. An image generation unit that outputs a generated key signal for identification and a generated depth value of the pixel data, and a live-action image that is output with the actual pixel data output of the imaging unit as an input and delayed by the processing time of the image generation unit. A delay section, a depth information delay section that receives the output of the depth information detection section as an input and delays the output by the processing time of the image generation section, and outputs the real image delay section output, the image generation section output, and the depth information delay section output. And an image synthesizing section for synthesizing the real shot pixel data and the generated pixel data based on the generated key signal, the real shot depth value, and the generated depth value, and outputting the synthesized pixel data.
An image output unit that receives the combined image data output from the image combining unit as an input and outputs the image as a video is provided.

According to a sixth aspect of the present invention, an image pickup section for picking up an image of a subject and outputting real shot pixel data and shooting conditions, a position of the subject shot by the image pickup section is measured, and a real shot depth value is output in synchronization with the image pickup section. Depth information detection unit, an imaging position detection unit that detects the position direction of the imaging unit in synchronization with the imaging unit, shooting conditions output from the imaging unit, and a position direction output from the imaging position detection unit Is input and the shape and attribute data of the three-dimensional geometric shape model and an image are generated based on the given key that is the identification criterion of the synthesis target area, and whether the generated pixel data and the generated pixel data are the synthesis target area. An image generation unit that outputs a generated key signal for identification and a generated depth value of the pixel data, and a live-action image that is output with the actual pixel data output of the imaging unit as an input and delayed by the processing time of the image generation unit. An extension part, a depth information delay part that receives the output of the depth information detection part as an input and delays the output by the processing time of the image generation part, and outputs the real image delay part as an input A synthetic key signal generation unit that outputs a synthetic key signal for identifying whether the region is a synthetic target region based on a certain key and actual pixel data corresponding to the synthetic key signal; an output of the synthetic key signal generation unit; The output of the image generation unit, the output of the depth information delay unit, and the background pixel data of the given background image are input, and the real pixel data based on the composite key signal, the generated key signal, the real depth value, and the generated depth value. And an image that combines the generated pixel data and outputs the combined pixel data, and an image that receives the combined image data output from the image combining unit as an input and outputs the image. It is characterized by comprising a power unit.

A seventh invention is characterized in that the image synthesizing unit of the third, fourth or fifth invention is replaced with an image synthesizing unit for outputting synthetic pixel data and a synthetic depth value corresponding to the synthetic pixel data. There is.

In an eighth aspect of the invention, the image synthesizing unit of the sixth aspect of the invention adds the background pixel data and the background depth value of a given background image to generate the synthetic pixel data and the synthetic depth corresponding to the synthetic pixel data. It is characterized in that it is replaced with an image combining unit that outputs a value.

A ninth aspect of the present invention further comprises an illumination condition extraction unit for detecting the illumination condition for the subject and outputting illumination condition data to each of the above inventions, and the image generation unit receives the illumination condition input from the illumination condition extraction unit. It is characterized in that it is replaced with an image generation unit that generates an image according to data.

[0016]

According to the first aspect of the present invention having the above-described structure, the image generation unit generates a CG image matched with a real image,
The real photographed pixel data is delayed by the processing time of the image generation unit to generate a key signal from the real photographed image side, and real-time key combination in which the real photographed image and the CG image are synchronized is performed.

According to a second aspect of the present invention, a CG image matched with the real image is generated in the image generation unit, the real pixel data is delayed by the processing time of the image generation unit to generate a key signal from the CG image side, and the real image is generated. And real-time key composition in synchronization with the CG image.

According to a third aspect of the present invention, the CG image matched with the photographed image is generated in the image generating section, and the photographed pixel data and the photographed depth value are delayed by the processing time of the image generating section and the photographed depth value and the depth of the CG image. Real-time Z synchronized between the real image and the CG image using the generated depth value which is the value
Buffer composition is performed and a composite image is output.

According to a fourth aspect of the present invention, a CG image matched with the real image is generated in the image generation unit, and the real pixel data and the real depth value are delayed by the processing time of the image generation unit and the real depth value and the depth of the CG image. By using the generated depth value that is a value and the synthesized key signal that is a key signal that is generated from the real shot image side, real-time Z buffer synthesis and real-time composite synthesis of the real shot image and the CG image are performed. Output a composite image.

According to a fifth aspect of the present invention, the CG image matched with the photographed image is generated in the image generating section, and the photographed pixel data and the photographed depth value are delayed by the processing time of the image generating section and the photographed depth value and the depth of the CG image. By using the generated depth value that is a value and the generated key signal that is a key signal generated from the CG image side, real-time image and CG image are combined in real time in a combined manner of Z buffer synthesis and key synthesis. Output a composite image.

According to a sixth aspect of the present invention, the CG image matched with the photographed image is generated in the image generating section, and the photographed pixel data and the photographed depth value are delayed by the processing time of the image generating section, and the photographed depth value and the CG image depth. A generated depth value which is a value, a synthetic key signal which is a key signal generated from the photographed image side, and a CG
By using a generated key signal which is a key signal generated from the image side, a real-time image and a CG image are combined in real time to perform a composite combination of Z buffer composition and key composition, and a composite image is output.

A seventh invention adds a composite pixel depth value to the output of the device of the third, fourth or fifth invention.

An eighth aspect of the present invention adds a background image and a background depth value to the apparatus of the sixth aspect of the invention as a compositing target and adds a synthesized pixel depth value to the output of the apparatus of the sixth aspect of the invention.

The ninth aspect of the present invention is adapted to further match the illumination condition to the composite image of the apparatus of each aspect.

[0025]

【Example】

(Embodiment 1) Hereinafter, Embodiment 1 of the present invention will be described with reference to FIG. FIG. 1 shows the configuration of an image processing apparatus according to the first embodiment of the present invention.

In FIG. 1, reference numeral 101 denotes an image pickup section for picking up an image of a subject and outputting the real shot pixel data 108 and shooting conditions 109. An imaging position detection unit that outputs data 110, an image generation unit 103 that outputs generated pixel data 111 from the shape and attribute data 114 of the given three-dimensional geometric shape model, the imaging position data 110, and the imaging conditions 109, Reference numeral 104 denotes an actual-image video delay unit that outputs delayed actual-photographed pixel data 115 from the actual-photographed pixel data 108; Synthetic key signal generator, 1
Reference numeral 06 denotes the generated pixel data 111, the synthetic key signal 112, the synthetic key signal corresponding real shot pixel data 116, and the synthetic pixel 11
3 is an image synthesizing unit that outputs 3, and 107 is an image output unit that outputs a synthetic video from the synthetic pixel 113.

Next, the operation of the first embodiment will be described. First, in the imaging unit 101, the subject 118 is photographed,
The image is output as the real image pixel data 108. Also, the zoom, focus, iris, etc. of the lens are output as the shooting condition 109. The imaging position detection unit 102
The position of the camera in the three-dimensional space is detected by a distance sensor (not shown) using ultrasonic waves, the direction of the camera is detected by a rotary encoder (not shown), and output as imaging position data 110.

Next, for the given shape and attribute data of the three-dimensional geometric shape model 114, the image generation unit 1
03 is the imaging condition 109 output from the imaging unit 101,
Imaging position data 1 output from the imaging position detection unit 102
10, the viewpoint position, the line-of-sight direction, and the angle of view at the time of image generation are determined, the image is generated based on these data, and the result is output as generated pixel data 111.

Next, the actually-photographed image delay unit 104 delays the actually-photographed pixel data 108 by the processing time of the image generation unit 103 using the FIFO, and outputs the delayed actually-photographed pixel data 115. Therefore, at the input stage of the image synthesizing unit 106, the delayed actual shot image data 115 and the generated pixel data 111 are synchronized.

Next, in the synthetic key signal generation unit 105, the synthetic key signal 1 is generated based on the key 117 designated in advance for the delayed actual image pixel data 115 as in the conventional example.
12 and the synthetic key signal 112 corresponding to the synthetic key signal 112 are output.

Next, the image synthesizing unit 106 selectively outputs the generated pixel data instead of the synthetic key signal corresponding real shot pixel data when a key is detected in the synthetic key signal 112 output from the synthetic key signal generating unit 105. If no key is detected in the combined key signal 112, the combined key signal corresponding real pixel data is output. Next, in the image output unit 107, the composite pixel 113 is converted into a video signal and output to a display or the like.

As described above, according to the present embodiment, the CG image matched with the photographed image is generated in the image generation section, the photographed pixel data is delayed by the processing time of the image generation section, and the key signal is transmitted from the photographed image side. Since it is generated, a CG image matched with the real photographed image is generated, and real-time key combination is possible in which a key signal is detected in the real photographed image in which the real photographed image and the CG image are synchronized.

Although the rotary encoder is used for measuring the direction of the camera in this embodiment, the direction measuring method is not limited to this, and various methods such as a method by image processing of a stereo camera and a measurement by a magnetic sensor are used. However, the same operation is possible.

In this embodiment, a distance sensor using ultrasonic waves is used to measure the position of the camera, but the position detecting method is not limited to this, and a rotary encoder or a magnetic sensor is used. It can also be obtained by the method.

In this embodiment, the chroma key method is used for generating the synthetic key signal in the synthetic key signal generating section, but the synthetic key signal generating method is not limited to this. It is also possible to generate a synthetic key signal by extracting a region by image recognition.

(Second Embodiment) A second embodiment of the present invention will be described below with reference to FIG. FIG. 2 shows the configuration of the image processing apparatus according to the second embodiment of the present invention.

In FIG. 2, reference numeral 201 denotes an image pickup section for picking up an image of an object and outputting the real shot pixel data 207 and shooting conditions 208, and 202, the position of the image pickup section 201 in the three-dimensional space and the image pickup position of the image pickup section such as the line-of-sight direction. An imaging position detecting unit 203 for outputting data 209 generates pixel data and generation from the given three-dimensional geometric shape model shape and attribute data 212, the imaging position data 209, the photographing condition 208 and the given key signal 214. An image generation unit that outputs the key signal 216, 204 is a real image delay unit that outputs delayed real image pixel data 213 from the real image pixel data 207, and 205 is generated pixel data 210, generated key signal 216, and synthetic pixel from the delayed actual image pixel data 213. An image composition unit that outputs 211, and an image output unit 206 that outputs a composite video from the composite pixel 211.

Next, the operation of the second embodiment will be described. First, in the imaging unit 201, the subject 215 is photographed,
The image is output as the real pixel data 207. Also, the zoom, focus, iris, etc. of the lens are output as the shooting condition 208. The imaging position detection unit 202
The position of the camera in the three-dimensional space is detected by the distance sensor using ultrasonic waves, the direction of the camera is detected by the rotary encoder, and the detected position data 209 is output.

Next, for the given shape and attribute data of the three-dimensional geometric shape model 212, the image generation unit 2
03 is an imaging condition 208 output from the imaging unit 201,
Imaging position data 2 output from the imaging position detection unit 202
09, the viewpoint position, the line-of-sight direction, and the angle of view at the time of image generation are determined, the image is generated based on these data, and the result is output as generated pixel data 210.
A generated key signal 216 indicates whether or not the generated pixel data 210 is detected as a key based on the key 214 given in advance, and is output in correspondence with the generated pixel data 210.

Next, the actually-photographed image delay unit 204 delays the actually-photographed pixel data 207 by the processing time of the image generation unit 203 using the FIFO, and outputs the delayed actually-photographed pixel data 213. Therefore, at the input stage of the image synthesizing unit 205, the delayed real shot image data 213 and the generated pixel data 210 are synchronized.

Next, the image synthesizing unit 206 makes the image generating unit 20.
When a key is detected in the generated key signal 216 output from No. 3, real pixel data is selectively output instead of the generated pixel data, and when no key is detected in the generated key signal 216, generated pixel data is output. .
Next, in the image output unit 206, the composite pixel 211 is converted into a video signal and output to a display or the like.

As described above, according to the present embodiment, the CG image matched with the photographed image is generated in the image generation unit, and the photographed pixel data is delayed by the processing time of the image generation unit C.
By generating a key signal from the G image side, a real image and CG can be obtained.
Real-time key composition that is synchronized with the image can be performed.

Although the rotary encoder is used for measuring the direction of the camera in the present embodiment, the direction measuring method is not limited to this, and various methods such as a method by image processing of a stereo camera and a measurement by a magnetic sensor are used. However, the same operation is possible.

In this embodiment, a distance sensor using ultrasonic waves is used for measuring the position of the camera, but the position detecting method is not limited to this, and a rotary encoder or a magnetic sensor is used. It can also be obtained by the method.

In this embodiment, the chroma key method is used in the generation of the generation key signal in the image generation unit, but the generation key signal generation method is not limited to this, and image recognition is not limited to this. It is also possible to generate the generated key signal by extracting the area with.

(Embodiment 3) A third embodiment of the present invention will be described below with reference to FIG. FIG. 3 shows the configuration of the image processing apparatus according to the third embodiment of the present invention.

In FIG. 3, reference numeral 301 denotes an image pickup section for picking up an image of a subject 319 and outputting the real shot pixel data 309 and shooting conditions 310. Reference numeral 302 denotes an image pickup section for picking up an image pickup section 301 such as a position in a three-dimensional space and a line-of-sight direction. An imaging position detection unit that outputs position data 311; a depth information detection unit 306 that measures the position of a subject imaged by the imaging unit 301 and outputs an actual depth value 313; and a shape 303 of a given three-dimensional geometric shape model. And an attribute data 318, an image pickup position data 311, and an image generation unit that outputs generated pixel data 312 and a generated depth value 315 based on the shooting condition 310, and a real image delay unit 304 that outputs delayed real image pixel data 316 from the real image pixel data 309. , 307 is a depth information delay unit that outputs a delayed actual-photographed depth value 314 from the actual-photographed depth value 313, and 305 is a generated pixel Over data 312 and generates a depth value 31
5, delayed actual shot pixel data 316, and delayed actual shot depth value 31
4 is an image synthesizing unit that outputs a synthetic pixel 317, and 308 is an image output unit that outputs a synthetic image from the synthetic pixel 317.

Next, the operation of the third embodiment will be described. First, in the imaging unit 301, a subject 319 is photographed,
The image is output as the real shot pixel data 309. Also, the zoom, focus, iris, etc. of the lens are output as the shooting condition 310. The imaging position detection unit 302
The position of the camera in the three-dimensional space is detected by the distance sensor using ultrasonic waves, the direction of the camera is detected by the rotary encoder, and output as the imaging position data 311.

Next, the depth information detecting section 306 measures the distance from the camera to the object 319 using a magnetic sensor. A distance measuring method using a magnetic sensor is shown in FIG. In FIG. 11, one of the magnetic sensors 1101 is attached to a part of the subject 1102 and the other is attached to a part of the camera 1103, and an actual depth value 1104 which is a distance from the camera 1103 to the subject 1102 is obtained. In the case of FIG. 11, the entire subject is represented by the real shot depth value 1104 and is treated as a planar object. Therefore, the photographed depth value 1104 must be fetched at the switching timing of the field or frame.

When the method shown in FIG. 11 is used for the depth information detecting section, there is a problem that only one subject can be handled and it is treated as a plane. However, the depth of the subject is calculated using a stereo camera. Then these problems are solved. Further, although the magnetic sensor is used as the distance sensor in FIG. 11, the same operation can be performed by using the ultrasonic sensor.

Next, with respect to the shape and attribute data of the given three-dimensional geometric shape model 318, the image generation unit 30
3 is the imaging condition 310 output from the imaging unit 301, and the imaging position data 31 output from the imaging position detection unit 302.
1, the viewpoint position, the line-of-sight direction, and the angle of view at the time of image generation are determined, image generation is performed based on these data, and the result is generated pixel data 312 and generated pixel data 312.
Is output as a generated depth value 315 which is a depth value corresponding to.

Next, the actually-photographed image delay unit 304 delays the actually-photographed pixel data 309 by the processing time of the image generation unit 303 by using the FIFO, and outputs the delayed actually-photographed pixel data 316. Therefore, at the input stage of the image synthesizing unit 305, the delayed real shot image data 316 and the generated pixel data 315 are synchronized.

Next, the depth information delaying unit 307 delays the actually-photographed pixel data 309 by the processing time of the image generating unit 303 by using the FIFO, and outputs the delayed actually-photographed pixel data 316. Therefore, the delayed actual shooting depth value 314 and the generated depth value 315 are synchronized at the input stage of the image composition unit 305.

Next, the image synthesizing unit 305, the image generating unit 30.
Generated depth value 315 and generated pixel data 3 output in 3
12 and the delayed actual captured depth value 314 output from the depth information delay unit 307 and the delayed actual captured pixel data 316 output from the actual captured image delay unit 304, a hidden surface removal process using the Z buffer method is performed to generate a composite pixel. 317 is output. Next, the image output unit 308 converts the composite pixel 317 into a video signal and outputs it to a display or the like.

As described above, according to the present embodiment, the image generation unit generates the CG image matched with the photographed image,
Real image pixel data and real depth value are delayed by the processing time of the image generation unit, and real-time Z-buffer synthesis in which the real image and the CG image are synchronized by using the real depth value and the generated depth value which is the depth value of the CG image. Can be done.

Although the rotary encoder is used for measuring the direction of the camera in this embodiment, the direction measuring method is not limited to this, and various methods such as a method by image processing of a stereo camera and a measurement by a magnetic sensor are used. However, the same operation is possible.

In this embodiment, a distance sensor using ultrasonic waves is used for measuring the position of the camera, but the position detecting method is not limited to this, and a rotary encoder or a magnetic sensor is used. It can also be obtained by the method.

(Fourth Embodiment) Next, a fourth embodiment of the present invention will be described with reference to FIG. FIG. 4 shows the configuration of the image processing apparatus according to the fourth embodiment of the present invention.

In FIG. 4, reference numeral 401 denotes an image pickup section for picking up an image of a subject 422 and outputting real shot pixel data 410 and shooting conditions 411. Reference numeral 402 denotes an image pickup section such as a position of the image pickup section 401 in a three-dimensional space and a line-of-sight direction. An imaging position detection unit that outputs position data 413, a depth information detection unit 406 that measures the position of the subject imaged by the imaging unit 401, and outputs an actual depth value 414, and a shape 403 of the given three-dimensional geometric shape model. And an image generation unit that outputs the generated pixel data 412 and the generated depth value 416 from the attribute data 421, the imaging position data 413, and the shooting condition 411, and a real image delay unit that outputs delayed real image pixel data 417 from the real image pixel data 410. , 407 is a depth information delay unit that outputs a delayed actual-shot depth value 415 from the actual-shot depth value 414, and 408 is a delayed actual-shot image. Synthetic key signal generator for outputting a synthetic key signal 418 a synthetic key signal corresponding photographed pixel data 419 from the key 423 with the given raw data 417, 4
Reference numeral 05 denotes the generated pixel data 412, the generated depth value 416, the combined key signal 418, and the combined key signal corresponding real shot pixel data 4
19 is an image synthesizing unit that outputs a synthetic pixel 420 based on the delayed actual shooting depth value 415, and 409 is an image output unit that outputs a synthetic video from the synthetic pixel 420.

Next, the operation of the fourth embodiment will be described. First, in the imaging unit 401, a subject 422 is photographed,
The image is output as the real shot pixel data 410. Further, the zoom, focus, iris, etc. of the lens are output as the shooting condition 411. The imaging position detection unit 402
The position of the camera in the three-dimensional space is detected by the ultrasonic distance sensor, the direction of the camera is detected by the rotary encoder, and the detected position data 413 is output.

Next, the depth information detecting section 406 measures the distance from the camera to the subject 422 as in the third embodiment.

Next, with respect to the shape and attribute data of the given three-dimensional geometric shape model 421, the image generation unit 40
3 is the imaging condition 411 output from the imaging unit 401 and the imaging position data 41 output from the imaging position detection unit 402.
3, the viewpoint position, the line-of-sight direction, and the angle of view at the time of image generation are determined, the image is generated based on these data, and the result is generated pixel data 412 and generated pixel data 412.
Is output as a generated depth value 416 which is a depth value corresponding to.

Next, the actually-photographed image delay unit 404 delays the actually-photographed pixel data 410 by the processing time of the image generation unit 403 using the FIFO, and outputs the delayed actually-photographed pixel data 417. Therefore, at the input stage of the image synthesizing unit, the synthetic key signal corresponding real shot pixel data 419 and the generated pixel data 416 are synchronized.

Next, the depth information delay unit 407 delays the actual captured depth value 414 by the processing time of the image generation unit 403 using the FIFO and outputs the delayed actual captured depth value 415.
Therefore, at the input stage of the image synthesizing unit, the delay actual shooting depth value 4
15 and the generated depth value 416 are synchronized.

Next, in the synthetic key signal generation unit 408, the synthetic key signal 4 is generated based on the key 423 previously designated for the delayed actual image pixel data 417, as in the conventional example.
18 and synthetic key signal 418 are output corresponding to synthetic key signal corresponding real pixel data 419.

Next, the image synthesizing unit 405 outputs the generated depth value 415 and the generated pixel data 412 output from the image generating unit, the synthetic key signal 418 output from the synthetic key signal generating unit 408, and the synthetic key signal corresponding real shot pixel data. 419 and the delay actual shooting depth value 4 output from the depth information delay unit 407.
15 is used to perform the chroma key processing and the hidden surface removal processing using the Z buffer method, and output the composite pixel 420.
This process is shown in FIG.

In FIG. 13, the comparison circuit 1305 is a CG.
The image-side depth buffer 1302 and the photographed image-side depth buffer 1304 are compared to select the input image whose depth value is closer to the front. The key signal generation unit 1307 performs the processing shown in the conventional example, and outputs the key signal 1309.
Is output. The input image selection circuit 1306 outputs the result selected by the comparison circuit 1305 as the selected image 1314 on the CG image side for the region where the key signal is detected, and the selector 130 for the region where the key signal is not detected.
6 switches images based on the selected image 1314.

Next, the image output section 409 converts the composite pixel 420 into a video signal and outputs it to a display or the like.

As described above, according to this embodiment, the image generation unit generates the CG image matched with the photographed image,
The actual-photographed pixel data and the actual-photographed depth value are delayed by the processing time of the image generator, and the actual-photographed depth value and the generated depth value that is the depth value of the CG image and the synthetic key signal that is the key signal generated from the actual-photographed image side are used. It is possible to perform real-time Z buffer composition and key composition compound composition in real time in synchronization with the CG image.

Although the rotary encoder is used for measuring the direction of the camera in the present embodiment, the direction measuring method is not limited to this, and various methods such as a method by image processing of a stereo camera and a measurement by a magnetic sensor are used. However, the same operation is possible.

In this embodiment, a distance sensor using ultrasonic waves is used for measuring the position of the camera, but the position detecting method is not limited to this, and a rotary encoder or a magnetic sensor is used. It can also be obtained by the method.

In this embodiment, the chroma key method is used for generating the synthetic key signal in the synthetic key signal generating section, but the synthetic key signal generating method is not limited to this. It is also possible to generate a synthetic key signal by extracting a region by image recognition.

(Fifth Embodiment) Next, a fifth embodiment of the present invention will be described with reference to FIG. FIG. 5 shows the configuration of an image processing apparatus according to the fifth embodiment of the present invention.

In FIG. 5, reference numeral 501 denotes an image pickup section for picking up an image of a subject 520 and outputting the real shot pixel data 509 and shooting conditions 510. An image pickup position detection unit that outputs position data 512, a depth information detection unit 506 that measures the position of a subject photographed by the image pickup unit 501 and outputs an actual shot depth value 513, and a shape 503 of a given three-dimensional geometric shape model. And the attribute data 519, the imaging position data 512, the imaging condition 510, and the image generation unit that outputs the generated pixel data 511, the generated depth value 515, and the generated key signal 516 from the given key 521, and 504 is delayed from the actual pixel data 509. A live-action image delay unit that outputs the live-action pixel data 517, and 507 changes the real-action depth value 513 to the delayed live-action depth value 514. Depth information delay section that force, 505
Is an image synthesizing unit that outputs a synthetic pixel 518 from the generated pixel data 511, the generated depth value 515, the generated key signal 516, the delayed actual shot pixel data 517, and the delayed actual shot depth value 514, and 508 is an image that outputs a synthetic video from the synthetic pixel 518. It is an output part.

Next, the operation of the fifth embodiment will be described. First, in the imaging unit 501, the subject 520 is photographed,
The image is output as the real pixel data 509. Further, the zoom, focus, iris, etc. of the lens are output as the shooting condition 510. The imaging position detection unit 502
The position of the camera in the three-dimensional space is detected by the ultrasonic distance sensor, the direction of the camera is detected by the rotary encoder, and output as imaging position data 512.

Next, the depth information detecting section 506 measures the distance from the camera to the object 520 as in the third embodiment. Next, for the given shape and attribute data of the three-dimensional geometric shape model 519, the image generation unit 503 outputs the image capturing condition 510 output from the image capturing unit 501 and the image capturing position output from the image capturing position detecting unit 502. The viewpoint position, the line-of-sight direction, and the angle of view at the time of image generation are determined according to the data 513, image generation is performed based on these data, and the result is generated as the generated pixel data 511 and the depth value corresponding to the generated pixel data 511. It is output as a certain generated depth value 516. Further, similarly to the conventional example, a generated key signal 515 is output to the generated pixel data 511 based on a key 521 given in advance.

Next, the actually-photographed image delay unit 504 delays the actually-photographed pixel data 509 by the processing time of the image generation unit 503 using the FIFO, and outputs the delayed actually-photographed pixel data 517. For this reason, the delayed actual-photographed pixel data 517 and the generated pixel data 511 are synchronized at the input stage of the image composition unit.

Next, the depth information delay unit 507 delays the actual captured depth value 513 by the processing time of the image generation unit 503 using the FIFO, and outputs the delayed actual captured depth value 514.
Therefore, at the input stage of the image synthesizing unit, the delay actual shooting depth value 5
14 and the generated depth value 516 are synchronized.

Next, the image synthesizing unit 505 outputs the generated depth value 516, the generated pixel data 511, the generated key signal 515, the delayed actual pixel data 517 output from the actual image delay unit 504, and the depth information output from the image generation unit. Delay unit 50
7 and the delayed real-life depth value 514 output from
Hidden surface removal processing using the chroma key processing and the Z buffer method is performed, and the composite pixel 518 is output. In this process, the key detection target image is changed to a CG image, and then the same process as in FIG. 13 is performed.

Next, the image output section 508 converts the composite pixel 518 into a video signal and outputs it to a display or the like.

As described above, according to this embodiment, the image generation unit generates the CG image matched with the photographed image,
The actual-photographed pixel data and the actual-photographed depth value are delayed by the processing time of the image generation unit, and the actual-photographed depth value, the generated depth value that is the depth value of the CG image, and the generated key signal that is the key signal generated from the CG image side are actually photographed. It is possible to perform composite real-time composite Z buffer composition and key composition of an image and a CG image.

Although the rotary encoder is used for measuring the direction of the camera in the present embodiment, the direction measuring method is not limited to this, and various methods such as a method by image processing of a stereo camera and a measurement by a magnetic sensor are used. However, the same operation is possible.

In this embodiment, a distance sensor using ultrasonic waves is used for measuring the position of the camera, but the position detecting method is not limited to this, and a rotary encoder or a magnetic sensor is used. It can also be obtained by the method.

In the present embodiment, the chroma key method is used to generate the generated key signal 515 in the image generation unit 503, but the generated key signal generation method is not limited to this. It is also possible to generate the generated key signal by extracting the area by image recognition.

(Sixth Embodiment) Next, a sixth embodiment of the present invention will be described with reference to FIG. FIG. 6 shows the configuration of an image processing apparatus in Embodiment 6 of the present invention.

In FIG. 6, reference numeral 601 denotes an image pickup section for picking up an image of the subject 623 and outputting the real shot pixel data 610 and the shooting conditions 611. An imaging position detection unit that outputs position data 613, a depth information detection unit 606 that measures the position of a subject imaged by the imaging unit 601, and outputs an actual depth value 614, and a reference numeral 603 represents the shape of a given three-dimensional geometric shape model. An image generation unit that outputs the generated pixel data 612, the generated depth value 616, and the generated key signal 620 from the attribute data 622, the image pickup position data 613, the shooting condition 611, and the given key 625, and 604 is a delay from the actually shot pixel data 610. A live-action video delay unit that outputs live-action pixel data 617, and 607 converts a delay live-action depth value 615 from a live-action depth value 614. Depth information delay section that force, 608
Is a synthetic key signal generation unit which outputs the synthetic key signal 618 and the synthetic key signal corresponding real image pixel data 619 from the delayed actual image pixel data 617 and the given key 624, and 605 is the generated pixel data 612, the generated depth value 616, and the generated key signal. 620, a synthetic key signal 618, a synthetic key signal corresponding real pixel data 619, a delayed real depth value 615, and an image synthesizing unit 609 which outputs a synthetic pixel 621 from background pixel data given in advance, and 609 a synthetic image from the synthetic pixel 621. It is an image output unit that does.

Next, the operation of the sixth embodiment will be described. First, in the imaging unit 601, a subject 623 is photographed,
The image is output as the real shot pixel data 610. Further, the zoom, focus, iris, etc. of the lens are output as the shooting condition 611. The imaging position detection unit 602
The position of the camera in the three-dimensional space is detected by the ultrasonic distance sensor, the direction of the camera is detected by the rotary encoder, and the detected position data 613 is output. Next, the depth information detection unit 606 measures the distance from the camera to the subject 623 as in the third embodiment.

Next, for the shape and attribute data of the given three-dimensional geometric shape model 622, the image generation unit 60
3 is the imaging condition 611 output from the imaging unit 601 and the imaging position data 61 output from the imaging position detection unit 602.
3, the viewpoint position, the line-of-sight direction, and the angle of view at the time of image generation are determined, image generation is performed based on these data, and the result is generated pixel data 612 and generated pixel data 612.
Is output as a generated depth value 616 which is a depth value corresponding to. Further, similarly to the conventional example, a generated key signal 620 is output to the generated pixel data 612 based on a key 625 given in advance.

Next, the real image delay unit 604 delays the real pixel data 610 by the processing time of the image generation unit 603 using the FIFO, and outputs the delayed real pixel data 617. Therefore, in the input stage of the image synthesizing unit, the synthetic key signal corresponding real shot pixel data 619 and the generated pixel data 612 are synchronized.

Next, the depth information delay unit 607 delays the actual captured depth value 614 by the processing time of the image generation unit 603 using the FIFO and outputs the delayed actual captured depth value 615.
Therefore, at the input stage of the image synthesizing unit, the delay actual shooting depth value 6
15 and the generated depth value 616 are synchronized.

Next, in the synthetic key signal generator 608, the synthetic key signal 6 is generated based on the key 616 designated in advance for the delayed actual pixel data 617, as in the conventional example.
18 and the synthetic key signal 618, the synthetic key signal corresponding real image pixel data 619 is output.

Next, the image synthesizing unit 605 outputs the generated depth value 615 output from the image generating unit, the generated pixel data 612, the generated key signal 620, and the combined key signal 618 and the combined key output from the combined key signal generating unit 608. Using the signal-based actual captured pixel data 619 and the delayed actual captured depth value 615 output from the depth information delaying unit 607, a hidden surface removal process using chroma key processing and the Z buffer method is performed, and a synthetic pixel 621 is output. This process is shown in FIG.

In FIG. 14, a CG image side key signal generation circuit 1407 outputs a C signal from the CG image side frame buffer based on a CG image side key 1406 given in advance.
The generated key signal 1414 which is the key signal on the G image side is output. This operation is a part of the operation of the image generation unit 603 in FIG. The captured image side key signal generation unit 1408 outputs a combined key signal 1415, which is a captured image side key signal, from the captured image side frame buffer 1403 based on a captured image side key 1409 given in advance. This operation is a part of the operation of the composite image generation unit 608 in FIG.

The comparison circuit 1410 uses the Z of the depth buffer 1402 on the CG image side and the depth buffer 1404 on the photographed image side.
The comparison result 1416 is output. Input image selection circuit 141
3 selects a photographed image for an area in which only the generated key signal 1414 is detected, selects a CG image for an area in which only the synthesized key signal 1415 is detected, and both the generated key signal 1414 and the synthesized key signal 1415. The background is selected for the detected area, and the generated key signal 1414 is selected.
For a region in which neither the composite key signal 1415 is detected, the image in front of the Z comparison result 1416 is selected. The selector 1412 switches the image output according to the selected image 1417 output from the input image selection circuit 1411.

Next, the image output unit 609 converts the composite pixel 621 into a video signal and outputs it to a display or the like.

As described above, according to the present embodiment, the image generation unit generates a CG image matched with a real image,
The actual pixel data and the actual depth value are delayed by the processing time of the image generation unit, the actual depth value and the generated depth value which is the depth value of the CG image, the synthetic key signal which is the key signal generated from the actual image side, and the CG image side. By using the generated key signal which is the generated key signal, real-time Z-buffer synthesis and key synthesis compound synthesis can be performed in real time in synchronization with the CG image.

Although the rotary encoder is used for measuring the direction of the camera in this embodiment, the direction measuring method is not limited to this, and various methods such as a method by image processing of a stereo camera and a measuring by a magnetic sensor are used. However, the same operation is possible.

Further, although the present embodiment discloses the one using the distance sensor using the ultrasonic wave for the position measurement of the camera, the position detecting method is not limited to this, and the rotary encoder or the magnetic sensor is used. It can also be obtained by the method.

Further, the present embodiment discloses that the chroma key method is used when the synthetic key signal generating unit 608 generates the synthetic key signal 618 and when the image generating unit 603 generates the generated key signal 620. However, the synthetic key signal generation method is not limited to this, and the synthetic key signal and the generated key signal can be generated by area extraction by image recognition.

(Seventh Embodiment) Next, a seventh embodiment of the present invention will be described with reference to FIG. FIG. 7 shows the configuration of the image processing apparatus according to the seventh embodiment of the present invention.

In FIG. 7, reference numeral 701 denotes an image pickup section for picking up an image of a subject 723 and outputting the real shot pixel data 710 and shooting conditions 711. An image pickup position detection unit that outputs the position data 713, a depth information detection unit 706 that measures the position of the subject imaged by the image pickup unit 701 and outputs an actual depth value 714, and a reference numeral 703 represents the shape of the given three-dimensional geometric shape model. An image generation unit that outputs the generated pixel data 712, the generated depth value 716, and the generated key signal 720 from the attribute data 722, the image pickup position data 713, the shooting condition 711, and the given key 725, and 704 is a delay from the actually shot pixel data 710. A live-action image delay unit that outputs the live-action pixel data 717, and 707 changes the delay-actual-use depth value 714 from the actual-use depth value 714. Depth information delay section that force, 708
Is a synthetic key signal generator for outputting the synthetic key signal 718 and the synthetic key signal corresponding real pixel data 719 from the delayed real image pixel data 717 and the given key 724, and 705 is the generated pixel data 712, the generated depth value 716 and the generated key signal. 720, composite key signal 718, composite key signal corresponding real image pixel data 719, delayed real image depth value 715, preset background pixel data 726 and background depth value 727.
Is an image combining unit that outputs a combined pixel 721 and a combined depth value 728, and 709 is an image output unit that outputs a combined image from the combined pixel 721.

Next, the operation of the seventh embodiment will be described. First, in the imaging unit 701, a subject 723 is photographed,
The image is output as the real pixel data 710. Also, the zoom, focus, iris, etc. of the lens are output as the shooting condition 711. The imaging position detection unit 702
The position of the camera in the three-dimensional space is detected by the ultrasonic distance sensor, the direction of the camera is detected by the rotary encoder, and the detected position data 713 is output. Next, the depth information detection unit 706 measures the distance from the camera to the subject 723 as in the third embodiment.

Next, for the shape and attribute data of the given three-dimensional geometric shape model 722, the image generation unit 7
03 is an imaging condition 711 output from the imaging unit 701,
Imaging position data 7 output from the imaging position detection unit 702
13, the viewpoint position, the line-of-sight direction, and the angle of view at the time of image generation are determined, image generation is performed based on these data,
The result is generated pixel data 712 and generated pixel data 71.
It is output as a generated depth value 716 which is a depth value corresponding to 2. Further, similarly to the conventional example, a generated key signal 720 is output to the generated pixel data 712 based on a key 725 given in advance.

Next, the actually-photographed image delay unit 704 delays the actually-photographed pixel data 710 by the processing time of the image generation unit 703 by using FIFO, and outputs the delayed actually-photographed pixel data 717. Therefore, at the input stage of the image synthesizing unit, the synthetic key signal corresponding real shot pixel data 719 and the generated pixel data 712 are synchronized.

Next, the depth information delay unit 707 delays the actual captured depth value 714 by the processing time of the image generation unit 703 using the FIFO and outputs the delayed actual captured depth value 715.
Therefore, at the input stage of the image synthesizing unit, the delay actual shooting depth value 7
15 and the generated depth value 716 are synchronized.

Next, in the synthetic key signal generator 708, the synthetic key signal 7 is generated based on the key 716 previously designated for the delayed actual image pixel data 717, as in the conventional example.
18 and composite key signal 718 corresponding to the composite key signal 718 are output.

Next, the image synthesizing unit 705 outputs the generated depth value 715 output from the image generating unit, the generated pixel data 712, the generated key signal 720, the combined key signal 718 output from the combined key signal generating unit 708, and the combined key. The signal-corresponding actual-photographed pixel data 719 and the delayed actual-photographed depth value 715 output from the depth information delay unit 707 are used to perform the hidden surface elimination process using the chroma key process and the Z-buffer method, and the synthetic pixel 721 and the synthetic depth value 728. Is output. This process is shown in FIG.

In FIG. 15, the CG image side key signal generation circuit 1507 outputs a C signal from the CG image side frame buffer based on a CG image side key 1506 given in advance.
A generated key signal 1514 which is a key signal on the G image side is output. This operation is a part of the operation of the image generation unit 603 in FIG. The photographed image side key signal generation unit 1508 outputs a synthesized key signal 1515 which is a key signal on the photographed image side from the photographed image side frame buffer 1503 based on a photographed image side key 1509 given in advance. This operation is a part of the operation of the composite image generation unit 608 in FIG.

The comparison circuit 1510 uses the Z in the depth buffer 1502 on the CG image side and the depth buffer 1504 on the photographed image side.
The comparison result 1516 is output. Input image selection circuit 151
3 selects a photographed image for an area in which only the generated key signal 1514 is detected, selects a CG image for an area in which only the synthesized key signal 1515 is detected, and both the generated key signal 1514 and the synthesized key signal 1515. The background is selected for the detected area, and the generated key signal 1514 is selected.
For the area where neither the composite key signal 1515 is detected, the image in front of the Z comparison result 1516 is selected. The selector 1512 switches the image output according to the selected image 1517 output from the input image selection circuit 1511. The selector 1520 is an input image selection circuit 15
According to the selected image 1517 output from No. 11, the output of the depth value corresponding to that image is switched.

Next, the image output unit 709 converts the composite pixel 721 into a video signal and outputs it to a display or the like.

Although the rotary encoder is used for measuring the direction of the camera in the present embodiment, the direction measuring method is not limited to this, and various methods such as a method by image processing of a stereo camera and a measurement by a magnetic sensor are used. The same operation can be performed by the method.

Further, although the present embodiment discloses the one using the distance sensor using the ultrasonic wave for measuring the position of the camera, the position detecting method is not limited to this, and the rotary encoder or the magnetic sensor is used. It can also be obtained by the method.

Further, the present embodiment discloses that the chroma key method is used when the synthetic key signal generation unit 708 generates the synthetic key signal 718 and when the image generation unit 703 generates the generation key signal 720. However, the synthetic key signal generation method is not limited to this, and the synthetic key signal and the generated key signal can be generated by area extraction by image recognition.

Further, as shown in the present embodiment, the same operation is performed in the sixth embodiment only by deleting the background pixel data and the background depth value as the input of the image synthesizing unit.

(Embodiment 8) An embodiment 8 of the present invention will be described below with reference to FIG. FIG. 8 shows the configuration of an image processing apparatus according to the eighth embodiment of the present invention.

In FIG. 8, reference numeral 801 denotes an image pickup section for picking up an image of the subject 823 and outputting the real shot pixel data 810 and the shooting condition 811, and 802 for picking up the image pickup section 801's position in the three-dimensional space, line-of-sight direction and the like. An imaging position detection unit that outputs position data 813, a depth information detection unit 806 that measures the position of a subject to be captured by the imaging unit 801, and outputs a captured depth value 814, and a reference numeral 829 that detects illumination conditions in the captured environment of the captured image. A lighting condition extraction unit that outputs lighting condition data 830. Reference numeral 803 denotes the shape and attribute data 822 of the given three-dimensional geometric shape model, the imaging position data 813, the shooting condition 811, and the given key 825, and the lighting condition. An image generation unit that outputs the generated pixel data 812, the generated depth value 816, and the generated key signal 820 from the data 830, and 804 is an actual image. A live-action video delay unit that outputs delayed live-action pixel data 817 from the data 810, a depth information delay unit 807 that outputs a delay live-action depth value 815 from the real-action depth value 814, and a delayed real-action pixel data 817 and a key 824 assigned to it. A synthetic key signal generation unit that outputs a synthetic key signal 818 and a synthetic key signal corresponding real pixel data 819, and 805 a generated pixel data 812, a generated depth value 816, a generated key signal 820, a synthetic key signal 818, and a synthetic key signal corresponding real pixel. The data 819, the delay image depth value 815, and the background pixel data 82 given in advance.
6 and a background depth value 827, an image combining unit that outputs a combined pixel 821 and a combined depth value 828;
21 is an image output unit that outputs a composite video.

Next, the operation of the eighth embodiment will be described. First, in the imaging unit 801, a subject 823 is photographed,
The image is output as the real pixel data 810. Also, the zoom, focus, iris, etc. of the lens are output as the shooting condition 811. The imaging position detection unit 802
The ultrasonic distance sensor detects the position of the camera in the three-dimensional space, the rotary encoder detects the direction of the camera, and outputs it as imaging position data 813.

Next, the depth information detecting unit 806 measures the distance from the camera to the subject 823 as in the third embodiment.

Next, the illumination condition extraction unit 829 extracts the illumination condition of the actual image by measuring the illumination condition or taking in the condition of the operation panel of the dimmer, and outputs the illumination condition data 830.

Next, for the shape and attribute data of the given three-dimensional geometric shape model 822, the image generation unit 8
03 is an imaging condition 811 output from the imaging unit 801;
Imaging position data 8 output from the imaging position detection unit 802
13, the viewpoint position, the line-of-sight direction, and the angle of view at the time of image generation are determined, the illumination simulation calculation is performed according to the illumination condition data 830 output from the illumination condition extraction unit 829 to obtain the illuminance data, and based on these data. The image is generated, and the result is output as the generated pixel data 812 and the generated depth value 816 which is the depth value corresponding to the generated pixel data 812. Further, the generated pixel data 812
In contrast to the conventional example, the key
Based on 25, the generated key signal 820 is output.

Next, the actually-photographed image delay unit 804 delays the actually-photographed pixel data 810 by the processing time of the image generation unit 803 using the FIFO, and outputs the delayed actually-photographed pixel data 817. Therefore, at the input stage of the image synthesizing unit, the synthetic key signal corresponding real shot pixel data 819 and the generated pixel data 812 are synchronized.

Next, the depth information delay unit 807 delays the actual captured depth value 814 by the processing time of the image generation unit 803 using the FIFO, and outputs the delayed actual captured depth value 815.
Therefore, at the input stage of the image synthesizing unit, the delay actual shooting depth value 8
15 and the generated depth value 816 are synchronized.

Next, in the synthetic key signal generation unit 808, the synthetic key signal 8 is generated based on the key 816 designated in advance for the delayed actual pixel data 817, as in the conventional example.
18 and synthetic key signal 818 corresponding to the synthetic key signal 818 are output.

Next, the image synthesizing unit 805 outputs the generated depth value 815, the generated pixel data 812, the generated key signal 820, the synthetic key signal 818 and the synthetic key output from the image generation unit 808. Using the signal-corresponding real-photographed pixel data 819 and the delayed real-photographed depth value 815 output from the depth information delay unit 807, the hidden surface erasing process using the chroma key process and the Z-buffer method is performed in the same manner as in the seventh embodiment, and the composite pixel is obtained. 821 and the composite depth value 828 are output.

Next, the image output unit 809 converts the composite pixel 821 into a video signal and outputs it to a display or the like.

Although the rotary encoder is used for measuring the direction of the camera in this embodiment, the direction measuring method is not limited to this, and various methods such as a method by image processing of a stereo camera and a measurement by a magnetic sensor are used. The same operation can be performed by the method.

Further, although the present embodiment discloses the one using the distance sensor using the ultrasonic wave for the position measurement of the camera, the position detecting method is not limited to this, and the rotary encoder or the magnetic sensor is used. It can also be obtained by the method.

Further, the present embodiment discloses that the chroma key method is used when the synthetic key signal generation unit 808 generates the synthetic key signal 818 and when the image generation unit 803 generates the generated key signal 820. However, the synthetic key signal generation method is not limited to this, and the synthetic key signal and the generated key signal can be generated by area extraction by image recognition.

The image synthesizing unit of the third, fourth, and fifth embodiments may be configured to output the synthetic pixel data and the synthetic depth value corresponding to the synthetic pixel data.

[0130]

As described above, according to the first aspect of the invention, the shape and the attribute data of the three-dimensional geometric shape model are also input with the shooting condition output from the image pickup section and the position direction output from the image pickup position detection section as inputs. An image generation unit that generates an image and outputs generated pixel data, and the real shot pixel data of the image pickup unit is delayed by the processing time of the image generation unit, and the synthesis target is based on the key that is the given identification reference of the synthesis target region. A synthetic key signal generation unit that outputs a synthetic key signal for identifying whether it is a region and real pixel data delayed corresponding to the synthetic key signal, and an image of the real pixel data and the generated pixel data based on the synthetic key signal. By providing an image synthesizing unit that outputs synthetic pixel data, a CG image matched with the real shot image is generated, and key signal detection is performed on the real shot image in which the real shot image and the CG image are synchronized. It is real-time key synthesis.

According to the second aspect of the present invention, the photographing condition output from the image pickup unit and the position direction output from the image pickup position detection unit are used as inputs, and the shape and attribute data of the three-dimensional geometric shape model and the given synthesis target region are combined. An image generating unit that generates an image based on a key that is an identification reference and outputs a generated pixel data and a generated key signal that identifies whether the generated pixel data is a combination target area, and an image generation of the actual pixel data of the imaging unit By providing an image synthesizing unit that delays the processing time of the image processing unit and delays the generated pixel data based on the generated key signal to synthesize the image, and output the synthesized pixel data, the CG that is matched with the photographed image. An image is generated, and real-time key combination can be performed in which a key signal is detected in a CG image in which a real image and a CG image are synchronized.

According to the third aspect of the invention, the depth information detecting section for measuring the position of the object to be photographed by the image pickup section and outputting the actually photographed depth value in synchronization with the image pickup section, the photographing condition and the image pickup position detection outputted from the image pickup section. An image generation unit that generates an image based on the shape and attribute data of the three-dimensional geometric shape model with the position and direction output from the unit as an input, and outputs the generated pixel data and the generated depth value of the generated pixel data, and an imaging unit Of the real shot pixel data of 1 is delayed by the processing time of the image generation unit, the real shot depth value is delayed by the processing time of the image generation unit, and the real shot pixel data and the generated pixel data are image-synthesized based on the delayed real shot depth value and the generated depth value. By providing the image synthesizing unit for outputting the synthetic pixel data, a CG image matched with the real shot image is generated, and the real time Z buffer in which the real shot image and the CG image are synchronized. Adult can.

According to the fourth aspect of the present invention, the depth information detecting section for measuring the position of the object photographed by the image pickup section and outputting the actual depth value in synchronization with the image pickup section, and the position direction of the image pickup section in synchronization with the image pickup section. The image pickup position detection unit to detect, the image pickup condition output from the image pickup unit and the position direction output from the image pickup position detection unit as input, image generation based on the shape and attribute data of the three-dimensional geometric shape model The image generation unit that outputs the data and the generated depth value of the generated pixel data, and the real pixel data of the imaging unit is delayed by the processing time of the image generation unit, and the real depth value is delayed by the processing time of the image generation unit. A synthetic key signal generation unit that outputs a synthetic key signal that identifies whether the region is a synthetic target region based on the key that is the identification criterion of the synthetic target region and the real pixel data that is delayed and that corresponds to the synthetic key signal; Synthesis CG that is matched with the actual shot image by including an image synthesizing unit for synthesizing the image of the actual shot pixel data and the generated pixel data based on the delayed signal and the delayed actual shot depth value and the generated depth value. An image is generated, and in real time synchronized with the real image and the CG image by using the real depth value and the generated depth value which is the depth value of the CG image, and the synthetic key signal which is a key generated from the real image side. It is possible to perform composite composition of Z buffer composition and key composition.

According to the fifth aspect of the invention, the depth information detecting section for measuring the position of the subject to be photographed by the image pickup section and outputting the actual photographed depth value in synchronization with the image pickup section, the photographing condition output from the image pickup section and the image pickup position detection. The image is generated based on the shape and attribute data of the 3D geometric shape model and the given key that is the identification criterion of the target area for synthesis, and the generated pixel data and the generated pixel data are combined. An image generation unit that outputs a generated key signal that identifies whether the region is a target region and a generated depth value of pixel data, and delays the real shot pixel data of the image pickup unit by the processing time of the image generation unit, and sets the real shot depth value to the image generation unit. The image combination that delays the processing time of, and generates the generated key signal, delays the actual shooting depth value, and the delayed actual shooting pixel data and the generated pixel data based on the generated depth value, and outputs the combined pixel data. By having a part, CG image is generated in which a well-aligned with the real image, the photographed depth value and C
By using the generated depth value which is the depth value of the G image and the generated key signal which is the key signal generated from the CG image side, the real-time image and the CG image are synchronized in real time for the Z buffer composition and the key composition. Complex synthesis is possible.

According to the sixth aspect of the invention, the depth information detecting section for measuring the position of the subject to be photographed by the image pickup section and outputting the actual photographed depth value in synchronization with the image pickup section, the photographing condition output from the image pickup section and the image pickup position detection. The image is generated based on the shape and attribute data of the 3D geometric shape model and the given key that is the identification criterion of the target area for synthesis, and the generated pixel data and the generated pixel data are combined. An image generation unit that outputs a generated key signal for identifying whether it is a target region and a generated depth value of pixel data, and a photographed image that is output with the actual pixel data output of the image pickup unit as input and delayed by the processing time of the image generation unit. A delay unit and a depth information delay unit that receives the output of the depth information detection unit as an input and delays the output by the processing time of the image generation unit and outputs the captured image delay unit as an input A synthetic key signal generator that outputs the original synthesis target region in which either corresponds to the synthesis key signal and the synthetic key signals identifying photographed pixel data key is,
An image synthesizing unit for synthesizing image data of real pixel data and generated pixel data, which are delayed based on the synthetic key signal, the generated key signal, the delayed actual shooting depth value and the generated depth value, and which correspond to the synthetic key signal, and output the synthetic pixel data. As a result, a CG image matched with the live-action image is generated, and the real-life depth value and the generated depth value that is the depth value of the CG image, the combined key signal that is the key signal generated from the real-image side, and the CG image side. By using the generated key signal which is the key signal generated from, the real-time image and the CG image can be combined in real time in combination with Z buffer composition and key composition.

In the seventh invention, the image synthesizing section of the third, fourth or fifth invention is replaced with an image synthesizing section which outputs synthetic pixel data and a synthetic depth value corresponding to the synthetic pixel data. Alternatively, depth values can be added to the output of the invention of 5.

In the eighth invention, the image synthesizing unit of the sixth invention is added with the background pixel data and the background depth value as an input and replaced with the image synthesizing unit which outputs the synthetic pixel data and the synthetic depth value corresponding thereto. Then, the depth value can be added to the output of the sixth invention.

In the ninth invention, an illumination condition extraction unit for detecting the illumination condition for the subject and outputting the illumination condition data to the image generation unit is further provided, and the image is generated in accordance with the illumination condition data input from the illumination condition extraction unit. By providing the image generation unit for generating, the illumination conditions of the real image and the CG image can be matched in the composite image.

[Brief description of drawings]

FIG. 1 is a configuration diagram of an image processing apparatus according to a first embodiment of the present invention.

FIG. 2 is a configuration diagram of an image processing apparatus according to a second embodiment of the present invention.

FIG. 3 is a configuration diagram of an image processing apparatus according to a third embodiment of the present invention.

FIG. 4 is a configuration diagram of an image processing apparatus according to a fourth embodiment of the present invention.

FIG. 5 is a configuration diagram of an image processing apparatus according to a fifth embodiment of the present invention.

FIG. 6 is a configuration diagram of an image processing apparatus according to a sixth embodiment of the present invention.

FIG. 7 is a configuration diagram of an image processing apparatus according to a seventh embodiment of the present invention.

FIG. 8 is a configuration diagram of an image processing apparatus according to an eighth embodiment of the present invention.

FIG. 9 is a diagram showing a conventional chroma key composition processing method.

FIG. 10 is a diagram showing a processing method of a conventional Z buffer method.

FIG. 11 is a diagram showing a distance measuring method using a magnetic sensor according to the present embodiment.

FIG. 12 is a diagram showing a time delay of a CG image generated in real time with respect to a photographed image.

FIG. 13 is a diagram showing a hidden surface erasing processing method using the chroma key processing and the Z buffer method according to the fourth embodiment.

FIG. 14 is a diagram showing a hidden surface erasing processing method using the chroma key processing and the Z buffer method of the sixth embodiment.

FIG. 15 is a diagram showing a hidden surface erasing processing method using the chroma key processing and the Z buffer method according to the seventh embodiment.

[Explanation of symbols]

 101 Image pickup unit 102 Image pickup position detection unit 103 Image generation unit 104 Real image delay unit 105 Synthetic key signal generation unit 106 Image synthesis unit 107 Image output unit 201 Image pickup unit 202 Image pickup position detection unit 203 Image generation unit 204 Real image delay unit 205 image Compositing unit 206 Image output unit 301 Imaging unit 302 Imaging position detection unit 303 Image generation unit 304 Real image delay unit 305 Image composition unit 306 Depth information detection unit 307 Depth information delay unit 308 Image output unit 401 Imaging unit 402 Imaging position detection unit 403 Image generation unit 404 Real image delay unit 405 Image synthesis unit 406 Depth information detection unit 407 Depth information delay unit 408 Synthetic key signal generation unit 409 Image output unit 501 Imaging unit 502 Imaging position detection unit 503 Image generation unit 504 Real image delay unit 505 Image synthesizer 506 Depth information detection unit 507 Depth information delay unit 508 Image output unit 601 Imaging unit 602 Imaging position detection unit 603 Image generation unit 604 Real image delay unit 605 Image synthesis unit 606 Depth information detection unit 607 Depth information delay unit 608 Synthetic key signal generation Part 609 Image output part 701 Imaging part 702 Imaging position detection part 703 Image generation part 704 Real image delay part 705 Image combination part 706 Depth information detection part 707 Depth information delay part 708 Synthetic key signal generation part 709 Image output part 801 Imaging part 802 Imaging position detection unit 803 Image generation unit 804 Real image delay unit 805 Image synthesis unit 806 Depth information detection unit 807 Depth information delay unit 808 Synthetic key signal generation unit 809 Image output unit 829 Lighting condition extraction unit

Claims (9)

[Claims] 1. An image pickup section for picking up an image of a subject and outputting real shot pixel data and shooting conditions, an image pickup position detecting section for detecting a position direction of the image pickup section in synchronization with the image pickup section, and an output from the image pickup section. An image generation unit that inputs an image capturing condition and a position direction output from the image capturing position detection unit as an input, generates an image based on the shape and attribute data of a three-dimensional geometric shape model, and outputs generated pixel data; Based on a key which is the identification reference of the synthesis target area given as an input, the real image delay unit that outputs the real image pixel data output as the input and delays the output by the processing time of the image generation unit A composite key signal generating section for outputting a composite key signal for identifying whether the area is a composition target area and actual pixel data corresponding to the composite key signal; an output of the composite key signal generating section; and an image generating section. An image combining unit that inputs the input force and outputs the combined pixel data by combining the imaged pixel data and the generated pixel data based on the combined key signal; and an image output unit that outputs the image combining unit output as an image. An image processing apparatus comprising: 2. An image pickup section for picking up an image of a subject and outputting real shot pixel data and shooting conditions; an image pickup position detecting section for detecting a position direction of the image pickup section in synchronization with the image pickup section; Image generation and generation based on the shooting conditions and the position and direction output from the image pickup position detection unit, based on the shape and attribute data of the three-dimensional geometric shape model and the given key that is the identification reference of the synthesis target area An image generation unit that outputs pixel data and a generated key signal that identifies whether the generated pixel data is a combining target region, and an actual pixel data output of the image pickup unit as input, and delay the processing time of the image generation unit. A live-action image delay unit for outputting, an input of the output of the live-action image delay unit and an output of the image generation unit, and an image combination of the real-image pixel data and the generated pixel data based on the generation key signal. An image processing apparatus comprising: an image synthesizing unit that outputs raw data; and an image output unit that receives the output of the image synthesizing unit as an input and outputs as an image. 3. An image pickup section for picking up an image of a subject and outputting real shot pixel data and shooting conditions, and depth information detection for measuring the position of the subject shot by the image pickup section and outputting a real shot depth value in synchronization with the image pickup section. Unit, an imaging position detection unit that detects the position direction of the imaging unit in synchronization with the imaging unit, a shooting condition output from the imaging unit, and a position direction output from the imaging position detection unit as a third order An image generation unit that generates an image based on the shape and attribute data of the original geometric shape model and outputs the generated pixel data and the generated depth value of the generated pixel data, and the actual pixel data output of the imaging unit as an input. A live-action video delay unit that delays and outputs the processing time of the image generation unit; a depth information delay unit that inputs the output of the depth information detection unit and delays and outputs the processing time of the image generation unit; The image delay unit output, the image generation unit output, and the depth information delay unit output are input, and the actual shot pixel data and the generated pixel data are image-synthesized based on the actual shot depth value and the generated depth value to output synthetic pixel data. An image processing apparatus comprising: an image synthesizing unit that outputs the image, and an image output unit that inputs the synthetic image data output from the image synthesizing unit and outputs the image as a video. 4. An image pickup section for photographing a subject and outputting real shot pixel data and shooting conditions, and depth information detection for measuring the position of the subject photographed by the image pickup section and outputting a real shot depth value in synchronization with the image pickup section. Unit, an imaging position detection unit that detects the position direction of the imaging unit in synchronization with the imaging unit, a shooting condition output from the imaging unit, and a position direction output from the imaging position detection unit as a third order An image generation unit that generates an image based on the shape and attribute data of the original geometric shape model and outputs the generated pixel data and the generated depth value of the generated pixel data, and the actual pixel data output of the imaging unit as an input. A live-action video delay unit that delays and outputs the processing time of the image generation unit; a depth information delay unit that inputs the output of the depth information detection unit and delays and outputs the processing time of the image generation unit; Outputs a composite key signal for identifying whether or not the target area is a composite target area based on a key, which is an identification reference of the composite target area, to which the output of the image delay unit is input, and the actual pixel data corresponding to the composite key signal. A synthetic key signal generating unit, and the synthetic key signal generating unit output, the image generating unit output, and the depth information delay unit output as inputs, and the real image based on the synthetic key signal, the actual depth value, and the generated depth value. An image processing apparatus including an image synthesizing unit for synthesizing image data of pixel data and the generated pixel data and outputting synthetic pixel data, and an image output unit for inputting and outputting synthetic image data output from the image synthesizing unit as an image. . 5. An image pickup section for photographing a subject and outputting real shot pixel data and shooting conditions, and depth information detection for measuring the position of the subject photographed by the image pickup section and outputting a real shot depth value in synchronization with the image pickup section. Unit, an imaging position detection unit that detects the position direction of the imaging unit in synchronization with the imaging unit, a shooting condition output from the imaging unit, and a position direction output from the imaging position detection unit as a third order An image generated based on the shape and attribute data of the original geometric shape model and a key that is a given identification criterion of the synthesis target area, and the generated pixel data and the generated key that identifies whether the generated pixel data is the synthesis target area. An image generation unit that outputs a signal and a generated depth value of the pixel data; and a photographed image delay unit that receives the photographed pixel data output of the image pickup unit as an input and delays the image generation unit by the processing time of the image generation unit and outputs the image. The depth information detecting unit output is input, and the depth information delaying unit delays and outputs the processing time of the image generating unit, the actual image delay unit output, the image generating unit output, and the depth information delaying unit input are input. An image synthesizing section for synthesizing the image of the photographed pixel data and the generated pixel data based on the generated key signal, the photographed depth value, and the generated depth value, and outputting synthesized pixel data, and a synthesized image output from the image synthesizing section. An image processing apparatus comprising an image output unit that receives data and outputs it as a video. 6. An image pickup section for picking up an image of a subject and outputting real shot pixel data and shooting conditions, and depth information detection for measuring the position of the subject shot by the image pickup section and outputting a real shot depth value in synchronization with the image pickup section. Unit, an imaging position detection unit that detects the position direction of the imaging unit in synchronization with the imaging unit, a shooting condition output from the imaging unit, and a position direction output from the imaging position detection unit as a third order An image generated based on the shape and attribute data of the original geometric shape model and a key that is a given identification criterion of the synthesis target area, and the generated pixel data and the generated key that identifies whether the generated pixel data is the synthesis target area. An image generation unit that outputs a signal and a generated depth value of the pixel data; and a photographed image delay unit that receives the photographed pixel data output of the image pickup unit as an input and delays the image generation unit by the processing time of the image generation unit and outputs the image. A depth information delay unit that receives the output of the depth information detection unit and outputs the image after delaying the processing time of the image generation unit, and a key that is the input of the output of the actual image delay unit and that is the identification reference of the synthesis target area And a synthetic key signal generating unit that outputs a synthetic key signal for identifying whether the region is a synthetic target region and real pixel data corresponding to the synthetic key signal, an output of the synthetic key signal generation unit, and an output of the image generation unit And the output of the depth information delay unit and the background pixel data of the given background image as input, the synthesized key signal, the synthesized key signal, the generated key signal, the photographed depth value, and the photographed pixel based on the generated depth value. An image synthesizing unit for synthesizing the image of the data and the generated pixel data and outputting the synthesized pixel data; Image processing apparatus including an image output unit for 7. The image processing apparatus according to claim 3, wherein the image synthesizing unit outputs synthetic pixel data and a synthetic depth value corresponding to the synthetic pixel data. 8. The image synthesizing unit additionally receives the background pixel data and the background depth value of a given background image, and outputs the synthetic pixel data and a synthetic depth value corresponding to the synthetic pixel data. Image processing device. 9. An illumination condition extraction unit that detects an illumination condition for a subject and outputs illumination condition data to an image generation unit, wherein the image generation unit outputs the illumination condition data output from the illumination condition extraction unit. The image processing apparatus according to claim 1, wherein the image processing apparatus generates an image according to the above.