JP2004282784A - Video processing apparatus and its method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To produce a synchronized video by compositing a real space and a virtual object only in a chroma key composition target area without depending on a position and the attitude of an image pickup means. <P>SOLUTION: Real video imaged with a camera 101 is outputted to a chroma key compositing apparatus 106 via a work station 102 as a fused video (foreground). When it is desired to indicate the virtual object in front of the real video, virtual object video watched from the position and attitude of the camera 101 is produced on the basis of an unillustrated table wherein data of the virtual object and coordinate values to dispose the virtual object are described, and position and attitude data of the camera 101, and composited with the real image to produce fused video, and outputted to the chroma key compositing apparatus 106. On the other hand, the work station 103 produces video of the background (CG (computer graphic) video) and outputs it to the chroma key compositing apparatus 106, and the fused video and the CG video are used to perform chroma key composition. Thus, mixed reality video is produced, outputted to a monitor 107 and displayed. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a video processing device and a video processing method for performing a video synthesis process of a real space and a virtual object.

  In order to experience Mixed Reality (hereinafter, MR), a computer generates an image of a virtual object (virtual space), and combines the generated image of the virtual object with an image of a real space captured by a camera. It is necessary to generate a synthesized video (mixed reality video). When generating a mixed reality video, a mixed reality video may be generated by combining a virtual object video with a background of a real object captured by a camera. Conventionally, the method described below has been used to generate such a mixed reality video. That is, the background plate of the chroma key color is arranged on the background of the real object, and the real object is imaged so that the background becomes the background plate. Then, chroma key synthesis of the captured image and the image of the virtual object is performed.

<Change in chroma key color>
However, conventionally, depending on the position and orientation of the light source and the position and orientation of the camera, the chroma key color of the background to be imaged changes, and the chroma key synthesis has not been successful. In other words, when the light from the light source is reflected by the background plate, if the imaging of the real object including the background plate is performed from the direction of reflection, the background plate is imaged more whitish. In this case, it is difficult to perform chroma key synthesis due to a change in chroma key color.
<Chroma key synthesis of parts not to be subjected to chroma key synthesis>
Conventionally, when there is a real object partially having a chroma key color other than the background (for example, when the real object is a building, when the window color is close to the chroma key color), a portion that is not a target of the chroma key synthesis is also subjected to the chroma key synthesis. Is done.
<Synchronization of generation when generating mixed reality video>
In addition, conventionally, when generating a mixed reality video, the timing of generating a foreground video and the timing of generating a video of a virtual object used as a background are different from each other due to processing speed. It is difficult to generate a realistic image.

  The present invention has been made in view of the above problems, and generates a synchronized video by synthesizing a real space and a virtual object only in a chroma key synthesis target area without depending on the position and orientation of an imaging unit. The purpose is to do.

In order to achieve the object of the present invention, for example, a video processing device of the present invention has the following configuration. That is, a video processing device that performs a video synthesis process of a real space and a virtual object,
Imaging means for imaging the real space;
Measuring means for measuring the position and orientation of the imaging means;
Foreground image generation means for generating a foreground image including the real space imaged by the imaging means, based on the position and orientation of the imaging means by the measurement means,
A background image generation unit configured to generate a foreground background image by the foreground image generation unit based on a position and orientation of the imaging unit by the measurement unit;
From the table in which the chroma key color corresponding to the position and orientation of the imaging unit is described, the chroma key color corresponding to the position and orientation of the imaging unit is specified by the measuring unit, and the foreground image generation unit is specified using the specified chroma key color. Synthesizing means for performing a chroma key synthesizing process of the foreground image by the background image generation means by the background image generation means,
A synchronizing signal generating unit that generates a synchronizing signal for inputting the foreground image and the background image to be synthesized by the synthesizing unit to the synthesizing unit.
The foreground image generation unit and the background image generation unit perform weight processing for a predetermined time set independently of each other, thereby synchronizing the foreground and background images synchronized in accordance with the synchronization signal. It is characterized by inputting to means.

  According to the configuration of the present invention, it is possible to generate a synchronized video by synthesizing the real space and the virtual object only in the chroma key synthesis target area without depending on the position and orientation of the imaging unit.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment]
FIG. 1 shows a schematic configuration of a video processing device according to the present embodiment. In the present embodiment, a camera 101 captures an image of a real object with a chroma key background plate as a background. FIG. 3 shows the state of this imaging.

  Reference numeral 301 denotes a chroma key background plate, 302 and 303 denote real objects, and the camera 101 images the real objects 302 and 303 so that the background plate 301 becomes a background.

  Returning to FIG. 1, reference numerals 102 and 103 denote workstations for graphic processing, which can execute software for performing graphic processing. Note that 102 and 103 may be general computers that can execute the above software. FIG. 4 shows the internal configuration of the workstations 102 and 103.

  Reference numeral 401 denotes a CPU that controls the entire workstation using program codes and data stored in the RAM 402 and the ROM 403, and also performs various video processing and matrix calculations. A RAM 402 has an area for temporarily storing program codes and data loaded from the external storage device 404 by the CPU 401, and also has a work area used when the CPU 401 performs various processes. Reference numeral 403 denotes a ROM which stores a workstation startup program, settings after startup, and the like, and also stores character codes and the like. An external storage device 404 stores program codes and data read from a storage medium such as a CD-ROM or a floppy (registered trademark) disk. An operation unit 405 is configured by a pointing device such as a keyboard and a mouse, and is an interface for a user to input various instructions. A display unit 406 is configured by a CRT, a liquid crystal screen, or the like, and is capable of displaying various video generation results, system messages, and the like. An interface (I / F) 407 can be connected to peripheral devices and other workstations, and can transmit and receive data. An NCU 408 can transmit and receive data to and from another device via a communication line. A bus 409 connects the above-described units.

  Returning to FIG. 1, reference numeral 104 denotes a position and orientation measurement device that measures the position and orientation of the camera 101 and outputs position and orientation data as a measurement result. Reference numeral 105 denotes a synchronization signal generator which outputs a synchronization signal at regular time intervals. Reference numeral 106 denotes a chroma key synthesizing device that performs chroma key synthesizing of the input images and outputs a mixed reality image as a synthesizing result. The chroma key color to be used (the chroma key color of the background plate 301 in the example shown in FIG. 3) needs to be registered in the chroma key synthesizing device 106 in advance so that the chroma key synthesizing device 106 can recognize the chroma key color. A monitor 107 can display an input video.

  A general flow of processing performed by the video processing device having the above configuration will be described with reference to FIG. The position and orientation of the camera 101 are measured by the position and orientation measurement device 104 and output to the workstation 102 and the workstation 103 as position and orientation data of the camera 101, respectively.

  The real image captured by the camera 101 is input to the workstation 102 (in the RAM 402), and output to the chroma key synthesizing device 106 as a fused image (foreground). Note that virtual object data is stored in the external storage device 404 of the workstation 102 (workstation 103), and it is necessary to display a virtual object image before the real image input from the camera 101. (Hereinafter, this virtual object is referred to as a foreground virtual object, and the image of the foreground virtual object is referred to as a foreground virtual object image). The data is loaded from the device 404 to the RAM 402. Further, a table (not shown) in which coordinate values for arranging the foreground virtual object are loaded is also loaded. Then, based on the position and orientation data of the camera 101 input from the position and orientation measurement device 104, a viewing transformation matrix of the foreground virtual object is generated, and a foreground virtual object image viewed from the position and orientation of the camera 101 is generated. Then, the generated foreground virtual object image and the real image are combined to generate a fusion image. As described above, the generated fused video is output to the chroma key synthesizing device 106.

  On the other hand, when the position and orientation data of the camera 101 is input into the workstation 103, the workstation 103 generates a background image (CG image) based on the position and orientation data. A method of generating a background image based on the position and orientation data of the camera 101 will be described later in detail. As described above, the generated CG video is output to the chroma key synthesizing device 106.

  The chroma key synthesizing device 106 performs chroma key synthesis using the input fusion image and CG image to generate a mixed reality image. The chroma key synthesizing process performed by the chroma key synthesizing device 106 will be described with reference to FIG.

  Reference numeral 501 denotes a fusion image input from the workstation 102, and reference numeral 502 denotes a CG image input from the workstation 103. The fusion image 501 includes images 501a and 501b of a real object, a foreground virtual object image 501c, and an image 501d of a chroma key background plate. When the chroma key synthesis of the fusion image 501 and the CG image 502 is performed, a mixed reality image 503 is generated in which the image of the corresponding region of the CG image 502 is synthesized in the region of the image 501d of the background board in the fusion image 501.

  Then, the mixed reality video generated by the chroma key synthesizing device 106 by the method described above is output to the monitor 107 and displayed.

  The above components (the camera 101, the workstation 102, the workstation 103, the position and orientation measuring device 104, and the chroma key synthesizing device 106) operate in synchronization with the synchronization signal output from the synchronization signal generator 105. That is, in synchronization with the synchronizing signal, the output of the actual image from the camera 101, the output of the fused image from the workstation 102, the output of the CG image from the workstation 103, the position and orientation of the camera 101 by the position and orientation measurement device 104 The measurement and output of the position and orientation data, and the start of the chroma key synthesizing process by the chroma key synthesizing device 106 are performed. Hereinafter, the synchronization of the operations of the above-described units will be described in more detail.

  As described above, the output of the actual image from the camera 101 and the output of the position and orientation data of the camera 101 from the position and orientation measurement device 104 are performed in response to the synchronization signal. In order to synchronize the photographed image to be taken into the RAM 402 of the camera 101) and the position and orientation data of the camera 101, the time required for capturing the photographed image is x frames. The position and orientation data of the camera 101 is output. In this way, the real image from the camera 101 and the position and orientation data of the camera 101 from the position and orientation measurement device 104 are synchronized and input to the RAM 402 in the workstation 102.

  Alternatively, the real image and the position and orientation data of the camera 101 are input to the workstation 102 from the camera 101 and the position and orientation measurement device 104, respectively, and the position and orientation data of the camera 101 are stored in the RAM 402 to generate a fused image. At this time, of the position and orientation data in the RAM 402, the data before x frames may be used. It should be noted that the time required for capturing the actual video is measured in advance.

  In addition, the times required to generate the fusion video and the CG video are often different between the workstation 102 and the workstation 103. FIG. 6 shows a timing chart of the video generation processing and output of the workstation 102 and the workstation 103 in this case.

  In the figure, t1 is the time required for the workstation 102 to generate the fused image, WT1 is the wait time described later, t2 is the time required for the workstation 103 to generate the CG image, WT2 is the wait time described later, t3 indicates the time required for the chroma key synthesizing device 106 to perform the chroma key synthesizing process.

  The WT1 and the WT2 are for keeping the difference in time required for one repetitive calculation between the workstation 102 and the workstation 103 within one frame and waiting for the timing when the chroma key synthesizing process by the chroma key synthesizing device 106 ends. Are the wait times provided for the workstations 102 and 103, respectively. By providing the wait times WT1 and WT2 in this way, the fused video and the CG video are output from the workstation 102 and the workstation 103 in synchronization with each other in accordance with the synchronization signal, and are input to the chroma key synthesizing device 106.

  Next, a method of generating a background image will be described. Since the background image is often a distant view, reproduction of a fine three-dimensional shape is not so necessary. Therefore, it is preferable that the time required to generate the background image be shorter. As a method of generating a background according to the present embodiment, as shown in FIG. 7, a hemisphere 701 including a foreground 702 is virtually set, and an image previously captured by a fisheye lens or the like is mapped inside the hemisphere 701. .

  As a result, based on the position and orientation data of the camera 101 input from the position and orientation measurement device 104, an image inside the hemisphere 701 that can be seen by the position and orientation of the camera 101 is generated and used as a background. The shape including the foreground 702 is not limited to a hemisphere, and may be a cylinder having a closed upper surface. Note that the above-described data (background data) of the hemisphere 701 is stored in the external storage device 404 of the workstation 103, and is read as needed.

  In each of the above processes, flowcharts of the processes in the workstation 102 and the workstation 103 are shown in FIGS. 8 and 9, respectively, and will be described below. The program codes according to the respective flowcharts are stored in the respective external storage devices 404 of the respective workstations, read out to the RAM 402 as needed, and read out by the respective CPUs 401 in the respective workstations. Will be executed.

  First, the processing in the workstation 102 will be described. First, the process waits for a synchronization signal from the synchronization signal generator 105 (step S801). When the process is started, first, a photographed image is captured from the camera 101 (step S802). Then, the position and orientation data of the camera 101 is input from the position and orientation measurement device 104 (step S803). The order of the processes in step S802 and step S803 may be reversed or may be performed in parallel. Then, the captured real image is drawn in the RAM 402 (step S804). A viewing transformation matrix for generating a foreground virtual object image is calculated based on the input position and orientation data of the camera 101 (position and orientation data delayed by the number of frames measured in advance) (step S805). Then, the foreground virtual object video generated based on the viewing transformation matrix is synthesized with the real video to generate a fusion video (step S806).

  Finally, the weight is waited for the wait time WT1 (step S807), and thereafter, the fused image is output to the chroma key synthesizing device 106 (step S808). After that, the process moves to step S801. If the foreground virtual object image is not drawn, the processing in steps S805, S806, and S803 is not necessary.

  Next, processing in the workstation 103 will be described with reference to FIG.

  As in step S801, synchronization is waited (step S901). When the process is started, the position and orientation data of the camera 101 is input as in step S803 (step S902), and a background CG is generated based on the position and orientation data. (Step S904). If the weight is equal to or greater than WT2, the background CG image is output, and the process proceeds to step S901.

  FIG. 10 shows a flowchart of the processing in the chroma key synthesizing device 106, which will be described below. The program code according to the flowchart of FIG. 5 is stored in a memory such as a RAM or a ROM (not shown) in the chroma key synthesizing device 106, and is read and executed by a CPU (not shown).

  First, after waiting for synchronization (step S1001), if processing is started, first, a fused image and a CG image are input (step S1002). Next, a chroma key synthesizing process is performed (step S1003), and the resulting mixed reality video is output to the monitor 107 (step S1004).

  In the present embodiment, the workstation that generates the fusion video and the CG video is separate, but may be performed in the same workstation. At that time, the program code according to the flowcharts shown in FIGS. 8 and 9 is stored in the external storage device of the workstation.

  The video processing device according to the present embodiment as described above can generate a synchronized mixed reality video.

[Second embodiment]
In the first embodiment, depending on the position and orientation of the camera 101 and the position and orientation of a light source (not shown), the color of the background plate (301 in FIG. 3, 501d in FIG. 5) changes in an image captured by the camera 101. In some cases. In this case, assuming that the position and orientation of the light source are fixed, it is necessary to change the chroma key combination parameter in the chroma key combination device 106 in real time according to the change in the position and orientation of the camera 101. In the present embodiment, such a chroma key synthesizing device 106 will be described. Note that the schematic configuration of the video processing apparatus according to the present embodiment is such that the position and orientation measurement device 104 and the chroma key synthesizing device 106 are connected to each other in FIG. Is entered.

  First, the chroma key combination parameters are adjusted in advance according to the different camera positions and orientations, and stored as a table (chroma key combination parameter table). Elements of the chroma keying parameters include hue, clip, gain, hue range, saturation, etc., and need to be more strictly adjusted in the order described. FIG. 11 shows a configuration example of the chroma key combination parameter table.

  In the table 1100 of FIG. 10, a column 1101 in which the position data of the camera 101 is described, a column 1102 in which the posture data of the camera 101 is described, and the position and orientation of the camera 101 described in the columns 1101 and 1102 are shown. A column 1103 in which the specified chroma key combination parameter is described is included. Note that data is not always described in the columns 1101 and 1102. For example, when the position of the camera 101 is sufficiently distant from the background plate, the position data of the camera 101 is described in the column 1101. You don't have to. In this case, even if the chroma key combination parameter is determined only by the attitude data of the camera 101, the error is small.

  Next, referring to the chroma key synthesizing parameter table, a process of specifying the chroma key synthesizing parameter corresponding to the position and orientation data of the camera 101 is shown in FIG. 16 and will be described below.

  If the position and orientation data of the camera 101 is given (step S1601), the chroma key combination parameter table is referred to (step S1602), and this data is described in the chroma key combination parameter table (1101, column 1102 in the example of FIG. 11). If the data is (step S1603), a chroma key combination parameter corresponding to the data may be used (step S1604). However, if the position and orientation data of the camera 101 is not described in the chroma key combination parameter table (columns 1101 and 1102 in the example of FIG. 11) (step S1603), interpolation is performed using the described position and orientation data and the chroma key combination parameter. (Step S1605), it is possible to specify the chroma key combination parameter corresponding to the position and orientation data not described in the table (Step S1606). Although the method of interpolation is not particularly limited, for example, spline interpolation or the like may be used.

  According to the above method, the chroma key synthesizing device 106 according to the present embodiment can perform the chroma key synthesizing process even if the chroma key color to be imaged changes depending on the position and orientation of the camera 101.

[Third Embodiment]
When the real object includes a chroma key color, for example, when the real object is a building and the window color is a chroma key color, the corresponding portion of the background CG image is synthesized by the chroma key synthesis in the window portion. Will be. Therefore, in the present embodiment, a process of excluding a mask region from a target of chroma key synthesis even when a real object includes a region of a chroma key color (hereinafter, a mask region) will be described.

  First, the shape and position of the mask area in the real object are specified in a three-dimensional space and stored in the external storage device 404 in the workstation. Then, by converting the three-dimensional mask shape data using a viewing transformation matrix generated based on the position and orientation data of the camera 101, the shape and position of the mask area in the video to be subjected to the chroma key synthesis processing are specified. Therefore, by outputting the information of this area to the chroma key synthesizing device 106, this area can be set as a non-target area of the chroma key synthesizing process.

[Fourth embodiment]
The processing performed by the video processing device in the first embodiment can be realized by other configurations. FIG. 12 shows a schematic configuration of a video processing device according to the present embodiment.

  The devices used for each unit are the same as those in the first embodiment, but the connection types between the devices and the operation timing of each device are different. The flow of each process in the video processing device of the present embodiment will be described with reference to FIG.

  Background data is stored in the external storage device 404 of the workstation 102, and is read out to the RAM 402 by the CPU 401 as necessary. Therefore, the workstation 102 generates a CG image of a background based on the position and orientation of the camera 101 based on the position and orientation data of the camera 101 input from the position and orientation measurement device 104 in the same manner as in the first embodiment. Is output to the chroma key synthesizing device 106.

  In addition, the actual image captured by the camera 101 is directly input to the chroma key synthesizing device 106. Then, the chroma key synthesizing unit 106 chroma chroma synthesizes the input real image and the background CG image to generate a mixed reality image that does not include the foreground virtual object. The generated mixed reality video (not including the foreground virtual object) is input to the workstation 103. Data of the foreground virtual object is stored in the external storage device 404 of the workstation 103, and is read out to the RAM 402 by the CPU 401, and then read out from the position and orientation measurement device 104 based on the position and orientation data of the camera 101. Is generated and synthesized with the video input from the chroma key synthesizing device 106 to generate a mixed reality video, which is then output to the monitor 107.

  Further, similarly to the first embodiment, the above-described units (the camera 101, the workstation 102, the workstation 103, the position and orientation measuring device 104, and the chroma key synthesizing device 106) synchronize with the synchronization signal output from the synchronization signal generator 105. Working. That is, in synchronization with the synchronizing signal, the output of the real image from the camera 101, the output of the CG image from the workstation 102, the output of the mixed reality image from the workstation 103, the position of the camera 101 by the position and orientation measuring device 104 The posture measurement, the output of the position and posture data, and the start of the chroma key synthesizing process by the chroma key synthesizing device 106 are performed.

  Note that, in order to synchronize the real shot video from the camera 101 input to the chroma key synthesizing device 106 with the background CG image from the workstation 102, the real shot video from the camera 101 is stored in a memory (not shown) in the chroma key synthesizing device 106. The CG image is input at an appropriate time (ideally, the timing at which the CG image is input from the workstation 102 and the position and orientation of the camera 101 used to generate the CG image. (Difference time from the delayed timing). This delay time is measured in advance. For example, information (generation time information) of “generation time” is added to each of a real video image from the camera 101 and a background CG image from the workstation 102, and an image having the same generation time information is chroma keyed. The time lag input to the synthesizer 106 is measured. As a result, this shift becomes a delay time.

  The workstation 103 also stores the position and orientation data of the camera 101 from the position and orientation measurement device 104 in the RAM 402 in order to synchronize the image input from the chroma key synthesis device 106 with the position and orientation data input from the position and orientation measurement device 104. And a suitable time (ideally, the timing at which the position / posture data of the camera 101 from the position / posture measuring device 104 is input, and the video from the chroma key synthesizing device 106 using this position / posture data. It is used with a delay (difference time from the input timing).

  14 and 15 show flowcharts of the processing of the workstations 102 and 103, respectively, and will be described below. The program codes according to the respective flowcharts are stored in the respective external storage devices 404 of the respective workstations, read out to the RAM 402 as needed, and read by the respective CPUs 401 in the respective workstations. Read and executed.

  First, the processing in the workstation 102 will be described. First, the synchronization waits (step S1401). When the process starts, first, the position and orientation data of the camera 101 is input from the position and orientation measurement device 104 (step S1402). Based on the position and orientation data obtained by delaying the input position and orientation data by an appropriate time as described above, a background CG image is generated (step S1404). Then, the generated background CG image is output (step S1405).

  Next, processing in the workstation 103 will be described. First, synchronization is waited for (step S1501), and when the processing is started, first, a real image is captured from the camera 101 (step S1502). Then, the position and orientation data of the camera 101 is input from the position and orientation measurement device 104 (step S1503). The order of each process of step S1502 and step S1503 may be reversed or may be performed in parallel. Then, the captured real image is drawn in the RAM 402 (step S1504). Then, a viewing transformation matrix for generating a foreground virtual object image is calculated based on the position and orientation data delayed by the number of frames measured in advance among the input position and orientation data of the camera 101 (step S1505). The foreground virtual object image generated based on the obtained viewing transformation matrix is synthesized with the image from the chroma key synthesizing device 106 to generate a mixed reality image (step S1506). Then, the generated mixed reality video is output (step S1507).

  The flowchart of the process in the chroma key synthesizing device 106 is the same as the flowchart in FIG. 10 except that a real image and a background CG image are input in step S1002 and the chroma key synthesis of these input images is performed in step S1003.

  Note that the workstations 102 and 103 in the present embodiment use one processor (CPU 401) to receive position and orientation data of the camera 101 and control other processes. That is, the workstation 102 receives position and orientation data of the camera 101 and generates a background image. On the other hand, in the workstation 103, one processor receives the position and orientation data of the camera 101 from the position and orientation measurement device 104 and receives the video from the chroma key synthesizing device 106.

  However, other than that, a processor dedicated to receiving the position and orientation data of the camera 101 from the position and orientation measurement device 104 may be newly provided in each workstation. That is, the reception-only processor periodically receives the position and orientation data of the camera 101 from the position and orientation measurement device 104 and stores the data in the RAM 402 in each workstation. The time required for this accumulation is approximately equal to zero. However, each workstation also waits for the generation of a CG image and the input of an image from the chroma key synthesizing device 106. Therefore, it is necessary to provide a wait time (for waiting time) in the reception-only processor in each workstation. Alternatively, the process of “receiving the position and orientation data of the camera → accumulating in the RAM 402” may be simply repeated without weighting. When the past position and orientation data is used for the processing, the position and orientation data at the time used in the RAM 402 may be referred to.

  The method described above is divided into a processor dedicated to receiving video data and a processor dedicated to receiving video data, and can be adapted to delay video data.

[Fifth Embodiment]
In the above embodiment, for example, a luminance key combining technique may be used for combining the foreground and the background in addition to the chroma key combining. In the above-described embodiment, the mixed reality video is output to the monitor 107. However, the present invention is not limited to this. For example, the mixed reality video may be output to a video or the like and recorded.

[Sixth Embodiment]
In the above-described embodiment, the generation is synchronized when the mixed reality video is generated. However, there is a case where the synchronization process is not performed because there is no significant problem even if the synchronization is not performed. As a configuration of the video processing apparatus at that time, the synchronization signal generator 105 is not required in FIGS. 1, 2, 12, and 13, and it is not necessary to provide the above-described weights of the respective units.

[Other embodiments]
An object of the present invention is to supply a storage medium (or a recording medium) in which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or an apparatus, and a computer (or a CPU or an MPU) of the system or the apparatus. Can also be achieved by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the storage medium storing the program code constitutes the present invention. In addition, the computer executes the readout program code, so that not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instructions of the program code. It goes without saying that a part or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing.

  Further, after the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is executed based on the instruction of the program code. It goes without saying that the CPU included in the expansion card or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

  When the present invention is applied to the storage medium, the storage medium includes the storage medium described above (FIG. 8 and / or FIG. 9 and / or FIG. 10 and / or FIG. 14 and / or FIG. And / or the program code corresponding to the flowchart (shown in FIG. 16) will be stored.

FIG. 1 is a schematic diagram illustrating a schematic configuration of a video processing device according to a first embodiment of the present invention. FIG. 2 is a block diagram illustrating a general flow of processing performed by the video processing device according to the first embodiment of the present invention. FIG. 4 is a diagram illustrating a state in which a camera 101 captures an image of a real object with a chroma key background plate as a background. FIG. 3 is a diagram showing an internal configuration of workstations 102 and 103. FIG. 4 is a diagram illustrating a chroma key synthesizing process performed by a chroma key synthesizing device 106. FIG. 9 is a diagram illustrating a timing chart of a video generation process and an output performed by the workstations 102 and 103. FIG. 4 is a diagram illustrating a method of generating a background. 5 is a flowchart of a process performed by a workstation 102 according to the first embodiment of the present invention. 5 is a flowchart of a process performed by a workstation 103 according to the first embodiment of the present invention. 5 is a flowchart of a process performed by the chroma key synthesizing device according to the first embodiment of the present invention. It is a figure showing the example of composition of a chroma key combination parameter table. It is a figure showing the schematic structure of the picture processing device in a 2nd embodiment of the present invention. It is a block diagram showing a rough flow of processing which a picture processing device in a 2nd embodiment of the present invention performs. 9 is a flowchart of a process performed by the video processing device according to the second embodiment of the present invention. 9 is a flowchart of a process performed by the video processing device according to the second embodiment of the present invention. It is a flowchart of a process of referring to a chroma key combination parameter table and specifying a corresponding chroma key combination parameter when position and orientation data of the camera 101 is given.

Claims (7)

An image processing device that performs a process of synthesizing an image of a real space and a virtual object,
Imaging means for imaging the real space;
Measuring means for measuring the position and orientation of the imaging means;
A foreground image generation unit that generates a foreground image including a real space imaged by the imaging unit based on the position and orientation of the imaging unit by the measurement unit;
A background image generation unit that generates a foreground background image by the foreground image generation unit based on the position and orientation of the imaging unit by the measurement unit;
From the table in which the chroma key color corresponding to the position and orientation of the imaging unit is described, the chroma key color corresponding to the position and orientation of the imaging unit is specified by the measuring unit, and the foreground image generation unit is specified using the specified chroma key color. Synthesizing means for performing a chroma key synthesizing process of the foreground image by the background image generation means by the background image generation means,
A synchronizing signal generating unit that generates a synchronizing signal for inputting the foreground image and the background image to be synthesized by the synthesizing unit to the synthesizing unit.
The foreground image generation unit and the background image generation unit perform weight processing for a predetermined time set independently of each other, thereby synchronizing the foreground and background images synchronized in accordance with the synchronization signal. An image processing apparatus characterized by inputting to a means. When the synthesizing unit specifies the chroma key color from the table according to the position and orientation of the imaging unit by the measuring unit,
When the position and orientation of the imaging unit by the measurement unit is not described in the table, by interpolating using the position and orientation of the imaging unit described in the table and a chroma key color corresponding thereto, the 2. The video processing apparatus according to claim 1, wherein a chroma key color corresponding to a position and orientation of the imaging unit is specified by a measurement unit. When the foreground image generation unit provides a virtual object in front of the real space by the imaging unit,
By generating an image of the virtual object based on the position and orientation of the imaging unit by the measurement unit using the information on drawing of the virtual object stored in the predetermined storage unit, and combining the generated image with the captured real space. The image processing device according to claim 1, wherein the image processing device generates a foreground image. 4. The foreground image generation unit according to claim 3, wherein the foreground image generation unit generates a foreground image by using the measured position and orientation of the imaging unit synchronized with a real space image captured by the imaging unit. Video processing equipment. The background video generation means uses background data stored in a predetermined storage means,
The image processing apparatus according to claim 1, wherein the measuring unit generates an image of a background that can be seen in the position and orientation of the imaging unit. A video processing method for performing a video synthesis process of a real space and a virtual object,
A measurement step of measuring the position and orientation of the imaging means for imaging the real space,
A foreground image generation step of generating a foreground image including the real space imaged by the imaging unit based on the position and orientation of the imaging unit in the measurement step;
A background image generation step of generating a foreground background image by the foreground image generation step based on the position and orientation of the imaging unit in the measurement step;
From the table in which the chroma key colors according to the position and orientation of the imaging unit are described, the chroma key colors according to the position and orientation of the imaging unit in the measurement step are specified, and the foreground image generation step is performed using the specified chroma key colors. A synthesizing step of performing a chroma key synthesizing process of a foreground image by the background image generation process by the background image generation step,
A synchronizing signal generating step of synchronizing the foreground image and the background image to be synthesized in the synthesizing step and generating a synchronizing signal for synthesizing in the synthesizing step,
In the foreground image generation step and the background image generation step, by performing a wait process for a predetermined time set independently of each other, the foreground and background images synchronized in accordance with the synchronization signal are respectively synthesized. A video processing method used in a process. A computer-readable storage medium storing a program for causing a computer to execute the video processing method according to claim 6.

Images