JPH09181972A - Image composition device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image composition device which can bring a virtual world close to an actual world by precisely composting/displaying a real space image and a virtual space image through the use of a luminance signal or a hue signal instead of a blue back signal as a key signal for discriminating an area to which the image is fitted. SOLUTION: An image pickup means 7 picking up the image, an image generation means 4 which is separately provided from the image pickup means 7 and which generates the other image, and an image processing means 8 discriminating a part where a luminance change or a hue change more than a prescribed value timewisely occurs within the range of the image that the image pickup means 7 picks up and a part where the change does not occur and executing an image processing for fitting one part which is previously set in the two discriminated parts to the other image are provided. Then, the luminance signal or the hue signal is used instead of the blue back signal as the key signal for discriminating the area to which the other image is fitted in the image. Thus, the synthesized image where the really taken image and the virtual space image by means of CG coexist is generated.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus for generating a synthetic image by fitting a CG image into a part of a photographed image.

[0002]

2. Description of the Related Art Recently, various virtual experience devices using a system called virtual reality (VR), which is a technique for presenting a subject with an image of a virtual world that does not actually exist, have been developed. ing. Such a V
A typical example of the related art regarding R is schematically shown in FIG. The subject 1 wears a head-mounted image display device (HMD) 2 as shown in FIG. A detection device (not shown) for detecting the position of the viewpoint and the line-of-sight direction of the subject in the three-dimensional real space of the subject is provided near the subject's viewpoint of the image display device 2, and three-dimensionally near the subject's head of the image display device 2. A space sensor 3 is provided, and the detected viewpoint position data, line-of-sight direction data, and three-dimensional space data are input to the image generation device 4. The image generation device 4 performs a calculation based on the viewpoint position data, the line-of-sight direction data, the three-dimensional space data, and the image data from the virtual space image database 5, and a world (virtual space) viewed from the viewpoint position and the line-of-sight direction.
An image corresponding to the scene is generated by computer graphics (CG) and output to the image display device 2. In the case of an interactive device, the image data in the virtual space image database 5 is updated by an operation signal from the auxiliary input device 6.

In such a conventional virtual experience device, since the entire world viewed by the subject is a virtual image by CG, the objects around the subject and the limbs of the subject, which should be in the visual field, are not drawn. , It may be CG even if it is drawn. This is not desirable from the perspective of reality, which is the most important issue of virtual reality technology, that is, how the virtual world feels as if it were the real world. Further, when an object or the like existing in the vicinity is drawn in a high-definition image in order to improve the reality, the load in the drawing process of the image generation device becomes large, the moving image speed becomes slow, and the delay of the moving image display during the simulation, etc. Will lead to problems. In this case, even if the processing speed of the image generating apparatus is increased, the drawn image is a CG image, and therefore the reality of the photographed image (real space image) cannot be exceeded. In addition, by attaching the image display device 2 to an unspecified number of people and outputting the same CG image from the image generation device 4 to each of the image display devices 2, an unspecified number of people can simultaneously use this virtual experience device. When configured for use, a CG image of an individual person should be reflected in the image, but it is virtually impossible to represent it in CG, and in this case also the reality of the image is impaired. Get lost.

In order to solve such a problem, for example, as shown in FIG. 1, an image pickup device 7 for picking up a real space image (real image) above the viewpoint of the subject 1 of the head-mounted image display device 2. Is provided, and a technique of fitting one of the real image captured by the image capturing device 7 and the virtual space CG image generated by CG using the virtual space image database 5 and the image generating device 4 by the image synthesizing device 8 is provided. JP-A-3-226198 and JP-A-5-3248.
No. 00, Japanese Patent Application Laid-Open No. 6-28451, Japanese Patent Application Laid-Open No. 6-1
No. 76131, Japanese Patent Laid-Open No. 6-268943, and the like. In these conventional techniques, a chroma key combining process is used as a combining method when fitting the other into one of a photographed image and a CG image.

In the conventional chroma key composition processing, one of the photographed image and the CG image is used as a composition screen,
In order to fit the other of the photographed image and the CG image into the composition screen, discriminate the hue difference in the composition screen, extract the color signal as the key signal, and perform image processing in real time based on the color signal. The CG image and the subject of the real image are combined by. As the color signal,
For example, when synthesizing a person image, blue, which is a complementary color of the flesh color, is selected as the background color of the person image. As described above, the so-called "blue back" is generally used as the composition screen.

[0006]

However, in the above-mentioned conventional chroma key composition processing, for example, when the person who is the subject in the photographed image wears blue clothes, the CG also appears in the blue clothes. The image is inset. When "mis-synthesis" occurs, such as when an image is composited where it should not be composited, or when an image is not composited where it should be composited, the most important reality in virtual reality is It will be damaged.

The present invention has been made in view of the above-mentioned problems, and uses a luminance signal or a hue signal instead of a blue back signal as a key signal for determining an area into which an image is fitted, and a real space image and a virtual image are used. An object of the present invention is to provide an image synthesizing device that can bring a virtual world closer to the real world by accurately synthesizing and displaying a spatial image.

To this end, the image synthesizing apparatus according to claim 1 of the present invention comprises an image pickup means for picking up an image, an image generating means provided separately from the image pickup means for generating another image, and the image pickup means. In the range of the image picked up by the means, a portion in which a luminance change or hue change that is equal to or greater than a predetermined value in terms of time is discriminated from a portion in which no luminance change or hue change has occurred, and one of the determined two portions is set in advance. An image processing means for performing image processing for fitting the other image is provided.

In the chroma key processing by the blue background of the above-mentioned conventional example, erroneous combination occurs when the photographed image includes a subject wearing blue clothes as described above. As a result, when the virtual reality experience system is used by an unspecified large number of people such as various games and attractions, the subject's clothing is limited, but actually the subject's clothing is limited. Since it is difficult to do so, there is a problem that erroneous synthesis cannot be avoided. In the image synthesizing apparatus according to claim 1 of the present invention, the above problem is solved by using the luminance signal or the hue signal instead of the blue back signal as a key signal for determining a region in which another image is fitted. . That is, the portion corresponding to the blue area of the chroma key composition processing by the blue background of the above-mentioned conventional example is a portion where a luminance change or a hue change of a predetermined value or more with time occurs (or, conversely, a predetermined value or more with time. (A portion where no luminance change or hue change has occurred), and another image such as a CG image is fitted into that portion.

As a result, for example, a composite image in which a real image and a CG virtual space image are mixed is generated. Since this synthetic image can highly accurately incorporate an actual object or person in a three-dimensional real space into a virtual space image that is arbitrarily set from an image database or the like, it produces a high sense of realism and a high sense of reality. can do.

For the above-mentioned purpose, an image synthesizing apparatus according to a second aspect of the present invention comprises an image pickup means for picking up an image, an image generating means provided separately from the image pickup means for generating another image, and the image pickup means. Image processing means for performing image processing for fitting the other image into a portion in which a luminance change of a predetermined value or more occurs temporally within the range of the image captured by the means and having a specific color signal. It is characterized by comprising.

In the chroma key processing of the above-mentioned conventional example, when only the luminance change is used as the discrimination condition of the area in which another image is fitted, the following problems occur. That is, in the case of the determination condition that the area in which the image is fitted is a portion in which the luminance change is equal to or more than a predetermined value in time, the degree of the luminance change is small in the peripheral portion of the portion in which the luminance change is generated, but the luminance change is not Has occurred. Therefore, even if the boundary between the area where the image is fitted and the other area is clear, the boundary between the area where the brightness change occurs and the other area may be unclear. It is determined that the above-mentioned luminance change has occurred even in a portion where it is not desired to fit, and the image may be fitted. In the image synthesizing apparatus according to the second aspect of the present invention, in order to more surely determine the area in which the other image is fitted, the determination condition is set to "a portion in which a luminance change of a predetermined value or more occurs temporally. "And a part having a specific color signal",
Another image such as a CG image is fitted only to a portion satisfying these two conditions.

As a result, the portion where the luminance change desired to be fitted into the image is colored with a specific color, and
A desired composite image can be obtained by fitting an image around it and coloring an unwanted portion with a different color. Furthermore, even if the subject wears blue clothes when the specific color is the same blue as the conventional example, the luminance change of the blue clothes part should be less than a predetermined value. As a result, it is possible to prevent the problem of the above-described conventional example that an image is undesirably fitted into the blue clothing portion.

For the above-mentioned purpose, an image synthesizing apparatus according to a third aspect of the present invention comprises an image pickup means for picking up an image, an image generating means provided separately from the image pickup means for generating another image, and the image pickup means. Image processing for performing the image processing in which the other image is fitted to a portion in which a luminance change of a predetermined value or more occurs temporally within a range of the image captured by the means and a portion having a specific color signal And means.

In the chroma key processing of the above-mentioned conventional example, when the condition for discriminating the area in which another image is to be fitted is only a portion where a luminance change of a predetermined value or more occurs temporally, the luminance change occurs. Since the peripheral portion of the portion having a high reflectance and the portion having a high reflectance are affected by the above-mentioned luminance change, the degree of the luminance change may be small, but the luminance change may occur. Therefore, even if the boundary between the area where the image is fitted and the other area is clear, the boundary between the area where the brightness change occurs and the other area may be unclear. It is determined that the above-mentioned luminance change has occurred even in a portion where it is not desired to fit, and the image may be fitted.

In the image synthesizing apparatus according to the third aspect of the present invention, in order to further ensure the determination of the area in which the other image is fitted, the determination condition is "a luminance change of a predetermined value or more occurs over time. Existing portion "and" a portion excluding a portion having a specific color signal ", and another image such as a CG image is fitted only to the portion satisfying these two conditions. As a result, a desired composite image can be obtained by coloring a portion in which luminance change may occur even though the user does not want to fit the image with a specific color.

The above-mentioned composite image will be specifically described below with reference to an example. For example, when the screen is colored in blue to change the brightness and a red table made of wood, for example, is arranged in the vicinity of the screen, the brightness change may occur in the table in which the CG image is not desired to be fitted. Moreover, since this table is arranged in the vicinity of the screen, the blue color of the screen is reflected on this table, and the imaging means recognizes that the table has blue color in addition to the original red color of the table. In this case, the determination condition of the area in which the CG image is fitted is "a portion in which a luminance change of a predetermined value or more occurs temporally" and "a portion having a specific color signal"
Under the condition, when a specific color signal is blue,
The CG image is also set on the table. Therefore,
By setting the condition for determining the area in which the CG image is fitted to be "a portion where a luminance change of a predetermined value or more occurs temporally" and "a portion excluding a portion having a specific color signal", the specific color If, for example, is made red, the table includes red, so that the CG image does not have to be fitted.
In this way, if the area that may not be embedded in the CG image but is likely to be erroneously synthesized has a specific color, or if it does not have the specific color, it is colored. The desired composite image is obtained.

[0018]

Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 1 is a diagram showing the configuration of an image synthesizing apparatus according to the first embodiment of the present invention. In the present embodiment, as shown in FIG. 1, the position of the viewpoint and the line-of-sight direction in the three-dimensional real space of the subject are detected in the vicinity of the viewpoint of the subject 1 wearing the head-mounted image display device (HMD) 2. And a three-dimensional space sensor 3 provided near the head of the subject of the image display device 2 to detect the viewpoint position data, the sight line direction data 3
The dimensional space data is input to the image generation device 4. Further, an image pickup device (an electronic image pickup device is used in the present embodiment) 7 for picking up a real space image (real shot image) is provided above the viewpoint, and the picked up real shot image is inputted to the image synthesizing device 8.
Further, in order to make the image synthesizing device of the present embodiment interactive, an auxiliary input device 6 is provided at a position where the subject can manually operate, and an operation signal from the auxiliary input device 6 is used as a virtual space image. The image data of the virtual space image database 5 is updated by inputting the data to the database 5.

The image generation device 4 uses the viewpoint position data,
Calculation is performed based on the line-of-sight direction data, the three-dimensional space data, and the image data from the virtual space image database 5,
A CG image corresponding to a scene of the world (virtual space) viewed from the viewpoint position and the line-of-sight direction is generated and output to the image synthesizing device 8. The image synthesizing device 8 uses the virtual space CG image generated using the virtual space image database 5 and the image generating device 4 and the photographed image captured by the image capturing device 7 to fit one image into the other image. Performs synthesis processing. In the present embodiment, the CG image is embedded in the photographed image, and at that time, the chroma key composition process is used as the image composition method.

The image synthesizing process of this embodiment will be described below by using a specific example. 2 to 4 are diagrams for explaining the image synthesizing process of the first embodiment, and an example in which the image synthesizing process of the present embodiment is applied to a drive simulator as a virtual experience device installed in a driving school or the like. Shows.
The driver 1 who is the subject, as shown in FIG.
2 is mounted, the automobile 11 is driven, the vehicle runs in a virtual space drawn by CG, and a simulation for the purpose of learning the driving technique is performed. By the way, since the conventional drive simulator uses, for example, a monitor for displaying an image, the driver's 1 field of view includes something other than the world of simulation, and the reality is poor. . Therefore, in the present embodiment, the HMD is incorporated into a virtual experience device represented by a drive simulator as an image display device, and a real image and a C image are displayed.
By presenting the synthetic image combined with the G image to the driver 1 by the HMD, the reality is improved and the simulation effect is enhanced.

The drive simulator shown in FIGS. 2 and 3 is adapted to control the posture, acceleration, etc. of the automobile 11 by hydraulic pressure or the like corresponding to the CG screen. At this time, the world seen through the window 12 as shown in FIG. 4 is a virtual world drawn by CG, and at the same time, the panel portion 13 such as various meters near the driver's seat of the automobile and the steering wheel 1 which are in the field of view.
4 (see FIG. 4), the passenger 15 and his / her limbs 16 (see FIG. 3) and the like are the image pickup device 7 (CCD) provided in the HMD 2.
Image pickup device (such as an image pickup device) and converted into an electric signal. Examples of a real image, a CG image, and a composite image in this drive simulator are shown by 21, 22, and 23 in FIG. 5, respectively. In such a situation, the technique used for synthesizing the image of the virtual world by the CG and the photographed image is the chroma key synthesizing process of this embodiment.

As shown in FIG. 6, a projector (or a light emitting body having directivity) 1 from the front of the vehicle body of the automobile 11.
7 projects light according to the shape of the window 12 of the automobile. The projector 17 repeats blinking in time as shown by hatching in each figure of FIG. 7, and after changing from the bright state of FIG. 7A to the dark state of FIG. It becomes a state. During this time, since the window 12 of the automobile is made of semi-transparent glass or semi-transparent plastic, the image pickup device 7
When the image is taken from the inside of the vehicle toward the window 12 by
Only the portion of the window is a portion in which a change in luminance over a predetermined value occurs with time. At this time, the transmittance of the member forming the window 12 is set so that the brightness change portion can be detected and extracted when the image processing is performed after the change in the brightness of the window portion is captured.
The parts other than the window are set to have such a transmittance that the brightness does not change. In addition to the imaging device 7, the HMD 2 is also provided with a spatial information detection device (three-dimensional real space sensor) (not shown) that detects three-dimensional spatial information such as the line-of-sight direction of the driver 1 and the position of the viewpoint. As this spatial information detection device, any one or two or more of an optical sensor, a magnetic sensor, a gyro sensor, an ultrasonic sensor, etc. is used.

Next, a method of synthesizing a real image and a CG image under the above-mentioned situation will be described with reference to FIG. In FIG. 8, the photographed image captured by the electronic image pickup device 7 is input to the image synthesizing device 8, and an area in which a luminance change of a predetermined value or more is temporally extracted from the photographed image is extracted. The CG image of the virtual space corresponding to the viewpoint position and the line-of-sight direction of the driver 1 detected by the space sensor 9 is fitted, and the photographed image is used as it is in the area where the brightness change has not occurred. As a result, a highly realistic composite image in which the photographed image and the CG image are mixed is generated, and the composite image is displayed on the image display element 2 a of the image display device 2. Therefore, the driver 1 can put himself in a world where he can see a part of the vehicle body, a passenger, his / her limbs, etc., which are actual objects in the real space, in an arbitrary three-dimensional virtual space.

The discrimination conditions for the CG image area and the photographed image area using the above-mentioned luminance signal may be modified as follows. For example, as shown in FIG. 9A, a light source A and a CG that illuminate the entire space where the automobile 11 is installed.
By providing a light source B (for example, a projector) that illuminates an area corresponding to the window portion of the automobile into which the image is fitted, and setting the phases of the luminance changes of the light source A and the light source B to be opposite phases,
This makes it easy to distinguish between a virtual world area in which a CG image is fitted and an area in which a photographed image is used as it is. That is, the state of FIG. 9A in which the light source A is in the bright state and the light source B is in the dark state is changed from the state of FIG. 9A in which the light source A is in the dark state and the light source B is in the bright state (b), and then the light source A is turned on again. In this state, the light source B is changed to the dark state (c).

When the phases of the brightness changes of the light source A and the light source B are made opposite to each other as described above, the area illuminated by the light source B consequently changes in brightness between FIGS. 9 (a), 9 (b) and 9 (c). Becomes smaller, and the change in brightness in areas other than that
Will depend on. Here, if the light intensities of the light source A and the light source B are the same in the area illuminated by the light source B, there is no change in luminance in the area illuminated by the light source B. Therefore, the above determination condition is "C
By fitting "G image", a desired composite image is obtained. Further, as a modified example of the above modified example, if the light source B does not change in brightness and the light quantity is constant, when the light quantity of the light source B is larger than that of the light source A, the area illuminated by the light source B has the brightness. There is a situation in which there is almost no change, the predetermined brightness is always maintained, and the areas other than the area change in brightness depending on the brightness change of the light source A. Therefore, a desired composite image can be obtained by setting the above-mentioned determination condition to "the brightness is equal to or more than a predetermined value in the area in which the CG image is fitted has a small brightness change (almost no brightness)".

FIGS. 10A to 10C are views for explaining the image synthesizing process of the second embodiment of the present invention in comparison with the prior art. The present embodiment is mainly related to a problem in the case of performing an image combining process based on a color signal of a specific color (for example, blue), for example, a blue background and a blue outfit shown by hatching in FIG. A photographed image 31 including the subject
It is effective as a remedy for the problem that when the CG image 32 is fitted into the CG image 32, the CG image 32 becomes the composite image 33 fitted into the part 31a of the subject dressed in blue of the photographed image 31. Further improvement can be achieved by using it.

For example, as shown by hatching in the horizontal direction in FIG. 10B, a portion 34a corresponding to a window in the photographed image 34 has a temporal change in luminance of a predetermined value or more ( On the contrary, when the CG image 35 is extracted as a part having a constant brightness and fitted therein, the brightness change leaks to the outer periphery of the window or the outside of the window frame, and the CG image is fitted to that part. As a result, weird composite image 3
However, this embodiment is also effective as a remedy for this problem. That is, in the present embodiment, only the color signal is obtained by setting the portion of the photographed image 37 into which the CG image 38 is fitted as “a portion having a temporal change in luminance of a predetermined value or more and having a specific color signal”. It is possible to simultaneously improve the defect when used and the defect when only the luminance signal is used. For example, FIG.
A desired composite image 39 as shown in is obtained. Therefore, there is a problem that the CG image is fitted to the outer periphery of the window or the portion outside the window frame where the brightness change in the photographed image 37 is leaking, or the blue clothing portion of the photographed image 31 when the specific color signal is blue. It is possible to solve the problem that the CG image 32 is fitted in the 31a.

Instead of the above, the window may be made of a member having polarizability, and a polarizing plate may be arranged in the image pickup device 7 to cut off unnecessary reflection. Further, it is not a case where an image in which a boundary (edge) between a real image area and a CG image area is clear in a composite image like the above-described window is expressed, and the boundary is unclear and gradually changes from a real image to a CG image. To express an image having such a special effect, on the contrary, the way of changing from the actual image to the CG image is adjusted according to the degree of leakage of the luminance change without using the color signal and the luminance signal together. Is preferred. By doing so, an image having a desired special effect can be generated.

FIG. 11 is a diagram for explaining the image synthesizing process of the third embodiment of the present invention. In the present embodiment, the luminance signal and the color signal are simultaneously used for the determination condition of the area in which the CG image is fitted in the photographed image, and the portion where the luminance change over time exceeds the predetermined value occurs in the photographed image. After extraction, a part not including a part having a specific color signal is extracted, and a CG image is fitted into the part. In this way, the determination condition of the area in which the CG image is fitted is set, and there is a possibility that the brightness change may have leaked out even though it is not desired to fit the CG image such as the outer periphery of the window frame in the photographed image 40 in FIG. By coloring (indicated by diagonal lines in FIG. 11) a color signal of a specific color (for example, red), even if a change in luminance over a predetermined value occurs with time, a portion having red is Since the CG image is not fitted, a desired composite image is obtained. Alternatively, when a person or the like has been affected by a change in brightness near the window, a desired composite image can be obtained by setting a specific color as a skin color.

FIGS. 12A and 12B are views for explaining the image synthesizing process of the fourth embodiment of the present invention. In the present embodiment, as in the first embodiment, the CG image is fitted to the portion of the photographed image in which the luminance change over time is equal to or greater than a predetermined value. The conditions are as follows. As described above, since the area on the outer periphery of the window in the live-action image is affected by the change in brightness, the light-action live-action image 41 shown in FIG.
The amount of change in luminance on the dotted line PQ when changing to the real image 42 in the dark state shown in (b) is as shown in the graph of FIG. In consideration of this, the maximum value of the brightness change amount in the window frame part is set as the threshold value, and the condition of "fitting the CG image in the part in which the brightness change above the threshold value occurs" is set. A CG image is not fitted to a portion where the luminance change occurs but the amount of luminance change is less than the threshold value, such as a portion inside the window frame edge (outer periphery of the window) shown in FIG. can get.

14 (a), 14 (b) and 16
(A), (b) is a figure for demonstrating the image compositing process of 5th Embodiment of this invention, respectively. The present embodiment has two light sources, a light source A that illuminates the whole as illustrated in FIGS. 9A to 9C in the first embodiment and a light source B that illuminates only a portion into which a CG image is fitted. , A case where the light source B that illuminates only the part into which the CG image is fitted maintains a constant brightness while the light source A that illuminates the entire body repeatedly blinks,
This is applicable to the case where the light source B that illuminates only the portion into which the CG image is fitted repeats blinking in a phase opposite to that of the light source A while the light source A that illuminates the whole repeats blinking. In any case, by setting the determination condition of the area in which the CG image is to be fitted in the photographed image to be “there is a luminance change rate of not more than a predetermined value and the minimum luminance is not less than a certain threshold value”, as described in detail below. The desired composite image is obtained. Here, the brightness change rate is defined as the amount of change in brightness with respect to the maximum brightness of the area.

FIG. 14A shows a photographed image 43 in the case where the light source A for illuminating the whole is bright and the light source B for illuminating the part into which the CG image is fitted maintains a constant brightness, and FIG. 14B illuminates the whole. The photographed image 44 in the case where the light source A is dark and the light source B that illuminates the part where the CG image is fitted maintains a constant brightness is shown.
At this time, the temporal luminance changes of the part (here, the window part) in which the CG images are fitted in the photographed images 43, 44 and the other parts are shown in FIGS. 15 (a) and 15 (b). Become. In the figure, the horizontal axis represents time and the vertical axis represents luminance. Here, when the brightness of the light source B is Q1, the brightness of the light source A is Q2, and the period of the light source A is T, the brightness of the window portion where the CG image is to be fitted is shown in FIG. As described above, Q1 is from 0 to T, and from T to 2T, the constant brightness Q1 is added with the brightness changing brightness Q2 for illuminating the whole. As a result, the maximum brightness is (Q1 +
Q2) and the amount of change in brightness is Q2, so the rate of change in brightness is Q2 / (Q1 + Q2). On the other hand, in the parts other than the window, the maximum brightness is Q2 and the brightness change amount is Q2.
Therefore, the rate of change in luminance is Q2 / Q2 = 1. Therefore, the condition that the CG image is not fitted is "the luminance change rate is Q2
/ (Q1 + Q2) or more and less than 1 value range ”
And Furthermore, set the threshold to the value indicated by the dotted line in the figure,
By setting the condition that "the CG image is not fitted to the site where the minimum value of the brightness is less than or equal to the threshold value", the CG image is fitted only to the desired site.

For example, a light source B for illuminating a region of a CG image
The brightness Q1 of the light source A for illuminating the whole is set to 200 (cd / m ^{2 where} cd; candela).
In the case of (cd / m ² ), the rate of change in brightness of the part of the window where the CG image is to be fitted is 1/6 = 0.167, and the rate of change in brightness of the entire room is 1. The desired result can be obtained by setting, for example, 0.2 to the threshold value of the brightness change rate. In this case, the rate of change in brightness increases as the distance from the window frame increases. Therefore, by setting the threshold value to a value as small as possible, it is possible to avoid the problem that the CG image fits outside the window frame. Furthermore, it is more preferable to set the threshold value of the minimum luminance to a value close to 1000 (cd / m ² ). For example, even if the CG image is a region where it is not desired to be fitted, the rate of change in brightness becomes small even when the brightness is shaded and the minimum brightness becomes equal to or less than the threshold value, so that the CG image is prevented from being accidentally fitted. .

In FIGS. 15 (a) and 15 (b), the region where the CG image is fitted and the region where the actual image is used may be discriminated based on the phase of the light source. That is, in FIGS. 15 (a) and 15 (b),
If the two light sources A and B have opposite phases, the brightness change in the window where the CG image is fitted becomes dark, bright, dark, bright,
Brightness, darkness,
Light and dark. Therefore, darken the part where the CG image is fitted,
A desired composite image can be obtained by conditioning the light, dark, and light-changing regions. Further, if the phases are different, not the opposite phases, it is possible to determine the part into which the CG image is fitted using the phases.

FIG. 16A shows a photographed image 45 when the light source A for illuminating the whole is bright and the light source B for illuminating the part where the CG image is fitted is dark, and FIG. 16B shows the light source A for illuminating the whole.
Shows a photographed image 46 in the case where is dark and the light source B for illuminating the part where the CG image is fitted is bright, and it is assumed that the light source A and the light source B are in opposite phases and have the same luminance. At this time, the photographed image 4
FIGS. 17 (a) and 17 (b) show temporal luminance changes of the portion (here, the window portion) in which the CG image is fitted and the other portions in FIGS. In the figure,
The horizontal axis represents time and the vertical axis represents luminance. Here, the light source B
When the luminance of the light source A is Q1, the luminance of the light source A is Q2 (= Q1), and the period of the light source A is T, the luminance of the window portion where the CG image is to be fitted is as shown in FIG. To 0
Q2 from T to T, Q from T to 2T
Since 1 and Q1 = Q2, the brightness of the window portion is constant. As a result, the luminance change rate is 0 / Q1 = 0.
On the other hand, the amount of change in luminance of the portion other than the window is Q2, and the rate of change in luminance is Q2 / Q2 = 1. Therefore, the condition in which the CG image is not fitted is defined as "a part having a luminance change rate greater than 0 and within a range of 1 or less". Further, by setting a threshold value to a value indicated by a dotted line in the drawing and setting a condition that "a CG image is not fitted to a portion whose minimum luminance value is less than or equal to the threshold value", the CG image is fitted only to a desired portion. It will be. As a result, it is possible to prevent the CG image from being embedded in a portion of the subject that becomes a shadow and originally has a desired luminance change but does not have a luminance change.

The image synthesizing method in the image synthesizing apparatus of the present invention is not limited to the above-mentioned one. For example, the color signal used together with the luminance signal is not only blue but also different color signals depending on the situation. It is possible to use various methods such as performing image composition using a plurality of color signals, or combining and combining luminance signals and color signals. Further, in the above-described embodiment, when the luminance change of the light source is made to be periodic, the period is synchronized with the scanning period of the imaging device, or is set to an integral multiple thereof to prevent unnecessary flicker of the captured image. be able to. In particular, when the light source A that illuminates the whole changes periodically, by synchronizing with the cycle, it is possible to prevent the flicker of the captured image from being felt in the combined image.

In the above-mentioned embodiment, the configuration example of the drive simulator using the HMD has been described.
In order to improve the general chroma key processing as shown in FIG. 10A, even if it is applied to a system that does not use HMD,
The image synthesizing method of the present invention using the luminance signal is based on a real image and C
This is effective when synthesizing with a G image. The difference between when the HMD is used and when it is not used is that the composite image is the same as the visual field image of the HMD wearer, and that the image corresponding to the wearer's movement, displacement, etc. can be viewed in real time. is there.

HMD represented by drive simulator
The virtual experience device using is also provided with the following advantages. That is, since the conventional drive simulator uses a monitor, the driver's field of view is not realistic because it is possible to see something other than the simulation world. In order to improve this, if an attempt is made to achieve a system that can present a realistic composite image without using an HMD, the system becomes extremely large-scale, and the cost of the system itself and the space occupied by the system become enormous. Occurs. As a countermeasure against this, by incorporating the HMD as an image display means in a virtual experience device typified by a drive simulator, both the cost and space problems can be greatly improved, but the image displayed on the HMD is subjected to a combining process. If there is no CG image, it lacks reality in another sense, and the quality as a simulation deteriorates. Therefore, by providing the image in which the real image and the CG image are combined by the HMD, it is possible to improve the reality, the cost, and the space. It should be noted that there are various other configuration examples using the HMD other than the above, and the above embodiment has shown the configuration example of the drive simulator.
The present invention is not limited to this, and can be applied to virtual experience devices including various attraction games.

Generally, in a virtual reality experience system using an HMD, an image generation device that supplies an image to the HMD is installed inside the vehicle of the virtual reality experience system, etc., away from the wearer, and the HMD is connected via a cable or the like. Send and receive signals to and from. If this vehicle is accompanied by vibration or violent movement, problems such as image distortion will occur, but in order to avoid this, the image generation device is installed at a position distant from the vehicle for control. To do. In this case, it is preferable that the signals are transmitted and received wirelessly (wirelessly, etc.) as desired, and if wireless is used, the virtual world will be caused by the cables becoming entangled in the HMD wearer due to violent movements or the cables being in contact with the body. It is possible to prevent a decrease in immersive feeling. Further, the space sensor attached to the HMD and detecting the space information of the HMD wearer may be mounted at a position different from the HMD, such as the body of the wearer, in order to reduce the weight. In that case, in order to provide a CG image from a position near the wearer's viewpoint, it is necessary to perform coordinate conversion to the position of the wearer's eyeball.

In order to further improve the reality in the virtual reality experience system using the HMD, FIG.
A stereoscopic (3D) system such as 8 may be used. That is, in order to generate a CG virtual space image (3D image) having right-eye data and left-eye data having parallax,
As shown in FIG. 18, a left-eye and right-eye image taken by left and right imaging means (not shown) having a visual field corresponding to the left eyeball 52L and the right eyeball 52R of the wearer and substantially corresponding to the visual field of each eyeball is displayed. It is input to the image synthesizing device 51, the CG image incorporating parts of these real images are extracted, the CG image for the left eye and the CG image for the right eye are respectively combined with the extracted parts, and the combined image for the left eye and the right eye are combined. Generate a composite image. As a result, a more realistic image can be provided.

Further, when the virtual experience device is an attraction such as a racing game, a plurality of cameras are provided in the virtual experience device in addition to the image pickup means provided in the HMD, and the photographed image picked up by them is the same as above. By combining the images with each other and displaying them separately on the HMD as desired, it is possible to provide an image 54 in which an image of the rearview mirror and images from different viewpoints as shown in FIG. 19 can be provided. The property is improved. As described above, the present invention has various possibilities that can be applied to attractions having entertainment properties and various applications by providing a method for generating a highly accurate synthetic image with reality.

Although an example of image composition by discriminating a portion where a luminance change has occurred has been described above, a hue change may be used as a discrimination key in the same manner. For example, the hue of the window portion into which the CG image shown in FIG. 6 is fitted is changed by illuminating it with the projector 61 shown in FIGS. 20 (a) and 20 (b). The projector 61 includes filters 63R, 63G, 63B for R, G, B colors supported on a shaft 62.
By rotating the lens 6 at a constant speed, the lens 6 is inserted through the opening 64 matched to the shape of the part (window) into which the CG image is fitted.
It is possible to emit light whose hue condensed at 5 changes with time. As a result, in the image taken by the imaging means from the inside of the vehicle, the hue of only the window part changes temporally, and it is possible to distinguish and extract from other parts. The cycle in which the hue changes and the hue to be used depend on the hue and movement of the object in the shooting range and the illumination, so the optimum cycle and color are set accordingly.

The present invention is not limited to the above-mentioned examples, and various changes and modifications can be added. For example, an image capturing unit that captures an image, an image generating unit that is provided separately from the image capturing unit and generates another image, and a luminance change over a predetermined value over time within the range of the image captured by the image capturing unit. Alternatively, an image processing unit for discriminating a portion where a hue change has occurred and a portion where no hue change has occurred, and performing image processing for fitting the other image into one of the two determined portions is set. In the image synthesizing device (Appendix 1), the luminance change or the hue change of a predetermined value or more in time is periodic (Appendix 2). . In that case, the change in the luminance or the hue is periodic, so that the above-mentioned two portions can be more easily and surely discriminated from each other, and a portion in which the other image (for example, a CG image) is not desired to be inlaid becomes a shadow. There is no problem that the CG image is inlaid when the brightness changes due to the non-existence.

In addition, in the additional item 1, the image synthesizing device may be provided with a light source of varying the light amount, which illuminates a portion of the image pickup means in which the image of the subject is fitted (additional item 3). In that case, when capturing a photographed image in a three-dimensional real space, another image (for example, CG
By illuminating the part where the image) is to be fitted with a light source with varying light intensity, the CG image is fitted to the part where the brightness change occurs when the above two parts are discriminated, and a composite image that is a real image is created for the other part. can do.
A light source that illuminates a portion into which the image is fitted from the front or the back side may be used as the light source whose light amount changes.

In addition, in the above-mentioned additional item 1, the image synthesizing device may be provided with a light source having a variable hue, which illuminates a portion of the image pickup means in which the image of the subject is fitted (additional item 4). In that case, when capturing a photographed image in a three-dimensional real space, another image (for example, CG
By illuminating the part where the image) is to be fitted with a light source that changes hue, a CG image is fitted to the part where the hue change occurs when the above two parts are discriminated, and a composite image that is a real image is created for the other part. can do.
As the light source of which the hue changes, it is preferable to use a light source of which the amount of light changes, which illuminates the portion where the image is fitted from the front or the back.

In addition, in the above-mentioned additional item 1, the first light source of varying brightness for illuminating the entire subject of the image pickup means, and the part of the subject that illuminates the other image are illuminated. It may be an image synthesizing device having a second light source (Appendix 5). In that case, when capturing a real image in a three-dimensional real space, the entire subject is illuminated by the first light source whose brightness changes, and the portion into which the other image (for example, CG image) is fitted is the first light source. In addition to being illuminated by the second light source, CG
The luminance characteristic of the second light source is more strongly reflected in the portion where the image is to be fitted than in other portions. With this, it is possible to determine the portion where the CG image of the photographed image is fitted. For example, when the second light source is an illumination having a luminance change different from that of the first light source, the determination can be performed.

In addition, in the above-mentioned additional item 5, the image synthesizing device may be characterized in that the second light source changes in a phase opposite to that of the first light source (additional item 6). In that case, the brightness of the second light source changes in a phase opposite to that of the first light source,
When the absolute values of the brightness are equal, the part illuminated by the second light source has a small change in brightness as a result, and it is possible to discriminate a part where the change in brightness is equal to or more than a certain value and a part where it is not.

In addition, in the additional item 5, the second light source may be an image synthesizing device having a constant light amount (additional item 7). In that case, when capturing a photographed image in a three-dimensional real space, the entire subject is illuminated by the first light source whose brightness changes, and the portion into which the CG is fitted is not only the first light source but also the second light source. Since the illumination is performed, when the second light source has a constant brightness, the brightness of the part where the CG image is to be fitted is substantially constant unlike the other parts. As a result, it is possible to determine the portion where the real image and the CG image are fitted.

Further, an image pickup means for picking up an image, an image generating means provided separately from the image pickup means for generating another image, and a predetermined time value or more within a range of the image picked up by the image pickup means. Image synthesizing device for performing image processing for fitting the other image into a portion having a change in luminance and having a specific color signal. In 8), the image synthesizing apparatus may be characterized in that the change in luminance above a predetermined value with time is periodic (Appendix 9). In that case,
Since the change in the luminance is periodic, the above-mentioned two portions can be more easily and surely discriminated from each other, and the portion where the user does not want to fit the above-mentioned other image (for example, a CG image) may not be shaded. Therefore, there is no problem that the CG image is fitted when the brightness changes.

In addition, in the additional item 8, there is provided a light source that illuminates a portion of the image pickup means into which an image of a subject is fitted, and the light source has a specific light amount when the image pickup means picks up the image. The image synthesizing device may have a color capable of generating a color signal (Appendix 1).
0). In this case, when the portion where the CG image is to be fitted is illuminated with a light source with a variable light amount, the light source has the specific color. It has a simple and reliable hue.

Further, an image pickup means for picking up an image, an image generating means provided separately from the image pickup means for generating another image, and a predetermined time value or more within a range of the image picked up by the image pickup means. And an image processing means for performing image processing for fitting the other image into a portion other than a portion having a change in luminance and having a specific color signal. In (Appendix 11), the image synthesizing apparatus may be characterized in that the temporal change in the luminance equal to or more than a predetermined value is periodic (Appendix 1).
2). In that case, since the change in the brightness is periodic, it becomes easier and more reliable to distinguish the above two parts,
The problem that the CG image is fitted does not occur when a change in luminance occurs because a portion where one does not want to fit another image (for example, a CG image) is hidden or not.

Further, in the above-mentioned additional item 11, illumination means for temporally changing the luminance of a portion of the subject of the image pickup means in which the other image is fitted, and the image corresponding to the specific color signal The image synthesizing device may have a means for presenting it on the outer periphery of the portion to be fitted (Appendix 13). In that case, since the above-mentioned two parts are discriminated depending on whether or not the two conditions, that is, whether or not the brightness change and the specific hue are included, are satisfied, a part where the other image (for example, a CG image) is desired to be fitted and a part of the photographed image. It will be certain that In particular, a specific color signal is generated at the time of imaging of a portion around the outer periphery of the portion where the CG image is desired to be embedded, which is not desired to be fitted with the CG image, but may be judged to have a change in brightness. As described above, by including the specific hue, the boundary between the portion where the CG image is to be fitted and the portion where the CG image is not embedded becomes clear,
Undesired CG image fitting is prevented.

In addition, in the above-mentioned additional item 13, the illuminating means may be an image synthesizing device characterized in that it is a light source of illuminating a portion into which the other image is fitted from the front side or the rear side and changing the light amount ( Appendix 14). In that case, when capturing a real image in a three-dimensional real space, by illuminating a portion of the real image to be fitted with the other image (for example, a CG image) from the front or the back with a light source whose light amount changes, When discriminating the two parts, it is possible to generate a composite image in which a CG image is fitted to a part which has a change in luminance and does not include the specific color signal, and the other parts are real images.

Further, an image pickup means for picking up an image, an image generation means provided separately from the image pickup means for generating another image, and a first time point within the range of the image picked up by the image pickup means. Image processing means for performing image processing for fitting the other image in a portion where a change in luminance of a predetermined value or more occurs and the maximum luminance thereof is a second predetermined value or more. The image synthesizing device may be (Appendix 15). In that case, the other image (for example, a CG image) is determined by discriminating the above two parts depending on whether the luminance change occurs and whether or not the maximum luminance satisfies the two conditions of the second predetermined value or more. The distinction between the part to be fitted and the part of the photographed image becomes reliable.

For example, in the case of using only the condition in which the portion in which the CG image is to be fitted is the portion in which the luminance change occurs,
Around the portion where the luminance change occurs, the degree of change is small, but the luminance change occurs. Therefore, the boundary is clear between the part where the image is fitted and the other part, but the boundary is not clear between the part where the brightness change occurs and the part where the brightness change does not occur. In this case, it is determined that the luminance change has occurred in a portion where the user does not want to fit the CG image, and the CG image may be fitted. In order to solve this, by setting the condition of the brightness change of the part in which the CG image is fitted to "fit the CG image only in the part where the maximum brightness exceeds the threshold value (second predetermined value)", a desired composite image is obtained. To be By appropriately selecting this second predetermined value,
It is possible to prevent the CG image from being embedded in a portion where it is not desired to embed the CG image, but a slight change in luminance has occurred.

In addition, in the above-mentioned additional item 15, the image synthesizing apparatus may be characterized in that the change in the luminance of the first predetermined value or more in time is periodic (additional item 16).
In that case, since the change of the luminance is periodic, the above-mentioned two parts can be more easily and surely discriminated from each other, and a part which is not desired to be fitted with the other image (for example, a CG image) is shaded. There is no problem that the CG image is fitted when the brightness changes due to the absence of the CG image.

In addition, in the above-mentioned additional item 15, a reflectance suppression region for suppressing the brightness to be less than the first predetermined value is provided on the outer periphery of the portion of the image pickup means where the other image is combined. The image synthesizing device may be characterized by (Appendix 17). In that case, the reflectance is low so that the maximum luminance is less than the first predetermined value in a portion around the portion in which there is a change in luminance that may be fitted without wanting to fit the CG image. By using a member or the like, a desired composite image can be obtained.

In addition, in the additional item 17, the image synthesizing apparatus may further include a light source that illuminates a portion into which the other image is fitted from the front side or the rear side so as to change the light amount (additional item 18). ). In that case, 3
When capturing a photographed image in a two-dimensional real space, by illuminating a portion where the other image (for example, a CG image) is to be fitted with a light source whose light amount changes from the front or the rear, the brightness at the time of discriminating the above two portions It is possible to generate a composite image in which a CG image is fitted in a portion where the change occurs and the maximum brightness is equal to or higher than the first predetermined value (threshold value), and the other portions are real images.

Further, an image pickup means for picking up an image, an image generation means provided separately from the image pickup means for generating another image, and a first predetermined value or more within the range of the image picked up by the image pickup means. Image processing means for discriminating a portion having a minimum luminance change rate of less than a second predetermined value and fitting the other image to the portion excluding the discriminated portion. The image synthesizing apparatus may be configured by (Appendix 19). In that case, in order to ensure the distinction between the portion where the other image (for example, a CG image) is desired to be fitted and the portion of the actual image, the above-mentioned two portions are discriminated depending on whether or not the following two conditions are satisfied. That is, the portion in which the CG image is not fitted is the portion with the rate of change in luminance equal to or higher than the first predetermined value and the minimum luminance less than the second predetermined value. As a result, the portion where the CG image is fitted becomes a portion satisfying the condition that the luminance change rate is less than the first predetermined value and the minimum luminance is not less than the second predetermined value, and the following problems can be prevented. it can.

For example, in the case of illuminating the entire subject of the image pickup means with the first light source which changes the luminance and illuminating the portion into which the CG image is fitted with the light source of constant luminance, the first illuminating the whole of the subject. The rate of change in brightness is reduced in the shadow part generated by the light source of. At this time, since it is recognized that there is no brightness change, the CG image is erroneously fitted in the shadow portion as well. However, as described above, "the lowest brightness is equal to or higher than the second predetermined value in the CG image fitting portion". Since such a condition is provided, such erroneous combination is prevented. Further, it is possible to prevent the inconveniences such as those mentioned in claims 2 and 3 and appendix 15 that may occur when the CG image is fitted. That is, for a portion having a luminance change rate in a range that is larger than the luminance change rate of the portion into which the CG image is fitted and smaller than the luminance change rate of the peripheral luminance change portion, the first predetermined value is set as the threshold value. Then, the desired composite image is obtained by fitting the CG image only when the value is below the threshold value.

In addition, in the above-mentioned additional item 19, the image synthesizing apparatus may be characterized in that the change in luminance of the first predetermined value or more with time is periodic (additional item 20).
In that case, since the change of the luminance is periodic, the above-mentioned two parts can be more easily and surely discriminated from each other, and a part which is not desired to be fitted with the other image (for example, a CG image) is shaded. There is no problem that the CG image is fitted when the brightness changes due to the absence of the CG image.

In addition, in the above-mentioned additional item 19, a first light source of varying brightness for illuminating the entire subject of the image pickup means and a part of the subject where the other image is fitted are illuminated. The image synthesizing apparatus may have a second light source having a constant light amount (Appendix 2).
1). In that case, by illuminating a portion into which the other image (for example, a CG image) is fitted with a second light source having a constant light amount, the minimum brightness is kept above a predetermined value and the rate of change in brightness is kept below a predetermined value. As a result, the above two parts can be easily and reliably discriminated.

Further, an image pickup means for picking up an image, an image generating means provided separately from the image pickup means for generating another image, and a luminance change at a constant cycle within a range of the image picked up by the image pickup means. And an image processing unit that performs image processing to determine two portions that are occurring and whose luminance changes have opposite phases to each other, and that fits the other image into one of the determined two portions. An image synthesizing device may be provided (Appendix 22). In that case, for example, by causing the brightness change between the real image and the CG image to have opposite phases to each other, the brightness change amount between the two parts becomes large, so that the part into which the image is fitted can be easily identified. Become.

In addition, in the above-mentioned additional item 22, a first light source of varying brightness for illuminating the entire subject of the image pickup means and a light source for illuminating a portion of the subject into which the other image is fitted are provided. The image synthesizing apparatus may have a second light source whose brightness changes in an opposite phase to the first light source (Appendix 23). In that case, by illuminating the entire imaging range of the image (for example, the real image) with the first light source, the real image is used as it is in the part of this phase, and the other image (for example, C
A desired composite image is obtained by illuminating a portion where the G image) is to be fitted so as to change the brightness.

Further, the above additional items 2, 9, 12, 16, 2
The image synthesizing apparatus may be characterized in that, at 0 and 22, the cycle of the change is substantially the same as the scanning cycle of the image pickup means or an integral multiple thereof (Appendix 24). In that case, it is possible to prevent flicker of the storage device (frame memory or the like) used in the image combining process.

In addition, in the above supplementary items 1 to 24, a head-mounted image display device is provided, the imaging means is fixed to the head-mounted image display device, and an image synthesized by the image synthesis device is displayed. An image synthesizing device with a head-mounted image display device may be used (Appendix 25). In that case, by adapting to a head-mounted image display device (HMD), the HMD observer can observe the omnidirectional (360
It is possible to put yourself in a virtual space of (degrees), and by accelerating the speed of image synthesis processing, you can feel yourself in the virtual space in real time. At this time, by providing the HMD with an image pickup means for picking up a photographed image, it is possible to photograph an image that is almost the same as the line of sight of the observer.
With the synthetic image incorporated as an image, it is possible to construct a virtual world that is closer to the real world. In addition, the sound effect is emitted from the speaker of the HMD, and the observer's viewing position and posture are not limited, so that the reality is improved. By thus incorporating the HMD into the system as an image display means, it is possible to reduce the size and cost of the system as compared with the case of achieving the same effect without using the HMD.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of an image composition device according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating an image combining process according to the first embodiment.

FIG. 3 is a diagram illustrating an image combining process according to the first embodiment.

FIG. 4 is a diagram illustrating an image combining process according to the first embodiment.

FIG. 5 is a diagram exemplifying a real shot image, a CG image, and a composite image when the first embodiment is applied to a drive simulator.

FIG. 6 is a diagram for explaining illumination of a photographed image area and a CG image area according to the first embodiment.

7A to 7C are diagrams for explaining illumination of a real image area and a CG image area according to the first embodiment.

FIG. 8 is a diagram illustrating an image combining method according to the first embodiment.

9A to 9C are views for explaining the operation of the modification of the first embodiment.

FIG. 10A to FIG. 10C are diagrams for explaining the image synthesizing process of the second embodiment of the present invention in comparison with a conventional technique.

FIG. 11 is a diagram illustrating an image combining process according to the third embodiment of the present invention.

FIG. 12A and FIG. 12B are views for explaining the image synthesizing process of the fourth embodiment of the present invention.

FIG. 13 is a diagram for explaining the amount of change in luminance according to the fourth embodiment.

FIGS. 14A and 14B are diagrams for explaining an example of image compositing processing according to the fifth embodiment of the present invention.

15 (a), (b) are respectively FIG. 14 (a),
It is a figure for demonstrating the time-dependent luminance change of each site | part in (b).

16A and 16B are diagrams for explaining another example of the image synthesizing process of the fifth embodiment of the present invention.

17 (a) and (b) are respectively FIG. 16 (a),
It is a figure for demonstrating the time-dependent luminance change of each site | part in (b).

FIG. 18 is a diagram illustrating a configuration when the present invention is applied to a stereoscopic system.

FIG. 19 is a diagram illustrating a composite image when the present invention is applied to an attraction such as a racing game.

20A and 20B are diagrams for explaining another example of the image synthesizing process according to the embodiment of the present invention.

FIG. 21 is a diagram for explaining a conventional technique.

[Explanation of symbols]

 1 subject (driver) 2 head-mounted image display device (HMD) 3 three-dimensional space sensor 4 image generation device (image generation means) 5 virtual space image database 6 auxiliary input device 7 imaging device (imaging means) 8 images Synthesizing device (image processing means) A, B Light source

Claims (3)

[Claims] 1. An image pickup unit for picking up an image, an image generation unit provided separately from the image pickup unit for generating another image, and a predetermined value or more in time within the range of the image picked up by the image pickup unit. And an image processing unit that performs image processing to discriminate a portion where a luminance change or hue change has occurred and a portion where no hue change has occurred, and to fit the other image into one of the two determined portions set in advance. An image synthesizing apparatus comprising: 2. An image pickup unit for picking up an image, an image generation unit provided separately from the image pickup unit for generating another image, and a predetermined value or more in time within the range of the image picked up by the image pickup unit. And an image processing means for performing image processing for fitting the other image in a portion having a change in luminance and having a specific color signal. 3. An image pickup unit for picking up an image, an image generation unit provided separately from the image pickup unit for generating another image, and a predetermined value or more in time within the range of the image picked up by the image pickup unit. And an image processing means for performing image processing for fitting the other image into a portion other than a portion having a change in luminance and having a specific color signal. .