JP5734080B2 - Information processing apparatus, processing method thereof, and program - Google Patents

Description

  The present invention relates to an information processing apparatus, a processing method thereof, and a program.

  In recent years, a technique has been developed that presents information that is not in real space to a user by synthesizing three-dimensional CG (Computer Graphics) with video in real space. Such a technique is called an MR (Mixed Reality) system.

  The MR system is attracting attention as a technology for enhancing the virtual space for the purpose of coexistence of the virtual space and the real space that can be experienced only in a situation separated from the real space.

  One technique for realizing the MR system is an alignment technique. This is a technique aimed at matching the position and orientation of the camera in the real space with the position and orientation of the virtual camera, and is a technique necessary for synthesizing the real space and the CG without a sense of incongruity.

  As a method for calculating the position and orientation of the camera in the real space, a method using a position / orientation measurement sensor, a method using image processing from the position of a feature point photographed by the camera (Patent Document 1), A method combining the technologies (Patent Document 2) has been proposed. However, in the above-described method, the depth information of the real space is insufficient, so that the relationship between the CG and the real object in the depth direction cannot be correctly expressed.

  Here, as shown in FIG. 9A, consider a case where the user performs some operation on the CG using his / her hand 102 with respect to the CG 101. Reference numeral 103 indicates a boundary portion between the CG and the real image. In this case, since there is no depth information of the hand, the depth relationship between the hand 102 and the CG 101 cannot be obtained. For this reason, as shown in FIG. 9B, a composite image in which the hand 102 is hidden by the CG 101 is obtained. Reference numeral 201 indicates a boundary portion between the CG 101 and the hand 102.

  As one of methods for dealing with this situation, a method of performing masking processing based on color information registered in advance has been proposed (Patent Document 3). In this method, rendering is performed in the priority order of “color region registered in advance in a real image” → “CG” → “real image”. For example, if the hand color is registered in advance, drawing is performed in the priority order of “hand” → “CG” → “real image other than hand”, and as shown by reference numeral 301 in FIG. A composite image in which the boundary portion with the hand 102 disappears is obtained. However, even in this method, since there is no depth information in the real space, the relationship in the depth direction cannot always be expressed correctly.

  Here, as a method for acquiring depth information of the real space, a method using a depth sensor using infrared rays, ultrasonic waves, or the like, a method for obtaining a distance using image processing from images taken by two or more cameras, and the like It has been known. However, in general, the resolution of the depth image is lower than the imaging resolution of the camera, and the resolution is often lowered as the measurable range is increased.

  Therefore, inaccurate depth information is obtained, and when such a depth information is used to synthesize a real image and a CG, as shown in FIG. 11A, a boundary portion 103 between the real image and the CG is displayed. Unnatural noise will occur. Further, when the resolution of the depth information is low, as shown in FIG. 11B, there may be a case where even the original correct positional relationship cannot be expressed. As one of methods for avoiding such a problem of depth images, a technique of super-resolution of a depth image using a real image taken by a camera has been proposed (Non-Patent Document 1).

JP 2003-319388 A JP 2007-164631 A JP 2003-29675 A

Qingxiong Yang, Ruigang Yang, James Davis, David Nister, Spatial-Depth Super Resolution for Range Images, CVPR 2007

  As described above, even if the depth information is obtained, if the resolution of the depth information is lower than the resolution of the real image or if the resolution is low, the synthesis of the real image and the CG becomes inaccurate. It was.

  Moreover, in order for the user to experience mixed reality, it is necessary to perform all processing in real time. However, in the above method (Non-Patent Document 1), since the super-resolution processing is performed on the entire area of the distance image, processing time is required and real-time performance may not be ensured.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique capable of synthesizing a real image and a virtual image (CG) accurately and at high speed.

In order to solve the above-described problem, an aspect of the present invention is an information processing apparatus that synthesizes a virtual image with a real image obtained by photographing a real space, and an acquisition unit that acquires depth information of each unit in the real space; The depth information calculating means for calculating the depth information of each part in the virtual image, the acquired depth information and the calculated depth information are compared for each corresponding position, and the difference exceeds the predetermined threshold value A region extracting means for extracting the virtual image as an extraction region, a correcting means for correcting the extraction region using color information in the real image, and prohibiting drawing of the virtual image in the extraction region corrected by the correcting means. Definition means for defining as a drawing prohibited area, drawing means for drawing a virtual image based on the drawing prohibited area, the real image and the drawn virtual Comprising a synthesizing means for synthesizing the image.

  According to the present invention, a real image and a virtual image (CG) can be synthesized accurately and at high speed.

The figure which shows an example of a structure of the information processing system which arranged and comprised the information processing apparatus 10 concerning one embodiment of this invention. 3 is a flowchart illustrating an example of a processing flow of the information processing apparatus 10 illustrated in FIG. 1. The flowchart which shows an example of the flow of the 2nd correction | amendment process shown to S107 of FIG. The figure which shows the outline | summary of the area | region (extraction area | region) extracted by the process of S104 of FIG. The figure which shows the outline | summary of the correction process shown to S106 and S107 of FIG. The figure which shows the outline | summary of the process of S108-S110 of FIG. 9 is a flowchart illustrating an example of a processing flow of the information processing apparatus 10 according to the second embodiment. The figure which shows the outline | summary of the process of S304 and S305 shown in FIG. The figure for demonstrating a prior art. The figure for demonstrating a prior art. The figure for demonstrating a prior art.

  Embodiments according to the present invention will be described below in detail with reference to the accompanying drawings.

(Embodiment 1)
FIG. 1 is a diagram showing an example of the configuration of an information processing system configured by arranging an information processing apparatus 10 according to an embodiment of the present invention.

  The information processing system includes an information processing device 10, an imaging device 21, a position / orientation measurement device 22, a depth value measurement device 23, and a display device 24.

  The information processing device 10, the imaging device 21, the position / orientation measurement device 22, the depth value measurement device 23, and the display device 24 have a built-in computer. The computer includes main control means such as a CPU (Central Processing Unit), and storage means such as ROM (Read Only Memory), RAM (Random Access Memory), and HDD (Hard Disk Drive). In addition, the computer may include other input / output means such as buttons, a display, or a touch panel, and communication means such as a network card. These components are connected by a bus or the like, and are controlled by the main control unit executing a program stored in the storage unit.

  Here, the imaging device 21 is realized by, for example, a CCD (Charge Coupled Device) camera or the like, and images a real space. The position / orientation measurement device 22 is realized by, for example, a gyro sensor or a porcelain sensor, and measures the position and orientation of the imaging device 21.

  The depth value measuring device 23 measures the depth value (depth information) of the real space using, for example, infrared rays, ultrasonic waves, or stereo images. The display device 24 is realized by a CRT liquid crystal monitor, for example, and displays an image (video) synthesized by the information processing device 10.

  Here, the information processing apparatus 10 has, as its functional configuration, an image acquisition unit 31, a position / orientation calculation unit 32, a storage unit 33, a Z buffer creation unit 34, a region extraction unit 35, and a depth value acquisition. A unit 36 and a correction unit 37 are provided. The information processing apparatus 10 is also provided with a stencil buffer creation unit 38, a CG drawing unit 39, and a synthesis unit 40. Hereinafter, functions of each unit will be described.

  The image acquisition unit 31 acquires an image of the real space photographed by the imaging device 21. The acquired real image is sent to the correction unit 37 and the synthesis unit 40. The real image is also sent to the position / orientation calculation unit 32 when the image is used for the position and orientation measurement processing of the imaging device 21.

  The position / orientation calculation unit 32 calculates the position and orientation of the imaging device 21. The position and orientation calculation unit 32 calculates the position and orientation (information) of the imaging device 21 by using at least one of the position and orientation measurement values measured by the position and orientation measurement device 22 and a real image.

  Here, when the position and orientation of the imaging device 21 are calculated using the measured values of the position and orientation, the position and orientation calculation unit 32 sets parameters indicating characteristics unique to the imaging device 21 and the attachment position of the position and orientation measurement device 22. And the position and orientation relationship between the lens center of the imaging device 21 and the lens center. Note that the parameter representing the characteristic unique to the imaging device 21 is, for example, information indicating the focal length, principal point position, and the like of the imaging device 21 and is stored in the storage unit 33 in advance.

  Further, when calculating the position and orientation of the imaging device 21 using a real image, the position / orientation calculation unit 32 includes parameters indicating characteristics unique to the imaging device 21, position / orientation definition information regarding an index existing in the real space, Position information relating to the index in the real image is used. The position and orientation (information) calculated by the position and orientation calculation unit 32 are sent to the Z buffer creation unit 34, the region extraction unit 35, and the CG drawing unit 39.

  The Z buffer creation unit 34 calculates a Z value (depth information) when a virtual image (CG) is projected based on the position and orientation information calculated by the position and orientation calculation unit 32. This creates a CG Z-buffer. The Z buffer indicates a memory area in which the depth information calculation result of each part in the virtual image is held as a Z value. The calculated Z value is sent to the region extraction unit 35.

  The depth value acquisition unit 36 acquires the depth value (depth map) of the real space measured by the depth value measurement device 23. In the depth map, depth information of each part in the real space photographed by the imaging device 21 is held as a depth value. The acquired depth value (depth map) is sent to the region extraction unit 35.

  The storage unit 33 stores various information. The storage unit 33 includes parameters representing characteristics unique to the imaging device 21 such as the focal length and principal point position of the imaging device 21, the position and orientation relationship between the attachment position of the position and orientation measurement device 22 and the lens center of the imaging device 21, and the like. Remembered. The storage unit 33 also stores position and orientation definition information of indices existing in the real space.

  The region extraction unit 35 compares the Z value of the CG calculated by the Z buffer creation unit 34 with the depth value of the real space acquired by the depth value acquisition unit 36 for each corresponding position, and the comparison result is obtained. Based on this, a region where the difference between the Z value and the depth value exceeds a predetermined threshold is extracted. This region is extracted from the real image. Note that the extraction region (extraction region) may be all regions where the predetermined threshold value is set to “0” and the Z value exceeds the depth value. In addition, when the resolution of the depth value is low, the threshold value may be set to a value smaller than “0” to include a region where the Z value is smaller than the depth value.

  The correction unit 37 performs correction processing on the region (extraction region) extracted by the region extraction unit 35. More specifically, the correction unit 37 corrects the range of the extraction region based on the color information of the extraction region.

  The stencil buffer creation unit 38 creates a stencil buffer that defines the area corrected by the correction unit 37 as a write prohibited area (drawing prohibited area). That is, CG rendering is prohibited in the area on the real image defined in the stencil buffer. The created stencil buffer is sent to the CG drawing unit 39.

  The CG drawing unit 39 draws a virtual image (CG) based on the stencil buffer created by the stencil buffer creating unit 38. More specifically, CG drawing is performed outside the area defined in the stencil buffer. The combining unit 40 combines the real image and the CG image. The synthesized video is displayed on the display device 24.

  The above is the description of an example of the configuration of the information processing system. Note that the functional configuration provided in the information processing apparatus 10 does not necessarily have to be realized as illustrated, and may be realized in whole or in part in any apparatus in the system.

  Further, the configurations of the various apparatuses are not necessarily realized as illustrated. For example, at least one of the imaging device 21, the position / orientation measurement device 22, the depth value measurement device 23, and the display device 24 may be provided as a configuration in the information processing device 10.

  Next, an example of the flow of processing in the information processing apparatus 10 illustrated in FIG. 1 will be described with reference to FIG.

  In this processing, first, imaging of the real space is performed in the imaging device 21. Then, the information processing apparatus 10 acquires an image (real image) captured by the imaging device 21 in the image acquisition unit 31 (S101). At this time, the depth value measurement device 23 measures the depth value of the real space, and the information processing device 10 acquires the depth value (that is, the depth map) in the depth value acquisition unit 36 (S102).

  The information processing apparatus 10 calculates the Z value when the CG is projected based on the position and orientation of the imaging device 21 calculated by the position and orientation calculation unit 32 in the Z buffer creation unit 34 (S103). That is, a CG Z buffer is created.

  The information processing apparatus 10 compares the depth value of the real space and the Z value of the CG for each corresponding position in the region extraction unit 35, and the difference between the Z value of the CG and the depth value of the real space is a predetermined threshold value. A region (extraction region) exceeding the threshold is extracted (S104). As described above, this extraction region is extracted from a real image.

  Here, the information processing apparatus 10 calculates (determines) color information representative of the extraction region in the correction unit 37 (S105). In this process, for example, a histogram of color information (luminance, color tone, etc.) in the extraction region may be created, and color information with the highest frequency may be calculated as a representative value (representative color information).

  In the information processing apparatus 10, the correction unit 37 corrects the range of the extraction region using the representative color information (first correction). More specifically, the representative color information and the color information of the pixel of interest in the extraction area are compared for each pixel. As a result of the comparison, if the color information of the target pixel is similar to the representative color information, the pixel is included in the extraction area, and if not similar, it is excluded from the extraction area (S106). Whether or not they are similar is determined by setting a predetermined reference (for example, a threshold value) for the color information, and whether the color information of the representative color information and the target pixel is more similar than the predetermined reference This may be done by comparing whether or not.

  Subsequently, in the correction unit 37, the information processing apparatus 10 further corrects the range of the extraction area processed in S106 (second correction) using the change amount of the color information of the area (S107). Details of this process will be described later.

  In the stencil buffer creation unit 38, the information processing apparatus 10 creates a stencil buffer that prohibits writing in the extraction region whose range has been corrected in the processing of S106 and S107 (S108). Then, the CG drawing unit 39 draws CG based on the created stencil buffer (S109). Thereafter, the information processing apparatus 10 combines the actual image acquired in the process of S101 and the CG drawn in the process of S109 in the combining unit 40 (S110), and ends this process.

  Here, the details of the second correction processing shown in S107 of FIG. 2 will be described with reference to FIG.

  The information processing apparatus 10 scans a real image in the correction unit 37 (S201). Then, it is determined whether or not the target pixel is the region (extraction region) extracted in the process of S106 in FIG. If it is not the extraction region (NO in S202), it is determined whether or not there is an unscanned pixel. If there is an unscanned pixel (YES in S203), the information processing apparatus 10 performs the process of S201 again. Do. On the other hand, if there is no unscanned pixel (NO in S203), the information processing apparatus 10 ends this process.

  If the result of the processing in S202 is an extraction region (YES in S202), the information processing apparatus 10 causes the correction unit 37 to within a predetermined range of the target pixel (region surrounding the target pixel: for example, eight neighborhoods of the target pixel (Inside), it is searched whether there is a pixel outside the extraction region (S204). If there is no pixel outside the extraction region (NO in S205), the information processing apparatus 10 determines whether there is an unscanned pixel. If there is an unscanned pixel (YES in S209), the information processing apparatus 10 performs the process of S201 again. On the other hand, if there is no unscanned pixel (NO in S209), the information processing apparatus 10 ends this process.

  As a result of the determination in S205, if there is a pixel other than the extraction region within the predetermined range of the target pixel (YES in S205), the information processing apparatus 10 causes the correction unit 37 to change the luminance around the target pixel. Calculate (S206). Note that the change in luminance may be obtained by, for example, differentiation of luminance or density, Laplacian, or the like.

  If the target pixel is an edge as a result of the process of S206 (YES in S207), the information processing apparatus 10 ends this process. If the target pixel is not an edge (NO in S207), the information processing apparatus 10 includes all of the target pixel within a predetermined range (for example, eight neighborhoods of the target pixel) in the extraction region (S208), and then the above-described S209. Proceed to the process. Note that in the processing of S207, an edge may be determined when the magnitude of the differential value exceeds a predetermined threshold value. Further, when the Laplacian code changes, it may be determined as an edge.

  Here, the outline of the region (extraction region) extracted in the process of S104 of FIG. 2 will be described with reference to FIG.

  Reference numeral 51 denotes a depth map of the real space, reference numeral 52 denotes a Z buffer of a virtual image (CG), and reference numeral 53 denotes a real image. Reference numerals 54 and 55 indicate regions (extraction regions) extracted based on the result of comparison between the depth value of the depth map 51 and the Z value of the Z buffer 52 in the depth map and the real image, respectively.

  Next, the outline of the correction process shown in S106 and S107 of FIG. 2 will be described with reference to FIG.

  Reference numeral 56 shows an enlarged view of the extraction region 55 shown in FIG. That is, it is an enlarged view of the extraction region 54 in the real image 53. Reference numeral 56 indicates a result of performing the first correction process on the extraction region 56. Reference numeral 57 indicates a result of performing the second correction process (see FIG. 3) on the extraction region 57 after the first correction process. That is, in the reference numeral 57, the range of the extraction area is corrected based on the representative color information, and in the reference numeral 58, the extraction area range is corrected based on the change amount of the color information.

  Next, the outline of the processing of S108 to S110 in FIG. 2 will be described with reference to FIG.

  Reference numeral 61 denotes a stencil buffer that defines the extraction area 64 after the second correction processing as a drawing prohibited area. Reference numeral 62 indicates a result of drawing the CG 65 based on the stencil buffer. Reference numeral 63 indicates a composite image 63 obtained by combining the CG drawing image 62 and the real image. In other words, at Numeral 62, CG is drawn outside the area defined by the stencil buffer, and at Numeral 63, the drawn CG is synthesized with the actual image.

  As described above, according to the present embodiment, the depth value of the real space and the Z value of the virtual image (CG) are compared to extract a region (extraction region) that is a candidate for the CG rendering prohibited region. Then, the range of the extraction area is corrected based on the color information, and a CG is drawn and combined based on the corrected extraction area.

  As described above, according to the present embodiment, in consideration of the depth relationship between the real image and the virtual image (CG), a composite video in which noise or the like is suppressed from occurring at the boundary portion between the CG and the real image is generated.

  Further, according to the present embodiment, since processing is performed only on the extraction region in order to suppress the occurrence of the noise and the like, the range to be processed can be suppressed, and the processing speed can be increased. That is, according to the present embodiment, it is possible to generate a composite video in which generation of noise or the like is suppressed in real time.

(Embodiment 2)
Next, Embodiment 2 will be described. The information processing apparatus 10 according to the second embodiment has the same configuration as that of FIG. Therefore, the description of the configuration of the information processing apparatus 10 according to the second embodiment is omitted. Here, the differences will be mainly described.

  Here, as a configuration different from FIG. 1 describing the first embodiment, a region extraction unit 35 and a correction unit 37 may be mentioned.

The region extraction unit 35 according to the first embodiment extracts a region from a real image, but the region extraction unit 35 according to the second embodiment extracts a region from a depth map.
In the correction unit 37 according to the first embodiment, the first correction process and the second correction process are performed to correct the range of the extraction area extracted from the real image. On the other hand, in the correction unit 37 according to the second embodiment, the correction process is performed on the extracted region extracted from the depth map, not the actual image. As the correction process, the super-resolution process is performed without performing the first correction process and the second correction process. Hereinafter, this point will be described more specifically.

  Here, an example of the flow of processing in the information processing apparatus 10 according to the second embodiment will be described with reference to FIG.

  The information processing apparatus 10 performs the same processing as S101 to S103 of FIG. 2 describing the first embodiment (S301 to S303). After that, the information processing apparatus 10 compares the depth value of the real space and the Z value of the CG for each corresponding position in the region extraction unit 35, and the difference between the Z value of the CG and the depth value of the real space is predetermined. An area (extraction area) exceeding the threshold is extracted (S304). As described above, in the second embodiment, the extraction region is extracted from the depth map.

  In the correction unit 37, the information processing apparatus 10 performs the super-resolution processing on the extraction region 54 (and / or the boundary region between the extraction region 54 and the other region) extracted in the process of S304 (S305). ). That is, the extraction area in the depth map is made super resolution. This super-resolution is performed using color information of a corresponding region (extraction region 55 and / or a boundary region between the extraction region 55 and other regions) in the real image. The super-resolution may be performed according to the method described in Non-Patent Document 1, for example.

  After that, the information processing apparatus 10 creates a stencil buffer in the stencil buffer creating unit 38 that prohibits writing the area that has been made super-resolution by the process of S305 (S306). The information processing apparatus 10 renders CG based on the created stencil buffer (S307). Thereafter, the information processing apparatus 10 combines the actual image acquired in the process of S301 and the CG drawn in the process of S307 in the combining unit 40 (S308), and ends this process.

  Here, the outline of the processing of S304 and S305 shown in FIG. 7 will be described with reference to FIG.

  Reference numeral 54 shows an enlarged view of the extraction region 54 shown in FIG. That is, the enlarged view of the extraction area | region in the depth map 51 is shown. Reference numeral 71 indicates a result of performing super-resolution on the extraction region 54 in the depth map 51 using the color information of the extraction region 55 in the real image. Note that the shape of the extraction area indicated by reference numeral 71 is changed because the resolution of the extraction area is increased by the super-resolution processing.

  As described above, according to the second embodiment, the depth value of the real space and the Z value of the virtual image (CG) are compared to extract the CG rendering prohibited area (extraction area). Then, after the extraction area is corrected by super-resolution, a CG is drawn and synthesized based on the corrected extraction area.

  Thereby, also in the second embodiment, in the same manner as the first embodiment, in consideration of the depth relationship between the real image and the virtual image (CG), it is possible to generate a composite video in which generation of noise and the like is suppressed in real time.

  The above is an example of a typical embodiment of the present invention, but the present invention is not limited to the embodiment described above and shown in the drawings, and can be appropriately modified and implemented without departing from the scope of the present invention. .

  For example, the present invention can take an embodiment as a system, apparatus, method, program, storage medium, or the like. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

An information processing apparatus that combines a virtual image with a real image obtained by photographing a real space,
Obtaining means for obtaining depth information of each part in the real space;
Depth information calculating means for calculating depth information of each part in the virtual image;
An area extracting means for comparing the acquired depth information and the calculated depth information for each corresponding position, and extracting an area where the difference exceeds a predetermined threshold as an extraction area;
Correction means for correcting the extraction region using color information in the real image;
Defining means for defining the extraction area corrected by the correcting means as a drawing prohibition area that prohibits drawing of the virtual image;
Drawing means for drawing a virtual image based on the drawing prohibited area;
An information processing apparatus comprising: a synthesis unit that synthesizes the real image and the drawn virtual image. The region extracting means includes
As the extraction area, an area where a difference between the acquired depth information and the calculated depth information from the real image exceeds a predetermined threshold is extracted.
The correction means includes
Correcting the range of the extraction region using color information in the real image;
The defining means is
The information processing apparatus according to claim 1, wherein the extraction area whose range is corrected by the correction unit is defined as the drawing prohibited area. The correction means includes
Determining means for determining color information representative of the extraction region;
The representative color information determined by the determining means and the color information of the extraction area are compared for each pixel, and the pixels in which the representative color information and the color information are similar to a predetermined standard are compared with the extraction area. The information processing apparatus according to claim 2, wherein pixels that are not included are excluded from the extraction region. The correction means includes
The information processing apparatus according to claim 2 or 3, wherein a boundary between the extraction region and other regions is changed based on a change amount of color information of the extraction region. The acquisition means includes
The acquired depth information is held in the depth map as a depth value,
The depth information calculating means includes
The calculated depth information is held in a Z buffer as a Z value,
The region extracting means includes
As the extraction region, extract a region where the difference between the depth value and the Z value exceeds a predetermined threshold from the depth map,
The correction means includes
The extracted area is super-resolution using the color information of the corresponding area in the real image,
The defining means is
The information processing apparatus according to claim 1, wherein the extraction area that has been super-resolved by correction by the correction unit is defined as the drawing prohibited area by using color information in the real image. A position / orientation calculating means for calculating the position and orientation of the imaging device that has captured the real space;
The depth information calculating means includes
6. The information processing apparatus according to claim 1, wherein depth information is calculated based on position and orientation information of the imaging apparatus calculated by the position and orientation calculation unit. A processing method of an information processing apparatus for combining a virtual image with a real image obtained by photographing a real space,
An obtaining unit obtaining depth information of each part in the real space;
Depth information calculating means calculating depth information of each part in the virtual image;
A region extracting means comparing the acquired depth information and the calculated depth information for each corresponding position, and extracting a region where each difference exceeds a predetermined threshold as an extraction region;
A step of correcting the extraction region using color information in the real image;
A defining unit defining the corrected extraction region as a drawing prohibited region that prohibits drawing of the virtual image;
A step of drawing a virtual image based on the drawing prohibited area;
A processing method comprising: a step of combining the real image and the drawn virtual image. A computer built in an information processing device that synthesizes a virtual image with a real image taken of a real space,
Acquisition means for acquiring depth information of each part in the real space;
Depth information calculating means for calculating depth information of each part in the virtual image;
A region extracting means for comparing the acquired depth information with the calculated depth information for each corresponding position and extracting a region where the difference exceeds a predetermined threshold as an extraction region;
Correction means for correcting the extraction region using color information in the real image;
Defining means for defining the extraction area corrected by the correcting means as a drawing prohibition area that prohibits drawing of the virtual image;
Drawing means for drawing a virtual image based on the drawing prohibited area;
A program for functioning as a synthesis means for synthesizing the real image and the drawn virtual image.