JP5025496B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention is characterized in that, in an image generation method based on computer graphics, in particular, a first virtual object and a second virtual object representing a real object are included in a three-dimensional space, and optically correct drawing is performed. Related to technology.

  Computer graphics (hereinafter referred to as CG) are superimposed on the background of a live-action scene and presented to the viewer as if a virtual object existed on the spot. There is a mixed reality (MR) technique.

  Conventionally, a system in which a virtual object (CG) casts a shadow on a real object (floor) by combining the mixed reality technology and ray tracing (ray tracing) has been proposed (Non-Patent Document 1). The drawing system shown in Non-Patent Document 1 generates a mixed reality image by drawing and superimposing a CG object by a ray tracing method against a background of a real image acquired as a video image. In an example of drawing using this system, a state in which a CG sphere as a virtual object casts a shadow on a floor as a real object is expressed. In order to realize such an expression, a virtual object representing an actual floor is defined. Then, an image of a shadow falling on a virtual object corresponding to the floor is generated and synthesized with a real image.

In addition, based on the result of sensing the position and orientation of a real object, a method for expressing the correct depth relationship between the virtual object and the real object in the mixed reality space by placing a CG model that represents the real object in the mixed reality space is proposed. (Patent Document 1).
JP 2005-293142 A Interactive Mixed Reality Rendering in a Distributed Ray Tracing Framework Andreas Pomi, and Philipp Slusallek IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR) 2004, Student Colloquium, Arlington, USA, November 2-5,2004

  In general, when a human grasps the positional relationship between objects from visual information, a shadow image is used as a clue. Therefore, even in the case of mixed reality, in order for the observer to correctly grasp the positional relationship between the objects, it is indispensable to correctly express the shadow. In other words, not only the shadow between the real objects and the shadow between the virtual objects, but also an expression in which the real object and the virtual object cast a shadow on each other is necessary.

  In Non-Patent Document 1, a shadow that falls from a virtual object to a virtual object (represented as a transparent object) representing a floor that is a real object is drawn. In Non-Patent Document 1, two virtual objects for representing a real object in a virtual space are a floor and a human. In real space, humans cast shadows on the floor. The shadow image in this case appears in an image taken in real space. Therefore, as a matter of course, it is not necessary to draw a human shadow with CG.

  However, since Non-Patent Document 1 does not consider such a case, an unnecessary shadow image is generated in addition to the shadow in the real image.

  This problem may also occur when a shadow (self-shadow) occurs inside a virtual object that imitates one real object. For example, when the observer rolls his / her hand and looks at the rolled palm, a darker shadow is generated due to a shadow of a virtual object imitating a real object in addition to a shadow actually generated. This problem is not limited to the ray tracing method, and can be said to be a general image generation method using CG.

  FIG. 1 is a principle diagram illustrating the above problem. In FIG. 1, a virtual light source 101, an observer's hand 102, a virtual object sphere 103, and a floor 104 are arranged in this order from the top of the scene. It is assumed that the actual light source situation matches the virtual light source situation.

  Here, a virtual object other than a realistic approximate object is referred to as a “non-realistic approximate object”, and all virtual objects including the real approximate object and the non-realistic approximate object are referred to as “all virtual objects”.

An optically correct drawing in the principle diagram will be described.
1. The observer's hand 102 (real object) casts a shadow on the sphere 103 (non-realistic approximate object) (shadow (1) 105).
2. The observer's hand 102 (real object) casts a shadow on the floor 104 (real object) (shadow (2) 106).
3. The sphere 103 (unrealistic approximate object) casts a shadow on the floor 104 (real object) (shadow (3) 107)
4). In the area where the real shadow (shadow (2) 106) and the virtual shadow (shadow (3) 107) overlap, only the real shadow is drawn and the virtual shadow is not drawn (shadow (4) 108).

  Shadow (1) 105 is a shadow that does not actually exist. Therefore, in order to present the shadow (1) 105 to the observer, it is necessary to calculate the shadow of the CG and draw it on the sphere. Here, in order for the real object to cast a shadow on the virtual object, it is necessary to define the shape information of the real object as a virtual object in the virtual space.

  In this proposal, a virtual object that simulates a real object defined in the virtual space is called a real approximate object (details of the real approximate object will be described later).

  When drawing the surface of a sphere (non-realistic approximation object), it is possible to draw the shadow of the observer's hand on the sphere by drawing the shadow of the real approximation object defined in the virtual space with CG. become.

  Next, the observer's hand casts an actual shadow on the floor (shadow (2) 106). This is a natural phenomenon and it is not necessary to draw a shadow with CG in the first place.

  The sphere 103 needs to cast a shadow on the floor 104 (shadow (3) 107). In order to realize this effect, it is necessary to define a floor in the virtual space as a physical approximation object and calculate a shadowed area as in the observer's hand.

  Finally, shadow (4) 108 is an area where shadow (2) 106 and shadow (3) 107 overlap. In this area, since the shadow of CG is drawn on the actual shadow, the shadow becomes darker than the actual shadow.

  An image processing apparatus according to the present invention for solving the above-described problem has the following configuration.

An image processing apparatus that draws a three-dimensional space including a realistic approximate object and a virtual object using computer graphics, and calculates a shadow area that the real approximate object drops in the three-dimensional space. The calculation means, the second shadow area calculation means for calculating the shadow area that the virtual object drops in the three-dimensional space, the area calculated by the first shadow area calculation means, and the second shadow area calculation means Determining an area for generating a shadow of the virtual object based on an overlapping area between the area calculated by the first shadow area calculating means and the area calculated by the second calculating means when the areas overlap. It is characterized by.

  ADVANTAGE OF THE INVENTION According to this invention, in the image generation method by computer graphics, it becomes possible to represent correctly the drawing of the shadow which a real object and a virtual object produce mutually.

  Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the accompanying drawings.

  The image processing apparatus according to the present embodiment has a region in which a realistic approximate object that is a noble shape existing in a three-dimensional space corresponding to a region occupied by a real object casts a region from a region where a non-realistic approximate object casts a shadow By excluding, the shadow area of only the unrealistic approximate object is calculated.

  Since the measurement / estimation method of the real object in the present embodiment uses a known technique, a detailed description thereof is omitted here. As an example, three-dimensional reconstruction by stereo matching using a plurality of cameras, a measurement / estimation method using a measurement device such as a three-dimensional range finder, and the like can be given.

(Calculate unrealistic approximate object area by subtracting shadow area)
FIG. 2 is a block diagram for explaining the configuration of the image processing apparatus according to the first embodiment. A virtual space database (hereinafter referred to as DB) 201 stores information such as the shape, material, light source, and virtual viewpoint position and orientation of all virtual objects. The virtual space DB 201 is sequentially updated when the position information of each virtual object is changed by a user operation or when attribute information such as a material is changed.

  The real object measurement / estimation unit 202 measures or estimates the geometric shape / position / attitude and material information of the real object, and outputs the measured information to the virtual space DB 201. In the present embodiment, depth estimation is performed by stereo matching from a photographed image obtained from the photographed image input unit 203. Based on the result obtained by the depth estimation, three-dimensional shape estimation is performed and registered in the virtual space DB 201.

  The virtual object output to the virtual space DB 201 by the real object measuring / estimating unit 202 is given attribute information indicating that it is a realistic approximate object. The attribute information given here is binary information indicating whether it is a realistic approximate object or a general virtual object (non-realistic approximate object). However, there is no limitation on the method for distinguishing the information given here. Further, instead of adding the attribute information, the real approximate object and the non-realistic approximate object may be distinguished from the attribute information such as the transparency of the virtual object and the name assigned to the virtual object. Based on these pieces of information, the shadowed area is calculated.

  The photographed image input unit 203 takes in video (image) taken by a video camera provided at the observer's head position as input data. The real image obtained as an input is supplied to the real object measurement / estimation unit 202 and the pixel value calculation unit 207.

  The viewpoint position and orientation control unit 204 controls the position and orientation of the virtual viewpoint by detecting the head position and orientation of the observer. As a method for detecting the head position / posture of the observer, there are a method of measuring by performing position / posture measurement using a six-degree-of-freedom sensor, photographing a two-dimensional marker with a camera, and converting it to a head position / posture. Since the method for detecting the head position and orientation of the observer uses a known technique, a detailed description thereof is omitted here.

  The first shadow area calculation unit 205 performs a process of calculating a shadow area of a realistic approximate object registered in the virtual space DB 201. There is a shadow map method for calculating the shadow.

  The shadow map is a method of calculating an area where a shadow is dropped by rendering a depth value (depth information) of a virtual scene viewed from each virtual light source. The depth value obtained by rendering is stored as a depth buffer or texture information. Image generation using a shadow map requires two-pass rendering.

  First, a shadow map is created in the first pass rendering. The shadow map renders the z value of the projection space with the virtual viewpoint at the position of the virtual light source. When the shadow map is visualized using the z value of the projection space, texture information is obtained in which the front color is black and the back color is white.

  In the second pass rendering, the scene of the virtual space DB 201 is rendered while comparing the depth value of the virtual object and the value of the depth buffer. The value to be compared is the distance from the virtual light source.

  As a comparison method, first, the depth value viewed from the virtual light source is written as a shadow map. Next, the shadow map is mapped on the screen viewed from the normal virtual viewpoint. On the other hand, the distance from each virtual light source is obtained for each pixel on the virtual object when the virtual object is observed from the normal virtual viewpoint. Then, the distance between the shadow map value and the calculated virtual light source for each pixel is compared, and a portion having a small shadow map value is determined as a shadow.

  The pixel determined to have the shadow fall can be calculated by subtracting the luminance of the light source from the pixel value before the shadow is dropped based on the luminance information of the light source causing the shadow. This process is repeated for each light source. Here, since the relationship between the light source luminance and the brightness of the object surface is linear, it is possible to calculate the shadow density by subtracting the luminance of each light source independently. In this way, the area where the shadow is dropped is calculated.

  The above is the outline of the shadow map. Since the calculation method using the shadow map is a method that is widely used as a publicly known technique, further detailed description is omitted.

  The second shadow area calculation unit 206 calculates an area where only the non-realistic approximate object registered in the virtual space DB 201 casts a shadow. The same method as that of the first shadow region calculation unit 205 is used as a method for calculating the region where the shadow is dropped.

  The pixel value calculation unit 207 renders based on the information of the virtual space DB 201 and the first and second shadow area calculation units according to the virtual viewpoint position acquired by the viewpoint position control unit 204 using the live-action image input unit 203 as a background image. To determine the pixel value.

  The screen output unit 208 outputs the result of all pixel values calculated by the pixel value calculation unit 207 to the display device.

  FIG. 3 is a diagram showing a flowchart of the image processing apparatus according to the first embodiment of the present invention. When the process is started, initialization is performed in step S301. Specifically, a storage area necessary for execution of processing is secured.

  In step S302, the viewpoint position / posture control unit 204 acquires the head position / posture information of the observer. A known technique is used for conversion from the head position / posture information to the viewpoint position / posture information.

  In step S303, the real object measurement / estimation unit 202 takes in the shape / material information / real light source information of the real object. A well-known technique is used for the acquisition method of the shape / material information of the real object and the acquisition method of the actual light source information. Here, if the direction and density of the shadow by CG are different from the direction and density of the actual shadow, it becomes a major factor that impairs the sense of reality. Therefore, it is desired that information on the position, orientation, and strength of the actual light source can be faithfully reflected in the virtual space.

  Conventionally, a method for reproducing an actual light source environment in a virtual space has been proposed. As an example, it is possible to calculate and estimate actual light source information (position, orientation, strength) by extracting and analyzing highlight components from live-action images acquired as video images (Katsufumi Ikenai, Yoichi Sato, Tsuyoshi Nishino, Imari Sato: “Realization of optical consistency in mixed reality,” Special Issue on Virtual Reality Society, “Mixed Reality”, Vol.4, No.4, pp.623-630 , December 1999).

  As described above, since a variety of methods for acquiring a three-dimensional shape and methods for estimating a light source in a real environment have been proposed, a suitable means may be selected according to the system to be constructed.

  In step S304, the virtual space DB 201 is updated based on the results of steps S302 and S303. Specifically, the observer's head position / posture information acquired in step S302 is assigned to the virtual viewpoint position / posture in the virtual space. Then, the real object shape information acquired in step S303 is registered in the virtual space.

  Here, in order to register the shape of the realistic approximate object in the virtual space, a triangular patch is generated from the point cloud data having the three-dimensional position information and polygonized. In addition, the shape of the real object does not necessarily need to be a polygon, and may be any shape expressed by a NURBS curved surface. Depending on the system to be constructed, a realistic approximation object may be represented by a method such as using a metaball representation in addition to a polygon or mesh representation.

  Further, the virtual light source is arranged in the virtual space according to the light source information of the real environment acquired in step S303. In step S305, the pixel value calculation unit 207 performs a rendering process based on the viewpoint position and orientation acquired in step S302. Details of the rendering process will be described later. In step S306, the screen output unit 208 draws the image generated in step S305 on the display device.

  In step S307, it is determined whether or not to end the system. If the system is not terminated, the process returns to step S302. When the system is terminated, the termination process is performed based on the program termination process.

  The above is the overall flow of the image processing apparatus according to the present embodiment. Next, details of the rendering portion will be described. FIG. 4 is a diagram for explaining the flow of the rendering portion.

  In step S401, a region where the realistic approximate object casts a shadow is calculated. In the calculation of the area where the shadow is dropped in step S401, a virtual viewpoint is arranged in each virtual light source, depth information of the virtual space DB 201 is calculated, and a shadow map is generated. At this time, the first shadow area calculation unit 205 generates a shadow map A from the virtual space DB 201 with a realistic approximate object as a calculation target. In step S402, the shadow map B is generated from the virtual space DB 201 using only the unrealistic approximate object as a calculation target. The process of calculating these two shadow maps is repeated for each light source. Then, a shadow map A and a shadow map B are generated for each light source and stored in the memory as texture information.

  Here, since the calculation of the shadow map A and the shadow map B is completely independent, each CPU can be assigned to each shadow map generation and can be calculated in parallel.

  In step S <b> 403, the image acquired by the live-action image input unit 203 is written into the frame buffer by the pixel value calculation unit 207 in order to draw the live-action image as the background.

  In step S <b> 404, the pixel value calculation unit 207 performs a process of drawing an unrealistic approximate object to be superimposed on the background real image.

  In step S405, the shadow area of the scene viewed from the virtual viewpoint is calculated based on the shadow map calculated in steps S401 and S402. The area where the shadow is actually drawn by CG can be calculated by subtracting the shadow area dropped by the realistic approximate object from the shadow area dropped by the non-realistic approximate object.

  The shadow area calculation in step S405 is performed by inputting virtual viewpoint information from the viewpoint position / orientation control section 204 to the first shadow area calculation section and the second shadow area calculation section and referring to the shadow map for each light source. To do. Then, a difference is taken between the shadow area B calculated with reference to the shadow map B at the virtual viewpoint and the shadow area A calculated from the shadow map A. The difference between the shadow area B and the shadow area A can be calculated by performing mask processing (or pixel value subtraction processing) on the two-dimensional image. In this way, the shadow area dropped only by the unreal approximation object is calculated.

  As a result, it is possible to draw a shadow with CG only in the shadow area where only the non-realistic approximate object is dropped, and it is possible to draw the shadow of the CG without hindering the actual shadow.

  Here, as a method for calculating the shadow area, in addition to the shadow map described in the present embodiment, a shadow volume method for calculating a shadow influence range as volume data or a precomputed radiance transfer (Precomputed Radiance Transfer). ) To calculate global illumination is also widely used. The present invention can be applied to any of the above, and there is no limitation on the shadow drawing technique.

  Further, in the present embodiment, the case where the CG sphere casts shadows on the observer's hand and the floor in FIG. 1 has been described. This embodiment is also effective for other arrangements. For example, this is the case where the vertical relationship between the CG sphere in FIG. 1 and the observer's hand is reversed. As described above, according to the present embodiment, the virtual object and the real object can be applied even if they have an arbitrary vertical relationship, and can be applied regardless of the number of objects.

  In addition, instead of calculating the shadow area of only the non-realistic approximation object, calculate the shadow area to be obtained by excluding the shadow area of the real approximation object as well when calculating the shadow area for all virtual objects. Is possible.

(Once all the virtual objects' shadows are drawn, cancel the shadow drawing)
The image processing apparatus according to the first embodiment subtracts a shadow area that only a real approximate object drops from a shadow area that all virtual objects drop, thereby generating a shadow area that is drawn by CG without hindering an actual shadow. It was something to calculate. The image processing apparatus according to the second embodiment subtracts the influence of the shadow caused by the CG dropped by the real approximation object when calculating the individual pixel values.

  Since the principle diagram and configuration diagram according to the present embodiment are the same as the principle diagram and configuration diagram of the first embodiment, description thereof will be omitted.

  FIG. 5 is a diagram for explaining the flow of rendering processing according to the second embodiment. The difference from the first embodiment is the processing in steps S501 and S502 in FIG. In step S402, the target for creating the shadow map is all virtual objects.

  In step S501, a shadow is drawn with CG on the image generated in step S404 based on the shadow area (shadow map B) dropped by all virtual objects. At this stage, the CG shadow is drawn in the shadow area dropped by all virtual objects including the realistic approximation object, so that the drawing is not correct (there is an area where the actual shadow and the CG shadow overlap).

  In step S502, a process of canceling the extra shadow area (the area where the real shadow and the CG shadow overlap) drawn in step S501 based on the shadow area (shadow map A) dropped only by the realistic approximate object is performed. This cancellation processing is realized by recalculating the value of each pixel in the shadow area calculated by the shadow map A. Specifically, the shadow map A is referred to, all the virtual light sources that are blocked by the realistic approximation object are calculated, and the luminance value of all the virtual light sources is added to the target pixel to cancel the shadow drawn by CG. It becomes possible.

  In this way, once the shadow area dropped by all the virtual objects is drawn, the shadow of the CG without obstructing the actual shadow by canceling the influence of the shadow caused by the CG in the shadow area dropped by the realistic approximate object. Can be drawn.

(Performs a shadow drawing judgment process in units of pixels)
The image processing apparatus according to the third embodiment performs a shadow drawing calculation for each pixel when drawing an unreal object. Since the principle diagram and configuration diagram according to the present embodiment are the same as the principle diagram and configuration diagram of the first embodiment, description thereof will be omitted.

  FIG. 6 is a diagram for explaining the flow of rendering processing according to the third embodiment. The difference from the first embodiment is the processing in steps S601 to S603 in FIG. Therefore, the processing from step S401 to step S404 is the same as the processing in the first embodiment, and a description thereof will be omitted.

  However, since the processing from step S601 to step S603 is a processing flow for each pixel in step S404, step S404 calculates the pixel value of the unreal approximation object, and then draws a shadow on the pixel. Whether to determine whether or not.

  In step S601, it is determined whether to draw a CG shadow on the target pixel. Specifically, first, the shadow map B is referred to, and it is checked whether or not the unreal approximation object casts a shadow on the target pixel. If the non-realistic approximation object does not cast a shadow on the target pixel, it is not necessary to draw a shadow with CG, and the process proceeds to the next process.

  Next, with reference to the shadow map A, it is examined whether or not the actual approximation object casts a shadow on the target pixel. When a realistic approximate object and a non-realistic approximate object cast a shadow, since there is a shadow in reality, it is not necessary to draw a shadow with CG, and the process proceeds to the next process.

  Then, only in the case where only the unrealistic approximate object casts a shadow, the process proceeds to step S602 to perform a process of drawing a shadow on the target pixel.

  Here, since the shadow map exists for each light source, the determination process in step S601 needs to repeatedly refer to the shadow map for each light source.

  In step S602, a shadow is drawn on the target pixel. Specifically, the virtual light source that causes the target pixel to cast a shadow is identified, and the shadow is drawn by subtracting the luminance value of the virtual light source from the pixel value of the actual image or the non-realistic approximation object that is the background It becomes possible.

  The virtual light source can be specified by specifying the virtual light source that generated the shadow map A that casts a shadow on the target pixel.

  In step S603, it is determined whether the calculation has been completed for all the pixels on the virtual screen. If the calculation has not ended, the process returns to step S404 to calculate the next pixel value. When the calculation is completed for all pixels, the rendering process is terminated.

  The object of the present invention can also be achieved as follows. That is, a storage medium in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. It goes without saying that the purpose is achieved even in this way.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the computer-readable storage medium storing the program code constitutes the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  Further, the embodiments according to the present invention are not limited to the case where the functions of the above-described embodiments are realized by executing the program code read by the computer. For example, an OS (operating system) running on a computer performs part or all of actual processing based on an instruction of the program code, and the functions of the above-described embodiments may be realized by the processing. Needless to say, it is included.

  Furthermore, the functions of the embodiment according to the present invention are also realized as follows. That is, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Then, based on the instruction of the program code, the CPU provided in the function expansion board or function expansion unit performs part or all of the actual processing. It goes without saying that the functions of the above-described embodiments are realized by this processing.

It is a principle figure explaining the principle of the image processing apparatus which concerns on 1st Embodiment. It is a block diagram explaining the structure of the image processing apparatus which concerns on 1st Embodiment. 4 is a flowchart for explaining a processing flow of the image processing apparatus according to the first embodiment. 6 is a flowchart illustrating details of rendering processing of the image processing apparatus according to the first embodiment. 10 is a flowchart illustrating details of rendering processing of the image processing apparatus according to the second embodiment. 14 is a flowchart illustrating details of rendering processing of the image processing apparatus according to the third embodiment.

Claims (7)

The 3-dimensional space and a real approximate the object and the virtual object, an image processing apparatus for drawing by using computer graphics,
First shadow region calculation means for calculating a shadow region that the realistic approximation object drops in the three-dimensional space;
A second shadow area calculation means for calculating a shadow area that the virtual object drops in the three-dimensional space;
When the area calculated by the first shadow area calculation means and the area calculated by the second shadow area calculation means overlap, the area calculated by the first shadow area calculation means and the area calculated by the second calculation means And an area where a shadow of the virtual object is generated is determined based on an area where the two overlap. Moreover, the area calculated by said first shadow region calculating means, and were calculated by the second shadow region means a region, from adding the combined area, in excluding the area calculated by said first shadow region calculating means area The image processing apparatus according to claim 1, further comprising a pixel value calculation unit that adds an influence of a shadow to the pixel value.   The pixel value calculation means cancels the influence of the shadow that the actual approximate object drops in the three-dimensional space from the pixel value for the pixels in the area calculated by the second shadow area calculation means. An image processing apparatus according to 1. The pixel value calculating means, and area calculated by said first shadow region calculating means, wherein the calculated region by the second shadow area means, based on, of whether to add the effect of shadows on the pixel value in pixel units The image processing apparatus according to claim 2, wherein a determination is made. This is an image processing method performed by an image processing apparatus that draws, using computer graphics, a three-dimensional space including a virtual object and a real approximate object that is a geometric shape existing in a three-dimensional space corresponding to a region occupied by a real object. And
A first shadow region calculation step in which a first shadow region calculation means of the image processing device calculates a shadow region that the real approximation object drops in a three-dimensional space;
A second shadow area calculating step in which the second shadow area calculating means of the image processing apparatus calculates a shadow area that the virtual object drops in the three-dimensional space;
When the region calculated by the first shadow region calculation step and the region calculated by the second shadow region calculation step overlap, the region calculated by the first shadow region calculation step and the region calculated by the second calculation step An area for generating a shadow of the virtual object is determined based on the overlapping area.   A program causing a computer to execute the image processing method according to claim 5.   A computer-readable recording medium storing the program according to claim 6.

Images