JP6025519B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to image processing in stereoscopic image display technology.

  Conventionally, as a method of creating a shadow of a virtual object represented by computer graphics, there is an image composition technique for synthesizing a virtual object and its shadow on an image obtained by capturing a real space by using a light source environment of the imaging scene. Known (for example, Patent Document 1). This image composition technology appropriately reflects the light source environment in the real space by referring to the light parameters set for a plurality of positions in the real space corresponding to the positions where the virtual objects are placed. A shadow was created in the image.

JP 2011-60195 A

  However, the above-described conventional technique only creates a shadow of a virtual object in an image. Currently, 3D image display technology that allows a viewer to perceive a stereoscopic image using binocular parallax has become widespread. This technique is a technique for perceiving a three-dimensional image by separately presenting to the left and right eyes images (parallax images) that differ in appearance by the amount of parallax between both eyes with respect to the same virtual object. It is impossible with the above-described conventional technology to create a shadow to be generated by a stereoscopic image that appears to jump out of such a screen on a floor or a wall in a real space.

  The image processing apparatus according to the present invention, when displaying a parallax image to make an observer perceive a stereoscopic image of a virtual object, creates a shadow image for creating a pseudo shadow of the stereoscopic image in a real space. It is generated based on data specifying the position and shape of the virtual object, and information indicating the position and size of the projection area by the image projection means for projecting the shadow image onto the real space.

  According to the present invention, a pseudo shadow or the like of a stereoscopic image generated by a parallax image can be created in a real space. Thereby, the realistic sensation of the video is improved.

1 is a diagram illustrating a system configuration example of a stereoscopic image display system 100 according to Embodiment 1. FIG. It is a figure which shows the internal structure of an image processing apparatus. 1 is a functional block diagram of an image processing apparatus according to Embodiment 1. FIG. 3 is a flowchart illustrating a flow of a series of processes in the image processing apparatus according to the first embodiment. It is a figure which shows the relationship between an observer's viewpoint position and a three-dimensional display area. It is a figure which shows an example of the parallax image by which the virtual object is arrange | positioned. It is a figure explaining the production | generation process of a shadow image. It is a figure which shows an example of a shadow image. FIG. 6 is a functional block diagram of an image processing apparatus according to a second embodiment. 12 is a flowchart illustrating a flow of a series of processes in the image processing apparatus according to the second embodiment. 12 is a flowchart illustrating a flow of processing for deriving a position of a stereoscopic image according to the second embodiment. It is a figure explaining the process in which the position of a stereo image is derived | led-out.

[Example 1]
In the first embodiment, model data representing the position and shape of a virtual object is acquired as position information of a stereoscopic image, a shadow image corresponding to this position information is generated, and the generated shadow image is projected onto a real space. Now, an aspect of creating a pseudo shadow of a stereoscopic image in the real space will be described.

  FIG. 1 is a diagram illustrating a system configuration example of a stereoscopic image display system 100 according to the present embodiment. The stereoscopic image display system 100 includes an image processing apparatus 101 that performs a shadow image generation process, a liquid crystal 3D display 102 as a stereoscopic image display apparatus, and a liquid crystal projector 103 as an image projection apparatus.

  The image processing apparatus 101 according to the present exemplary embodiment generates a parallax image including a left-eye image and a right-eye image using model data. The generated parallax image data is displayed on the liquid crystal 3D display 102, thereby displaying a stereoscopic image 104 of the virtual object. In addition, the image processing apparatus 101 generates a shadow image 105 according to the model data. The generated shadow image data is projected onto the floor in the real space by the liquid crystal projector 103, thereby creating a pseudo shadow of the stereoscopic image.

  In the following, the origin of the three-dimensional coordinates used in this embodiment is the screen center O of the liquid crystal 3D display 102, and the x, y, and z axes are the horizontal, vertical, and normal lines of the liquid crystal 3D display 102, respectively. This will be described as a direction.

  FIG. 2 is a diagram illustrating an internal configuration of the image processing apparatus 101.

  The image processing apparatus 101 includes a CPU 201, RAM 202, ROM 203, hard disk drive (HDD) 204, HDD I / F 205, input I / F 206, output I / F 207, and system bus 208.

  The CPU 201 executes programs stored in the ROM 203 and the HDD 204 using the RAM 202 as a work memory, and comprehensively controls each component to be described later via the system bus 208. Thereby, various processes described later are executed.

  The HDD I / F 205 is an interface such as serial ATA (SATA), for example, and connects the HDD 204 as a secondary storage device. The CPU 201 can read data from the HDD 204 and write data to the HDD 204 via the HDD I / F 205. Further, the CPU 201 can expand the data stored in the HDD 204 in the RAM 202 and similarly can store the data expanded in the RAM 202 in the HDD 204. The CPU 201 can regard the data expanded in the RAM 202 as a program and execute it. The secondary storage device may be a storage device such as an optical disk drive in addition to the HDD.

  The input I / F 206 is a serial bus interface such as USB or IEEE1394. The input I / F 206 connects various input devices such as a keyboard / mouse 209 and a position sensor 210 using infrared rays or electromagnetic waves. The CPU 201 can acquire various data from the keyboard / mouse 209 and the position sensor 210 via the input I / F 206.

  The output I / F 207 is a video output interface such as DVI or HDMI, and connects the liquid crystal 3D display 102 and the projector 103. Through this output I / F 207, parallax image data is sent to the liquid crystal 3D display 102, and shadow image data is sent to the liquid crystal projector 103. The liquid crystal 3D display 102 that has received the parallax image data displays the parallax image on the screen. (Hereinafter, the screen area on which the parallax image is displayed in the stereoscopic image display device is referred to as a “stereoscopic image display area”.) The liquid crystal projector 103 that has received the shadow image data projects the shadow image onto a floor or the like in the real space. To do.

  In the system shown in FIG. 1, the liquid crystal 3D display 102 is used as the stereoscopic image display device, but is not limited thereto. For example, a plasma 3D display or an organic EL 3D display may be used instead of the liquid crystal 3D display. Further, although the liquid crystal projector 103 is used as the image projection apparatus, a DLP projector or an LCOS projector may be used.

  FIG. 3 is a functional block diagram of the image processing apparatus 101 according to the present embodiment. The image processing apparatus 101 includes a viewpoint position information acquisition unit 301, a display area information acquisition unit 302, a model data acquisition unit 303, a projection characteristic information acquisition unit 304, a parallax image generation unit 305, and a projection image generation unit 306.

  The viewpoint position information acquisition unit 301 acquires information indicating the positions of the left and right eyes of the observer (hereinafter referred to as “viewpoint position information”). In the present embodiment, three-dimensional coordinates EL (xL0, yL0, zL0) and ER (xR0, yR0, zR0) indicating the positions of the left and right eyes of the observer are acquired via the position sensor 210, for example. Alternatively, a plurality of cameras that capture the face of the observer are prepared separately, and the position of the eye on the obtained image is obtained by using face recognition technology for the obtained face images, and the position of the eye on the obtained image And three-dimensional coordinates of the left and right eyes may be derived based on the captured camera posture information. The acquired viewpoint position information is sent to the parallax image generation unit 305.

  The display area information acquisition unit 302 acquires information (hereinafter referred to as “display area information”) of the size (W × H) of the stereoscopic image display area in the stereoscopic image display device (here, the liquid crystal 3D display 102). . For example, in the case of a 16: 9 50-inch display, the size of the stereoscopic image display area is a value such as W: 110 cm and H: 62 cm. The display area information may be acquired by a user input via the keyboard / mouse 209 or the like, or may be acquired by storing in advance in the HDD 204 or the like and reading it. The acquired display area information is sent to the parallax image generation unit 305.

  The model data acquisition unit 303 acquires model data for specifying the position and shape of a virtual object displayed as a stereoscopic image from a secondary storage device such as the HDD 204 via the HDD I / F 205. The model data in this embodiment is mesh data. Here, the mesh data is data necessary for displaying a mesh model that expresses the shape of an object with a set of plate-like elements having a plurality of sides (for example, coordinate data of each vertex of a quadrilateral). . The data format of the model data is not particularly limited as long as the position and shape of the virtual object can be specified. For example, the model data may be a parametric model expressed by a NURBS surface. Further, the model data may include information representing the color and texture of the virtual object such as the reflectance, transmittance, and refractive index of the virtual object. In this embodiment, the model data is used as the position information of the stereoscopic image. The acquired model data is sent to the parallax image generation unit 305 and the projection image generation unit 306.

  The projection characteristic information acquisition unit 304 acquires projection characteristic information indicating the position and size of the projection area of an image projection apparatus (here, the liquid crystal projector 103) that projects a shadow image. In this embodiment, the three-dimensional coordinates P0 (x0, y0, z0) indicating the position of the liquid crystal projector 103, the point P1 (x1, y1, z1) through which the optical axis of the liquid crystal projector 103 passes, the horizontal angle of view θ, the vertical image Angle ψ, horizontal resolution Wp, and vertical resolution Hp are used as projection characteristics. The projection characteristic information is obtained by storing data obtained by measurement in advance in the HDD 204 or the like, and reading it through the HDD I / F 205 at the time of processing execution. Alternatively, a separate position sensor or tilt sensor may be attached to the liquid crystal projector 103, and the position and orientation may be acquired via the input I / F 206, or the image may be displayed via the output I / F 207 capable of bidirectional communication from the liquid crystal projector 103. A corner may be acquired. The acquired projection characteristic information is sent to the projection image generation unit 306.

  The parallax image generation unit 305 generates a parallax image composed of a left-eye image and a right-eye image having parallax using the received viewpoint position information, display area information, and model data. The generated parallax image data is sent to the liquid crystal 3D display 102.

  The projection image generation unit 305 generates an image to be projected (here, a shadow image) using the received model data and projection characteristic information. The generated shadow image data is sent to the liquid crystal projector 103.

  FIG. 4 is a flowchart showing a flow of a series of processes in the image processing apparatus 101 according to the present embodiment. This series of processing is realized by reading a computer-executable program describing the following procedure from the ROM 203 or HDD 204 onto the RAM 202 and then executing the program by the CPU 201.

  In step 401, the acquisition units 301 to 304 acquire the above-described viewpoint position information, display area information, model data, and projection characteristic information from the HDD 204 and the like, respectively. Each acquired information is sent to the parallax image generation unit 305 and / or the projection image generation unit 306 as described above.

  In step 402, the parallax image generation unit 305 generates a parallax image using the received viewpoint position information, display area information, and model data. Specifically, a parallax image is generated by rendering a scene in a virtual space that appears over the stereoscopic image display area indicated by the display area information from the positions of the left and right eyes indicated by the viewpoint position information. In this embodiment, the position of the virtual camera used for rendering is set as the observer's viewpoint position. The optical axis direction of the virtual camera is assumed to coincide with a vector that passes through the observer's viewpoint position and is perpendicular to the stereoscopic image display area. 5A shows the relationship between the observer's viewpoint position in the real space and the stereoscopic display area (liquid crystal 3D projector 102), and FIG. 5B shows the observer's viewpoint position and the stereoscopic display area in the virtual space. Shows the relationship. Four vectors from the observer's viewpoint position (left eye: EL, right eye: ER) toward the four corners (r0 to r3) of the stereoscopic image display area in the virtual space where the virtual object 501 indicated by the model data is arranged Render the range enclosed by. In this way, by performing rendering processing using the positions of the right eye and left eye of the observer as viewpoint positions, a parallax image composed of a left-eye image and a right-eye image that have a parallax with each other is generated. 6A and 6B are diagrams illustrating an example of a parallax image in which a virtual object 501 is arranged, in which FIG. 6A illustrates an image for the left eye, and FIG. 6B illustrates an image for the right eye. The generated parallax image data is sent to the liquid crystal 3D display 102. By displaying the parallax image obtained as described above on the liquid crystal 3D display according to a predetermined 3D image display method, a stereoscopic image of the virtual object exists at a predetermined position indicated by the model data in the real space. Can be perceived by the observer.

  In step 403, the projection image generation unit 306 generates a shadow image using the mesh data and the projection characteristic information in the received model data. Hereinafter, an example in which a shadow image f having the same resolution (Wp × Hp: for example, 1920 × 1080) as the resolution of the liquid crystal projector 103 is generated as a 256 gray scale image will be described in detail. FIGS. 7A and 7B are diagrams for explaining the process of generating a shadow image. FIG. 7A shows a case when viewed from obliquely above, and FIG. 7B shows a case when viewed from the −x direction (x0 = x1).

  First, for the shadow image f of Wp × Hp pixels, all pixel values are initialized to 255. A straight line L extending from an arbitrary point Q on the shadow image plane 701 from the point P0 (x0, y0, z0) indicating the position of the liquid crystal projector 103 is a plate-like element (hereinafter referred to as a surface shape of the virtual object). , Simply referred to as “mesh”) 702. When it is determined that the straight line L intersects with the mesh 702, the pixel value of the pixel on the shadow image f corresponding to the point Q is set to 0 or the like, and the black pixel or the like indicating the shadow portion is set. On the other hand, when it is determined not to intersect, the pixel value on the shadow image f corresponding to the point Q is left at the initial value (ie, 255). If it is determined that they do not intersect, the pixel value on the shadow image f corresponding to the point Q is set to a predetermined value other than 0 (for example, the illuminance in the virtual space where the mesh exists or the real space where the shadow image is projected) It is good also as determined according to lighting conditions, such as. Here, the shadow image plane is a virtual plane that is orthogonal to the optical axis of the liquid crystal projector 103 (that is, the straight line P0-P1) at the center thereof. If the intersection of the optical axis of the liquid crystal projector 103 and the shadow image plane 701 is the origin, the horizontal direction of the liquid crystal projector is the horizontal axis s, and the vertical direction is the vertical axis t, the shadow corresponding to the pixel (u, v) of the shadow image f The coordinates (s, t) of the point Q on the image plane can be obtained by the following equations (1) and (2).

Here, D is the distance from the point P0 (x0, y0, z0) to the shadow image plane 701. Θ is an angle formed by line segments connecting the point P0 (x0, y0, z0) and the end points 701a / 701c of the shaded image plane on the horizontal axis s, and ψ is the point P0 (x0, y0, z0) and the horizontal This is the angle formed by the line segments connecting the end points 701b / 701d of the shadow image plane on the axis s.

  For all the pixels of the shaded image f, the corresponding points Q are obtained using the above formulas (1) and (2), intersection determination with the mesh 702 representing the surface shape of the virtual object is performed, and pixel values are set. A shadow image f is obtained. FIG. 8 is an example of the shadow image f obtained by the above-described processing, and a corresponding region 801 (region having a pixel value of 0 or the like) of the stereoscopic image exists as a shadowed portion. The shadow image data generated in this way is output to the liquid crystal projector 103 and projected as a shadow of a three-dimensional image on the floor of the real space.

  In this case, when a real object (for example, a part of the observer's body) exists between the liquid crystal projector 103 and the stereoscopic image, no shadow is generated for the mesh that becomes the shadow of the real object. It is desirable. For example, a distance sensor along the optical axis of the liquid crystal projector 103 is separately prepared to acquire the distance to the real object, and the mesh existing in front of the real object (position closer to the liquid crystal projector 103 than the real object) is obtained. Only the above-described intersection determination may be performed to generate a shadow image. Thereby, it is possible to prevent an unnecessary shadow from being generated on the real object.

  In this embodiment, the example in which the shadow generated by the virtual object is projected onto the real space has been described. However, for example, a pseudo-condensing pattern image generated by the virtual object is generated and projected onto the real space. Good. In this case, the projection image generation unit 306 uses the information indicating the optical properties representing the color and texture of the virtual object such as the reflectance, transmittance, and refractive index of the virtual object included in the model data to collect the light. A pattern image is generated. Specifically, in step 403 described above, a section where the straight line L passes through the virtual object is obtained, and the intensity of the transmitted light is derived based on the distance, the refractive index of the virtual object, and the like, and according to the obtained intensity. A condensing pattern image (corresponding to the shadow image f described above) in which the pixel value is replaced with the value may be obtained. Further, by generating a condensing pattern image as a color image and deriving the intensity of transmitted light for each wavelength of light to determine pixel values, a more elaborate condensing pattern can be obtained.

  As described above, according to the present embodiment, it is possible to cause a shadow or the like of a virtual object that appears to jump out of the screen in the real space.

[Example 2]
In the first embodiment, model data for specifying the position and shape of a virtual object displayed as a three-dimensional image is acquired, and a shadow for causing a shadow or the like according to the position of the three-dimensional image specified by the model data in the real space. Images etc. were generated. Next, an embodiment in which the position of the stereoscopic image is derived from the parallax image data and the viewpoint position information of the observer to generate a shadow image or the like will be described as a second embodiment. Note that description of parts common to the first embodiment is simplified or omitted, and here, differences will be mainly described.

  FIG. 9 is a functional block diagram of the image processing apparatus 101 according to the present embodiment. The system configuration of the stereoscopic image display system is the same as that of the first embodiment.

  The parallax image data acquisition unit 901 acquires parallax image data from a secondary storage device such as the HDD 204 via the HDD I / F 205. In this case, the parallax image data may be obtained by imaging with a multi-lens camera or the like, or may be generated using commercially available 3D image generation software. The acquired parallax image data is sent to the stereoscopic image position deriving unit 905 and the liquid crystal 3D display 102 as a stereoscopic image display device.

  The display area information acquisition unit 902 acquires display area information of the stereoscopic image display device. This is the same as the display area information acquisition unit 302 according to the first embodiment. The acquired display area information is sent to the stereoscopic image position deriving unit 905.

  The viewpoint position information acquisition unit 903 acquires the viewpoint position information of the observer. This is the same as the viewpoint position information acquisition unit 301 according to the first embodiment. The acquired viewpoint position information is sent to the stereoscopic image position deriving unit 905.

  The projection characteristic information acquisition unit 904 acquires projection characteristic information of the image projection apparatus. This is the same as the projection characteristic information acquisition unit 304 according to the first embodiment. The acquired projection characteristic information is sent to the projection image generation unit 906.

  The three-dimensional image position deriving unit 905 can specify the position and shape of a virtual object displayed as a three-dimensional image based on the received parallax image data, display area information, and viewpoint position information (here, the mesh data described above) Is generated.

  The projection image generation unit 906 generates a shadow image using the mesh data received from the stereoscopic image position deriving unit 905 and the projection characteristic information of the image projection apparatus received from the projection characteristic acquisition unit 904. The generated shadow image data is sent to the liquid crystal projector 103.

  FIG. 10 is a flowchart illustrating a flow of a series of processes in the image processing apparatus 101 according to the present embodiment.

  In step 1001, each acquisition unit 901 to 904 acquires the above-described parallax image data, display area information, viewpoint position information, and projection characteristic information from the HDD 204 or the like. The acquired parallax image data and each information are sent to the stereoscopic image position deriving unit 905 or the projection image generating unit 906 as described above.

  In step 1002, the stereoscopic image position deriving unit 905 derives the position of the virtual object displayed as a stereoscopic image based on the received parallax image data, display area information, and viewpoint position information. Here, it demonstrates in detail using another flowchart. FIG. 11 is a flowchart showing the flow of processing for deriving the position of this stereoscopic image. FIG. 12 is a diagram illustrating a process in which the position of a stereoscopic image is derived.

  In step 1101, the stereoscopic image position deriving unit 905 associates each pixel between the left eye image and the right eye image included in the parallax image data. Specifically, it is as follows. First, a region image composed of a pixel of interest (uL, vL) on the left eye image gL and its neighboring pixels is used as a template. Then, pattern matching using this template is performed on the right-eye image gR, and the corresponding pixel (uR, vR) on the right-eye image gR for the pixel of interest (uL, vL) is obtained. This process is performed for each pixel on the left-eye image gL, and all the pixels are associated with the pixels on the right-eye image gR.

  In step 1102, the stereoscopic image position deriving unit 905 displays the pixel pair (uL, vL) and (uR, vR) associated with the left-eye image and the right-eye image on the liquid crystal 3D display 102. In this case, the pixel position on the display screen is derived. Specifically, the three-dimensional coordinates PL (xL1, yL1, zL1) and PR (xR1, yR1, zR1) indicating the pixel position are obtained according to the following equations (3) and (4).

Here, W and H are horizontal and vertical sizes (cm) in the liquid crystal 3D display 102 indicated by the display area information, and Wg and Hg are the number of pixels in the horizontal and vertical directions in the parallax image.

  In step 1103, the stereoscopic image position deriving unit 905 uses the information on the pixel position on the screen of the stereoscopic display device obtained in step 1102 and the viewpoint position information of the observer acquired in step 1003 to generate a stereoscopic image. Find the position. Specifically, three-dimensional coordinates PL (xL1, yL1, zL1) / PR (xR1, yR1, zR1) indicating pixel positions on the screen of the stereoscopic display device and three-dimensional indicating positions of the left and right eyes of the observer From the coordinates EL (xL0, yL0, zL0) / ER (xR0, yR0, zR0), the straight line connecting EL and PL (straight line EL-PL) and the straight line connecting ER and PR (straight line ER-PR) Find the intersection P (x, y, z). The coordinates of this intersection P (x, y, z) correspond to the vertex coordinates indicating the position of the stereoscopic image formed by the pixel pair (uL, vL) and (uR, vR) associated between the parallax images. .

  In this case, when the straight line EL-PL and the straight line ER-PR are parallel to each other, it is considered that an image is formed at a position where the depth is infinite, and is not subject to the shadow image generation processing described later. To do. If the straight line EL-PL and the straight line ER-PR do not exist on the same plane, the point P'L on the straight line EL-PL and the point on the straight line ER-PR that minimize the distance between the two points Find P'R and consider the midpoint of these two points as the intersection of two straight lines.

  The above processing is performed for all pixel pairs to obtain vertex coordinate groups constituting a stereoscopic image.

  In step 1104, the stereoscopic image position deriving unit 905 applies a known surface shape restoration technique or tessellation technique to the obtained vertex coordinate group, estimates the surface shape of the stereoscopic image, and displays a virtual object displayed as a stereoscopic image. Generate mesh data that can identify the position of.

  In the present embodiment, a shadow image is generated using the mesh data obtained as described above as the position information of the stereoscopic image.

  Returning to the flowchart of FIG.

  In step 1003, the parallax image data acquisition unit 901 outputs the acquired parallax image data to the liquid crystal 3D display 102. Then, the liquid crystal 3D display 102 displays a stereoscopic image according to the parallax image data.

  In step 1004, the projection image generation unit 906 generates a shadow image using the projection characteristic information of the image projection apparatus acquired in step 1001 and the mesh data generated in step 1002. Details of the shadow image generation are as described in step 403 of the flowchart of FIG. 4 according to the first embodiment.

  The shadow image data generated in this way is output to the liquid crystal projector 103 as in the first embodiment, and the shadow of the virtual object according to the shadow image data is projected onto the floor or the like in the real space.

(Other embodiments)
The object of the present invention can also be achieved by executing the following processing. That is, a storage medium that records a program code of software that realizes the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU, MPU, etc.) of the system or apparatus is stored in the storage medium. This is the process of reading the code. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code and the storage medium storing the program code constitute the present invention.

Claims (12)

  When making a viewer perceive a stereoscopic image of a virtual object by displaying a parallax image, the shadow image for creating a pseudo shadow of the stereoscopic image in the real space is specified, and the position and shape of the virtual object are specified. An image processing apparatus characterized in that the image processing apparatus generates data based on data indicating the position and size of a projection area by image projection means for projecting the shadow image onto a real space. The data specifying the position and shape of the virtual object is mesh data,
A plate in which a straight line connecting a point indicating the position of the image projecting unit and an arbitrary point on a virtual plane orthogonal to the optical axis of the image projecting unit indicates the surface shape of the virtual object specified by the mesh data It is determined whether or not to intersect with an element of a shape, and when it is determined to intersect, the shadow image is generated by setting a pixel corresponding to the intersecting point as a pixel indicating a shadow portion. The image processing apparatus according to claim 1.   If there is a real object between the image projection unit and the stereoscopic image, whether or not to intersect only with the plate-like element existing at a position closer to the image projection unit than the real object The image processing apparatus according to claim 2, wherein the determination is performed.   The parallax image is generated based on information indicating the viewpoint position of the observer, data specifying the position and shape of the virtual object, and information indicating the size of the display area in the stereoscopic image display means for displaying the parallax image. The image processing apparatus according to claim 1, further comprising means. Means for acquiring data of the parallax image;
Means for generating the mesh data based on the acquired parallax image data, information indicating the size of the display area in the stereoscopic image display means for displaying the parallax image, and information indicating the viewpoint position of the observer;
The image processing apparatus according to claim 2, further comprising: The parallax image is composed of an image for the left eye and an image for the right eye,
The means for generating the mesh data associates the image for the left eye with the image for the right eye, derives a pixel position on the display screen of the stereoscopic image display means for the associated pixel pair, and derives By obtaining a vertex coordinate group constituting the stereoscopic image from the information indicating the pixel position and the observer's viewpoint position, and estimating the surface shape of the stereoscopic image using the obtained vertex coordinate group, The image processing apparatus according to claim 5, wherein mesh data is generated.   When making a viewer perceive a stereoscopic image of a virtual object by displaying a parallax image, a condensing pattern image for creating a pseudo condensing pattern generated by the stereoscopic image in the real space is displayed on the virtual object. An image generated based on data specifying a position, a shape, and optical characteristics, and information indicating a position and a size of a projection area by an image projecting unit that projects the condensed pattern image onto a real space. Processing equipment.   When making a viewer perceive a stereoscopic image of a virtual object by displaying a parallax image, the shadow image for creating a pseudo shadow of the stereoscopic image in the real space is specified, and the position and shape of the virtual object are specified. An image processing method comprising: generating data based on data to be displayed and information indicating a position and a size of a projection area by an image projection unit that projects the shadow image onto a real space.   When making a viewer perceive a stereoscopic image of a virtual object by displaying a parallax image, a condensing pattern image for creating a pseudo condensing pattern generated by the stereoscopic image in the real space is displayed on the virtual object. An image generated based on data specifying a position, a shape, and optical characteristics, and information indicating a position and a size of a projection area by an image projecting unit that projects the condensed pattern image onto a real space. Processing method.   The program for functioning a computer as an image processing apparatus of any one of Claims 1-7. The image processing apparatus according to any one of claims 1 to 6,
Stereoscopic image display means for displaying the parallax image;
Image projection means for projecting the shadow image onto a real space;
A stereoscopic image display system comprising: An image processing apparatus according to claim 7;
Stereoscopic image display means for displaying the parallax image;
Image projecting means for projecting the focused pattern image on a real space;
A stereoscopic image display system comprising:

Images