JP2013003848A - Virtual object display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To create a virtual image in which a virtual object is arranged at an optional position in an image of a real space.SOLUTION: A virtual object display device comprises: an imaging part for outputting image data by imaging an object existing in a real space; a distance sensor part for outputting depth map information indicating a distance to the object; a light information estimation part for calculating the position of a light source in the real space, a color and an intensity of light emitted from the light source, and information indicating the light reaching a virtual object on the basis of the image data and depth map information; a face position detection part for detecting a position of the face of a viewer on the basis of the depth map information; an image creation part for creating an image acquired by setting the viewer's face as an eye point in a virtual space generated by arranging the virtual object at a predetermined position in the real space on the basis of the image data, information indicating the virtual object, the position, color and intensity of the light source, and the information indicating the light reaching the virtual object; and a display part for displaying the image created by the image creation part.

Description

  The present invention relates to a virtual object display device.

  There have been researches on techniques for expressing virtual objects as if they existed in real space. A typical one is one that reflects binocular parallax or motion parallax parallax information, and a method of displaying a virtual object in three dimensions has been studied (for example, Non-Patent Document 1).

  Further, a technique for displaying a virtual object superimposed on a real space has been studied in the field of augmented reality. There is a method of measuring information of light reaching a certain point using an omnidirectional camera or the like, and adding light and darkness or shadow to a virtual object based on the information. As a result, the virtual object can be displayed in the real space image as if it was present and the light from the real light source was applied. A method in which an observer observes a real space and a superimposed image of a virtual object through a head mounted display, a portable display, or the like has been studied (for example, Non-Patent Document 2).

Y. Kunita, T. Kikukawa, et al, Image-based 3D Telecopier: A System for Sharing a 3D Object by Multiple Groups of People at Remote Locations, Proceedings of the 17th ACM Symposium on Virtual Reality Software and Technology, pp.51- 54, 2010. M. Knecht, C. Traxler, et al, Differential Instant Radiosity for Mixed Reality, IEEE International Symposium on Mixed and Augmented Reality 2010 Science and Technology Proceedings, pp.99-107, 2010.

  However, in the above method, a device that detects a light source such as a camera is arranged at a position where a virtual object is displayed, and reflects the color of the light source, the position (direction) of the light source, and an object existing in the real space. Therefore, it is necessary to detect in advance object reflected light that is light that should reach the virtual object. Therefore, there is a problem that a virtual object cannot be freely displayed at an arbitrary position in an image obtained by imaging the real space.

  The present invention has been made to solve the above problem, and an object of the present invention is to provide a virtual object display device that can generate a virtual image in which a virtual object is arranged at an arbitrary position with respect to an image obtained by capturing a real space. It is to provide.

  In order to solve the above-described problem, the present invention stores in advance virtual object information indicating the shape of a virtual object that is virtually placed at a predetermined position in the real space and the reflection characteristics of the surface. A virtual object information storage unit, an imaging unit that captures an image of a real object existing in real space and outputs image data, and a distance sensor that measures the distance to the real object and outputs depth map information indicating the measured distance The light source position in the real space, the color and intensity of the light emitted from the light source, and the light that is reflected by the real object and reaches the virtual object, based on the image data and the depth map information. An optical information estimation unit that calculates information indicating a certain object reflected light, a face position detection unit that detects the position of the face of the observer observing the virtual object based on the depth map information, and the real position in the real space Virtual object In a virtual space arranged at a predetermined position, an image obtained with the face position detected by the face position detection unit as a viewpoint, the image data, the virtual object information, the position of the light source, A virtual object comprising: an image generation unit that is generated based on color and intensity, and information indicating light that reaches the virtual object; and a display unit that displays an image generated by the image generation unit It is a display device.

Further, the present invention is characterized in that, in the above-described invention, the optical information estimation unit sets a position corresponding to a pixel having the highest luminance in the image data as a position of a light source.
Further, according to the present invention, in the invention described above, the optical information estimation unit detects a plurality of specularly reflected portions of the captured real object based on the image data, and detects the detected specular reflection. With respect to the normal line of the specularly reflected part for each part, a straight line from the position of the imaging unit to the specularly reflected part and a straight line having a mirror image relation are calculated, and the intersection of each calculated straight line Is the position of the light source.
In the invention described above, the optical information estimation unit may determine whether or not there is a pixel having a luminance equal to or higher than a predetermined luminance threshold in the image data. If there is a pixel, the position corresponding to the pixel with the highest luminance in the image data is the position of the light source, and if there is no pixel equal to or higher than the luminance threshold, the real object imaged based on the image data Detecting a plurality of specular reflection parts, and for each detected specular reflection part, with respect to the normal of the specular reflection part, a straight line from the position of the imaging unit toward the specular reflection part and A straight line having a mirror image relationship is calculated, and the intersection of the calculated straight lines is set as the position of the light source.

  According to the present invention, a virtual image in which a virtual object is arranged in the real space is generated based on the position of the light source in the real space, the color and intensity of the light emitted from the light source, and the object reflected light. The effect of the object reflected light, the shadow on the light source, and the reflection on the surrounding object are reflected, and the image quality can be improved.

It is a figure which shows an example of the external appearance of the virtual object display apparatus 1 in this embodiment. It is a schematic block diagram which shows the structure of the virtual object display apparatus 1 in the embodiment. It is a schematic block diagram which shows the structure of the optical information estimation part 13 in the embodiment. It is the schematic which shows the method of estimating the position of the light source in the embodiment.

First, an outline of the present invention will be described.
For example, when discussing a remotely existing object or a CG model of a product prototype, it is important to display a virtual object as if it existed in the same way as an actually existing object. Until now, in order to perform such display, it is considered important to display a virtual object in three dimensions, and efforts have been made to incorporate binocular parallax and motion parallax. The inventor reflects not only such binocular parallax and motion parallax but also the influence of light reflected from the light source and the object in the real world so that the virtual object is present in the real world. It came to the knowledge that it was important.
For example, when there is a mug in a room, the mug becomes visible by receiving light from a light source (such as a fluorescent lamp) in the room. At that time, the shadow of the mug is made according to the positional relationship between the light source and the mug, and when the surface of the mug is glossy, an observer or a surrounding object may be reflected in the mug. Such a phenomenon is a phenomenon that can be observed when an object exists in real space.

The present invention represents a virtual object in consideration of the phenomenon caused by the light from the light source and the light reflected from the surrounding object (hereinafter referred to as object reflected light). According to the present invention, a virtual object can be shaded by light hitting a virtual object from an actual light source, an object that actually exists casts a shadow on the virtual object, an observer or an object in the real space is reflected in the virtual object, etc. When a virtual object exists in the real world, a phenomenon that should occur can be added to the image. As a result, an image similar to the case where a virtual object exists in the real space can be created.
The image displayed according to the present invention is a reproduction of an image that appears when a virtual object actually exists on the other side across a transparent plate such as a window glass. Assuming that the display surface is a transparent plate, the light from the light source and the light that passes through the transparent plate, such as object reflection light, are estimated and applied to the virtual object. That occur when an object is present (a light source hits a virtual object to make a shadow, a real object casts a shadow on a virtual object, an observer or an object in real space is a virtual object) Can be added to the image.

  Hereinafter, a virtual object display device according to an embodiment of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a diagram illustrating an example of an appearance of a virtual object display device 1 according to the present embodiment. As shown in the figure, the virtual object display device 1 includes a display as a display unit, a camera as an imaging unit, and a distance sensor as a distance sensor unit.

FIG. 2 is a schematic block diagram showing the configuration of the virtual object display device 1 in the present embodiment. As shown in the figure, the virtual object display device 1 includes an imaging unit 11, a distance sensor unit 12, a display unit 17, and a calculation unit 20. The calculation unit 20 includes an optical information estimation unit 13, a face position detection unit 14, a virtual object information storage unit 15, and an image generation unit 16. Note that the calculation unit 20 includes, for example, a calculation device, and uses software stored in a storage area of the calculation device, so that the optical information estimation unit 13, the face position detection unit 14, and the virtual object information storage unit 15 and the image generation unit 16 may be realized. In the present embodiment, it is assumed that the imaging unit 11 and the distance sensor unit 12 are previously aligned.
In the virtual object display device 1, the light information estimation unit 13 estimates information on the light source and the object reflected light using image data obtained by the imaging unit 11 and information indicating the depth measured by the distance sensor unit 12. Further, the three-dimensional object is rendered based on the position of the face of the observer calculated by the face position detection unit 14, and a final output image is generated and displayed on the display unit 17.

The imaging unit 11 images a real space and outputs image data indicating the captured image. The imaging unit 11 has an optical system with a wide angle of view including an omnidirectional lens optical system and an omnidirectional mirror optical system, and images an object (subject) existing in a real space around the imaging unit 11. The image data output by the imaging unit 11 is information indicating the imaged object using the lightness of each of the three primary colors of light (red, green, and blue).
The distance sensor unit 12 measures the depth of the real space and outputs depth map information indicating the measured depth. In other words, the distance to the object existing in the real space imaged by the imaging unit 11 is measured and the depth map information is output. As the distance sensor unit 12, for example, an infrared pattern projection type distance sensor may be used as in the case of KINET (registered trademark) sold by Microsoft (registered trademark). Further, the distance sensor unit 12 may be configured using a plurality of sensors so that the distance to an object within the imaging range of the imaging unit 11 can be measured.

The optical information estimation unit 13 estimates the three-dimensional coordinates of the light source, the light source color, and the light intensity in the image data based on the image data output from the imaging unit 11 and the depth map information output from the distance sensor unit 12. I do. Further, the light information estimation unit 13 obtains, as estimation results, three-dimensional coordinates (x, y, z) indicating the position of the light source in the real space, color information (R, G, B) indicating the light source color, and light intensity. The intensity information (S) indicating is output. As the coordinate system in the virtual object display device 1, for example, a coordinate system based on one of the imaging unit 11 and the distance sensor unit 12 may be used.
In addition, when a virtual object is arranged in the real space, the optical information estimation unit 13 outputs three-dimensional object data for calculating the reflection of the object existing in the real space on the surface of the virtual object. This three-dimensional object data is information indicating the position and shape of an object existing in real space, calculated based on image data and depth map information. For example, a basic figure (such as a polyhedron indicating the shape of an object). Polygon) and polygon information indicating its size, information indicating the three-dimensional coordinates of the object, and the like.

  FIG. 3 is a schematic block diagram illustrating the configuration of the optical information estimation unit 13 in the present embodiment. As shown in the figure, the light information estimation unit 13 includes a light source position estimation unit 131, a light source color estimation unit 132, a light source intensity estimation unit 133, and an object reflected light estimation unit 134. In the present embodiment, processing is performed assuming that the light source is within the imaging range of the imaging unit 11.

The light source position estimation unit 131 determines that there is a light source at a position corresponding to the pixel with the highest luminance in the image data output from the imaging unit 11. Further, the light source position estimation unit 131 calculates the three-dimensional coordinates of the position corresponding to the pixel with the highest luminance based on the depth map information output from the distance sensor unit 12.
Since the light source itself is reflected, the light source color estimation unit 132 calculates the light source color emitted from the light source from the position determined as the light source by the light source position estimation unit 131 and the image data. Specifically, the light source color estimation unit 132 shortens the exposure time sufficiently so that saturation by the light source does not occur, and determines the light source according to the color information (R, G, B) values in the reflected light source pixels. Calculate the color. Here, the exposure time at which saturation does not occur is, for example, the exposure time of the imaging unit 11 in which image data for which there is no maximum value that can be taken in the lightness values of red, green, and blue in the color information is obtained. is there.

  Since the light source itself is reflected, the light source intensity estimation unit 133 calculates the intensity of the light source from the position determined as the light source by the light source position estimation unit 131 and the image data. Specifically, the light source intensity estimating unit 133 calculates the intensity of the light source from the light source color calculated by the light source color estimating unit 132, the exposure time, and the luminance value of the reflected light source pixel.

  The object reflected light estimation unit 134 calculates three-dimensional object data used when calculating the reflection of an object existing in the real space with respect to a virtual object arranged at a predetermined position in the real space. The object reflected light estimation unit 134 calculates the position and shape of each object captured in the image data from the image data and the depth map information, and outputs the calculation result as three-dimensional object data. That is, the object reflected light estimation unit 134 models an object in real space as a three-dimensional object, and outputs information obtained by the modeling as three-dimensional object data.

Returning to FIG. 2, the description of each part constituting the virtual object display device 1 will be continued.
The face position detection unit 14 detects the position of the observer's face from the depth map information output by the distance sensor unit 12. For example, the face position detection unit 14 determines the position of the observer's head based on the skeleton information of the observer's head, torso, arms, and legs obtained using the Open NI skeleton processing function (or skeleton tracking). By specifying (three-dimensional coordinates), the position of the face is calculated. The face position detection unit 14 outputs face position information that is three-dimensional coordinates (ex, ey, ex) indicating the face position. Here, the face position is a coordinate representing a viewpoint when a virtual object is arranged in the real space. Therefore, the face position detection unit 14 may calculate the eye position instead of the face position.

  The virtual object information storage unit 15 stores in advance virtual object information used when representing a virtual object on an image. The virtual object information includes polygon information indicating the shape of the virtual object, texture information indicating an image expressing a pattern and a texture on the surface of the virtual object, and information indicating light reflection characteristics on the surface of the virtual object. .

The image generation unit 16 is stored in the virtual object information storage unit 15 and the three-dimensional object data of the object existing in the real space and the position of the light source, the light source color, and the intensity output from the light information estimation unit 13. Based on the virtual object information, a CG space (virtual three-dimensional space) in which the virtual object is arranged at a predetermined position in the real space is generated. At this time, the image generation unit 16 performs light information estimation on a light source that emits light indicated by the color information (R, G, B) and intensity information (S) output from the light information estimation unit 13 in the generated CG space. The virtual camera is arranged based on the face position information output from the face position detection unit 14 and is rendered at the three-dimensional coordinates (x, y, z) output from the unit 13, and the image captured by the camera is rendered. (Ray tracing, etc.) The virtual camera is arranged at three-dimensional coordinates (ex, ey, ez) included in the face position information. The image generation unit 16 updates the texture information included in the virtual object information based on the virtual camera position, the virtual object information, and the three-dimensional object data. Thereby, the object reflected on the surface of the virtual object is expressed.
The display unit 17 has a display screen such as a liquid crystal panel, for example, and displays the image generated by the image generation unit 16.

By having the above-described configuration, the virtual object display device 1 estimates light source and object reflected light information in the real space, performs rendering using the estimated information, and generates an image in the CG space including the virtual object. To do. At this time, the optical information estimation unit 13 uses depth map information indicating the depth measured by the distance sensor unit 12 and image data obtained by imaging using an optical system in which the imaging unit 11 has a wide angle of view. In addition to estimating the position, color, and intensity of the light source, the object reflected light is estimated. Further, the face position detection unit 14 estimates the observer's viewpoint based on the image data output by the imaging unit 11 and the depth map information indicating the depth measured by the distance sensor unit 12.
Thereby, the virtual object display device 1 expresses the light source color, the position (direction) of the light source, the shadow on the effect light source due to the object reflected light, and the reflection on the surrounding object without obtaining information on the light source in advance. The generated image can be generated, and the quality of the virtual image showing the state where the virtual object exists in the real space can be improved. As a result, it is possible to make the observer who sees the image displayed by the virtual object display device 1 feel that the virtual object exists in the real space.

(Second Embodiment)
The virtual object display device according to the second embodiment displays a virtual image indicating a situation in which a virtual object exists in real space when no light source exists within the imaging range of the imaging unit 11. is there. Since the virtual object display device in the present embodiment has the same functional blocks as the virtual object display device 1 in the first embodiment, description of each functional block will be omitted, and processing different from that in the first embodiment will be described.

In the virtual object display device according to the present embodiment, the process of estimating the position of the light source in the optical information estimation unit 13 is different from that of the first embodiment as follows. In the present embodiment, as in the first embodiment, it is assumed that the imaging unit 11 and the distance sensor unit 12 are aligned in advance.
The light source position estimation unit 131 detects the position and shape of an object existing in the real space based on the image data output from the imaging unit 11 and the depth map information output from the distance sensor unit 12. Further, the light source position estimation unit 131 detects a portion (a portion that performs specular reflection) that strongly reflects light from the light source in an object that exists in the real space, based on the luminance of the image data. The light source position estimation unit 131 calculates the normal of the object plane in a portion where specular reflection is performed on the object from the position and shape of the object existing in the real space. In addition, the light source position estimation unit 131 has a reflection relationship with a vector directed from the position of the imaging unit 11 toward a portion where specular reflection is performed with respect to the calculated normal of the object plane. A vector is calculated, and a point at which the calculated mirror vectors intersect is set as the position of the light source.

FIG. 4 is a schematic diagram showing a method for estimating the position of the light source in the present embodiment. As shown in the figure, the light source position estimation unit 131 detects a specular reflection surface in each of the object A and the object B. The light source position estimation unit 131 calculates the normal of the specular reflection surface of the object A and the normal of the specular reflection surface of the object B. In addition, the light source position estimation unit 131 relates to the normal of the object surface specularly reflected in the object A, a straight line from the position of the imaging unit 11 (camera) toward the part that is specularly reflected in the object A, A straight line having the relationship is calculated. Similarly, the light source position estimating unit 131 has a mirroring relationship with a straight line from the position of the imaging unit 11 toward the specularly reflected portion of the object B with respect to the normal of the object surface that is specularly reflected at the object B. A straight line is calculated. Then, the light source position estimation unit 131 sets the intersection of two straight lines having the calculated reflection relationship as the light source position. That is, the light source position estimation unit 131 calculates the light source position on the assumption that the light beam reflected at the same angle as the incident angle with respect to the specular reflection surfaces of the objects A and B has reached the imaging unit 11.
At this time, the light source position estimation unit 131 may detect two points with high luminance in the image data and detect the specular reflection surfaces in the objects A and B, respectively.

For example, the light source color estimation unit 132 is described in Reference 1 (GJ Klinker, SA Shafer, T. Kanade, Using a Color Reflection Model to Separate Highlights from Object Color, Proceedings of the First International Conference on Computer Vision, pp.145-150. , 1987.) and a known technique described in Reference 2 (Japanese Patent Laid-Open No. 8-046990) and the like, the light source color is estimated. In Reference Document 1, when acquiring light source information, when acquiring a light source position, a light source color, and a light source intensity, the image is divided into a highlight part and a non-glossy part, and the light source color is determined from the color of the highlight part. Techniques for estimating are described. Further, Reference 2 describes a technique for estimating a light source color based on a gray hypothesis (a hypothesis that if all the colors of objects existing in a scene are averaged, the scene becomes gray in light source conversion).
The light source intensity estimation unit 133 considers that the brightest pixel in the image data reflects the intensity of light emitted from the light source, and calculates the intensity of the light source based on the luminance value of the brightest pixel.

The virtual object display device having the above-described configuration estimates the position of the light source even when there is no light source within the imaging range of the imaging unit 11, and determines the color of the light source, the position (direction) of the light source, and object reflection. It is possible to generate an image that expresses shadows on the effect light source by light and reflections on surrounding objects, and improves the quality of the virtual image that shows the state in which the virtual object exists in the real space . Thereby, it is possible to make the observer who sees the image displayed by the virtual object display device 1 feel that the virtual object exists in the real space.
As a result, in order to detect the position of the light source, the shadow and gloss of the virtual object can be reduced without using a device (for example, a fisheye lens) that measures the light around the entire place where the virtual object is placed. It can be reflected in the image according to the place to be arranged, and the quality of the virtual image showing the state where the virtual object exists in the real space can be improved.
Further, in the virtual object display device according to the present embodiment, the position of the light source can be calculated even when the light source is not included in the image data output from the imaging unit 11, and thus the optical included in the imaging unit 11. The system may not be an optical system with a wide angle of view such as an omnidirectional lens optical system.

The light source position and the like may be estimated by combining the light source estimation in the first embodiment and the light source estimation in the second embodiment. Specifically, the imaging unit 11 outputs image data captured at an exposure time that does not cause saturation in the image data to the optical information estimation unit 13. The light source position estimation unit 131 determines whether or not there is a pixel having a luminance equal to or higher than a predetermined luminance threshold in the image data. The optical information estimation unit 13 determines that there is a light source in the image captured by the imaging unit 11 when there is a pixel having a luminance equal to or higher than the luminance threshold, and estimates the light source in the first embodiment. On the other hand, when there is no pixel having a luminance equal to or higher than the luminance threshold, the optical information estimation unit 13 determines that there is no light source in the image captured by the imaging unit 11, and performs estimation of the light source in the second embodiment. Here, the luminance threshold is a luminance value that is predetermined by performing statistical processing or the like on a plurality of image data.
Thereby, the virtual object display device 1 can display an image corresponding to the position of the light source in the real space, and further improve the quality of the virtual image indicating the state in which the virtual object exists in the real space. be able to.

In the first and second embodiments, a binocular parallax type three-dimensional display may be used as the display unit 17. By using a binocular parallax type three-dimensional display and displaying the image generated by the image generation unit 16 with parallax, the virtual object is three-dimensionally present in front of or behind the display screen of the display unit 17. Can be displayed. When a virtual object is displayed behind the display screen of the display unit 17, the same processing as in the above embodiment is performed. When displaying a virtual object with a parallax floating above the display screen, the light information estimation unit 13 estimates the light from the light source and the object reflected light at the position where the virtual object floats. The image generation unit 16 generates a virtual image indicating a state in which a virtual object exists in the real space. Thereby, an image similar to the case where the virtual object actually exists in front of the display screen can be created.
Accordingly, when viewing is intended, viewing can be performed by displaying the object using the virtual object display device 1 without owning an object (such as a sculpture or an image). In addition, it is possible to apply it to observing and enjoying things that can only be seen in a museum in a remote place (remote museum).

Moreover, although the virtual object display apparatus in said 1st and 2nd embodiment demonstrated the structure which detects the position of an observer's face based on the depth map information which the distance sensor part 12 outputs, it is not restricted to this. Also good. For example, when the virtual object display device is configured using a portable information terminal (for example, a mobile phone) having an acceleration sensor, the position of the observer's face is detected as follows. Also good.
The acceleration sensor outputs information indicating how much the portable information terminal is inclined with respect to the horizontal. The distance between the display unit 17 (display) of the portable information terminal and the face of the observer is assumed to be constant, and the face position detection unit 14 determines the observer's face based on the inclination of the portable information terminal detected by the acceleration sensor. The position of the face may be calculated. Here, an average distance between the display unit 17 and the face when the observer uses the portable information terminal may be calculated in advance, and the distance may be stored in the face position detection unit 14.

  In the virtual object display devices of the first and second embodiments described above, the imaging unit 11, the distance sensor unit 12, and the display unit 17 may not be housed in one housing. For example, the imaging unit 11 and the distance sensor unit 12, the display unit 17, and the calculation unit 20 may be different devices.

  Further, the virtual object display device described above may have a computer system inside. In that case, the processes performed by the face position detection unit 14, the optical information estimation unit 13, the virtual object information storage unit 15, and the image generation unit 16 described above are stored in a computer-readable recording medium in the form of a program. The above processing is performed by the computer reading and executing this program. Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

  DESCRIPTION OF SYMBOLS 1 ... Virtual object display apparatus, 11 ... Imaging part, 12 ... Distance sensor part, 13 ... Optical information estimation part, 14 ... Face position detection part, 15 ... Virtual object information storage part, 16 ... Image generation part, 17 ... Display part , 20 ... calculating part, 131 ... light source position estimating part, 132 ... light source color estimating part, 133 ... light source intensity estimating part, 134 ... object reflected light estimating part

Claims (4)

A virtual object information storage unit that stores in advance virtual object information indicating the shape of a virtual object, which is an object that is virtually placed at a predetermined position in the real space, and the reflection characteristics of the surface;
An imaging unit that captures an image of a real object existing in the real space and outputs image data;
A distance sensor unit that measures the distance to the real object and outputs depth map information indicating the measured distance;
Based on the image data and the depth map information, the position of the light source in the real space, the color and intensity of the light emitted from the light source, and the object reflection that is the light that is reflected by the real object and reaches the virtual object An optical information estimation unit for calculating information indicating light;
A face position detection unit that detects the position of the face of the observer observing the virtual object based on the depth map information;
In a virtual space in which the virtual object is arranged in a predetermined position in the real space, an image obtained from the viewpoint of the face position detected by the face position detection unit is represented by the image data, the virtual object information, An image generation unit that generates the light source based on the position, color, and intensity of the light source and information indicating light reaching the virtual object;
A virtual object display device comprising: a display unit configured to display an image generated by the image generation unit. The optical information estimation unit includes:
The virtual object display device according to claim 1, wherein a position corresponding to a pixel having the highest luminance in the image data is a light source position. The optical information estimation unit includes:
Based on the image data, a plurality of specular reflection portions of the captured real object are detected, and for each detected specular reflection portion, the imaging unit is related to the normal of the specular reflection portion. 2. The virtual object display according to claim 1, wherein a straight line that is in a mirror image relation with a straight line that goes from the position to the specularly reflected portion is calculated, and an intersection of the calculated straight lines is set as a light source position. apparatus. The optical information estimation unit includes:
It is determined whether or not there is a pixel having a luminance equal to or higher than a predetermined luminance threshold in the image data, and when there is a pixel equal to or higher than the luminance threshold, a position corresponding to the highest luminance pixel in the image data Is a position of the light source, and when there is no pixel equal to or greater than the luminance threshold, based on the image data, a plurality of specular reflection portions of the captured real object are detected, and the detected specular reflection portion Each time, with respect to the normal of the specularly reflected portion, a straight line from the position of the imaging unit to the specularly reflected portion and a straight line having a mirror image relation are calculated, and the intersection of the calculated straight lines is calculated as the intersection of the light source. The virtual object display device according to any one of claims 1 to 3, wherein the virtual object display device is a position.

Images