JP7088721B2 - Information processing system and program - Google Patents

Description

The present invention relates to an information processing system and a program.

Currently, it means a combination of a real space (real space) and a space virtually created using a computer (virtual space), not virtual reality (Virtual Reality or VR) or augmented reality (AR). A technique called mixed reality (MR) is attracting attention. In the space where the complex reality is realized (composite reality space), the object in the real space and the object in the virtual space superimpose the shape information of the two three-dimensional spaces of the real space and the virtual space, and influence in real time. A matching experience is possible.
For example, in Patent Document 1, when a real object is located behind a virtual object (when the real object cannot be seen by the user), the user is informed in advance of the existence of the real object approaching. The technique is described. Specifically, when the distance between the real object and the user is within a predetermined distance, the display of the virtual object located on the front side is controlled to be semi-transparent or contour line display, and the reality is located behind. A technique that enables the visual recognition of an object is described.

Japanese Unexamined Patent Publication No. 2016-4493

On the other hand, in the conventional technology, when there is a virtual object (virtual object) within the range where the image of reflection is reflected on the real object due to the positional relationship between the real object (real object) and the user, the real object Since only the shape information is used as the criterion for drawing, even if the real object has reflectivity, a method of not drawing the image of the virtual object on the real object is adopted regardless of the reflectivity of the real object. ing. Such a phenomenon that violates the laws of nature is unnatural for users who are experiencing mixed reality, as if the legendary vampire could see the real image but the mirror image would not be reflected in the mirror and it would be perceived as a vampire. It gives an impression.

The present invention enables an experience as if a virtual object actually exists by drawing an image of a virtual object on a real object having reflectivity even in a situation where the reflectivity of the real object is not known in advance. The purpose is to.

The invention according to claim 1 is an estimation means for estimating information on the reflectivity of a real object from an image obtained by capturing an image of the real object, and a position between the real object estimated to have reflectivity and a user. When there is a virtual object within the range where the image of reflection is reflected on the real object due to the relationship, the generation means for generating the image of the virtual object based on the estimated reflectivity information and the above-mentioned It has a drawing means for drawing an image of the virtual object with respect to a real object, and the estimation means has a relationship between the position of a real light source and the inclination of the surface of the real object defined as an estimation target. Therefore, when the light from the real light source is reflected by the real object defined as the estimation target, it is the position where the reflected reflected light irradiates, and at the position where the light from the real light source directly irradiates. It is an information processing system characterized in that when a bright part exists at a position specified as being, it is estimated that the actual object defined as an estimation target has reflectivity .
According to the second aspect of the present invention, the estimation means estimates that the larger the difference between the brightness of the specified position and the brightness around the specified position, the greater the reflectance of the actual object. The information processing system according to claim 1, wherein the information processing system is characterized by the above .
The invention according to claim 3 comprises a computer, an estimation means for estimating information on the reflectivity of a real object from an image obtained by capturing an image of the real object, and the real object presumed to have reflectivity. Generation that generates an image of the virtual object based on the estimated reflectivity information when there is a virtual object within the range where the image of reflection is reflected on the real object due to the positional relationship with the user. The means and the estimation means are made to function as a drawing means for drawing an image of the virtual object on the real object, and the estimation means is the position of the actual light source and the inclination of the surface of the real object defined as the estimation target. In relation to the above, when the light from the real light source is reflected by the real object defined as the estimation target, it is the position where the reflected reflected light irradiates, and the light from the real light source directly irradiates. It is a program characterized in that when a bright part exists at a position specified as a position to be estimated, the actual object defined as an estimation target is estimated to have reflectivity .

According to the invention of claim 1, even in a situation where the reflectivity of a real object is not known in advance, is the virtual object actually existing by drawing an image of a virtual object with respect to the real object having reflectivity? You can enable an experience like this.
According to the invention of claim 2, even in a situation where the reflectivity of a real object is not known in advance, is the virtual object actually existing by drawing an image of a virtual object with respect to the real object having reflectivity? You can enable an experience like this.
According to the invention of claim 3, even in a situation where the reflectivity of the real object is not known in advance, is the virtual object actually existing by drawing an image of the virtual object with respect to the real object having the reflectivity? You can enable an experience like this .

It is a figure explaining the principle that a user wearing a glasses-type terminal which can see through the outside world can experience mixed reality. It is a figure which shows an example of the hardware composition of the glasses type terminal. It is a figure which shows an example of the functional structure of a glasses-type terminal. It is a flowchart explaining an example of the processing operation executed when drawing the image of a virtual object with a glasses-type terminal. It is a figure explaining an example of the method used for the estimation of the reflectivity. (A) shows the positional relationship of the real object, and (B) shows the positional relationship of the real object and the eyeball. It is a figure explaining another example of the method used for the estimation of the reflectivity. It is a figure explaining another example of the method used for the estimation of the reflectivity. It is a figure explaining the reflection area which the image of the virtual object is reflected in the real object. (A) shows the relationship between the positions of the real object and the virtual object, and (B) shows the reflection area of the real object in which the image of the virtual object is reflected. It is a figure explaining the difference between the drawing by the conventional technique and the drawing by this embodiment. (A) is a drawing example by the conventional technique, and (B) is a drawing example by the present embodiment. It is a figure explaining the influence which the difference in the reflectance of a real object has on the drawing of the image of a virtual object. (A) is an example of drawing an image of a virtual object when the reflectance of a real object is high, and (B) is an example of drawing an image of a virtual object when the reflectance of a real object is low. It is a figure explaining the influence which the difference in color of a real object has on the drawing of the image of a virtual object. (A) is an example of drawing an image of a virtual object when the real object is colored in light blue, and (B) is an example of drawing an image of a virtual object when the real object is colored in light red. be. It is a figure explaining the influence which the difference of the pattern attached to a real object has on the drawing of the image of a virtual object. (A) is an example of drawing an image of a virtual object when a diagonal line extending diagonally is formed on the surface of the real object, and (B) is a case where a mesh pattern is formed on the surface of the real object. It is a drawing example of the image of the virtual object of. It is a figure explaining the influence which the difference in the shape of a real object has on the drawing of the image of a virtual object. (A) is an example of drawing an image of a virtual object in a flat plate-shaped real object, and (B) is an example of drawing an image of a virtual object in a cylindrical real object. It is a figure explaining the influence which the difference of the angle on which a real object is placed has on the drawing of the image of a virtual object. (A) is an example of drawing an image of a virtual object when a flat plate-shaped real object is placed parallel to a vertical plane, and (B) is an example of drawing a flat plate-shaped real object diagonally to a vertical plane. This is an example of drawing an image of a virtual object when it is placed. It is a figure explaining the influence which the difference of the information about the polarization of the reflected light in a real object has on the drawing of the image of a virtual object. (A) shows a case where the influence of the polarization of the reflected light on a real object is not taken into consideration, and (B) shows a case where the influence of the polarization of the reflected light on a real object is taken into consideration. It is a figure explaining the case where total reflection occurs and the case where total reflection does not occur in a real object. (A) shows the positional relationship of the eyeball, the real object, and the virtual object when total reflection does not occur, and (B) shows the positional relationship of the eyeball, the real object, and the virtual object when total reflection occurs. It is a figure explaining the drawing example of the image of the virtual object in the case of a plurality of real objects. (A) is a diagram showing the positional relationship between a real object and a virtual object, and (B) is a diagram showing a mixed reality perceived by the user. It is a figure explaining the principle that the user who wears the display device which displays the image which combined the virtual object with the image of the outside world imaged in real time in the experience of mixed reality, experiences the mixed reality. It is a figure which shows an example of the functional structure of a display device.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<Embodiment 1>
In the present embodiment, a case where a glasses-type terminal that can transparently see the outside world is used for the experience of mixed reality will be described.
FIG. 1 is a diagram illustrating a principle that a user wearing a glasses-type terminal 1 that can transparently see the outside world experiences mixed reality.

The hardware portion of this type of terminal 1 has already been put into practical use by a plurality of manufacturers. For example, Microsoft's HoloLens ™, Sony's SmartEyeglass ™, and Konica Minolta's Wearable Communicator ™. This type of terminal 1 is also called a transmissive device, a retinal projection device, or the like.
The glasses-type terminal 1 shown in FIG. 1 has a light guide plate 2 having high transparency, a small display unit 3 for displaying an image, and a virtual object drawing unit 4 for drawing a virtual object (virtual object 11). ing.
The glasses-type terminal 1 here is an example of an information processing device and also an example of an information processing system. Further, the virtual object drawing unit 4 is an example of a generation means and a drawing means.

The light guide plate 2 is made of, for example, a member having a transparency of 85% or more, and a visible light transmission type diffraction grating (not shown) is arranged inside the light guide plate 2. For the visible light transmission type diffraction grating, for example, a holographic diffraction grating is used.
The visible light transmission type diffraction grating acts to linearly transmit the external light B1 incident from the front of the light guide plate 2 and guide it to the user's eyeball 5. On the other hand, the visible light transmission type diffraction grating refracts the display light B2 incident on the light guide plate 2 from the display unit 3 to propagate inside the light guide plate 2, and then refracts the display light B2 in the direction of the eyeball 5. It works like that.
The external light B1 and the display light B2 are combined in the eyeball 5. As a result, the user wearing the terminal 1 perceives a mixed reality landscape in which a virtual object (virtual object 11) is synthesized with a real object (real object 12). Incidentally, in the example of FIG. 1, the virtual object 11 is located on the front side of the real object 12.

<Hardware configuration of glasses-type terminal 1>
FIG. 2 is a diagram showing an example of the hardware configuration of the glasses-type terminal 1.
The terminal 1 shown in FIG. 2 includes a CPU (Central Processing Unit) 21 that controls the entire device through execution of a program (including basic software), and a ROM 22 that stores programs such as a BIOS (Basic Input Output System) and basic software. , A RAM (Random Access Memory) 23 used as a program execution area.
The ROM 22 is composed of, for example, a non-volatile semiconductor memory in which data can be electrically rewritten.
The CPU 21, ROM 22, and RAM 23 function as a computer 20.

The computer 20 includes display units 3L and 3R that display virtual objects, cameras 24L and 24R that capture the outside world, inertial measurement sensors 25 that measure inertial information such as angles, angular speeds, and accelerations, and even real objects. A depth sensor 26 for measuring the distance of the object, an illuminance sensor 27 for detecting the brightness of the surroundings, and a wireless communication unit 28 used for communication with the outside are connected.
An image for the left eye is displayed on the display unit 3L for the left eye, and an image for the right eye is displayed on the display unit 3R for the right eye. Parallax is reproduced in the image for the left eye and the image for the right eye. Therefore, the user wearing the terminal 1 can see the virtual object 11 stereoscopically.

The camera 24L is arranged on the left eye side of the user, and the camera 24R is arranged on the right eye side of the user. The surroundings of the terminal 1 are photographed in stereo by the cameras 24L and 24R. The images captured by the cameras 24L and 24R are used for recognizing a real object and measuring the distance to the surface of the real object. A camera used for measuring the distance to a real object and a camera used for recognizing a real object may be prepared separately.
The inertial measurement sensor 25 is used for measuring the position and orientation of the head, and is used for tracking the line of sight.
The depth sensor 26 measures the distance to an object existing in the real space by using infrared rays or ultrasonic waves.

<Functional configuration of glasses-type terminal 1>
FIG. 3 is a diagram showing an example of the functional configuration of the glasses-type terminal 1.
The functional configuration shown in FIG. 3 is realized through the execution of a program by the CPU 21.
In the functional configuration shown in FIG. 3, among various functions realized through the execution of the program, a virtual object is within the range in which the image of reflection is reflected on the real object due to the positional relationship between the real object and the user. It describes the function of making the user perceive the complex real space in a certain case.

In the case of FIG. 3, the CPU 21 has a real space information acquisition unit 31 that acquires real space information from images captured by the cameras 24L and 24R, and a reflection of a real object 12 (see FIG. 1) based on the captured images. A real object reflection estimation unit 32 for estimating sex, a real object reflection information acquisition unit 33 for acquiring information about the reflectivity of the real object 12 (hereinafter referred to as reflection information), and an eyeball 5 (see FIG. 1). A real object reflection area determination unit 34 for determining a region (hereinafter referred to as a reflection region) in which an image of a virtual object 11 (see FIG. 1) is reflected in a real object 12 having reflectivity based on a position, and a display unit 3L. And 3R (see FIG. 2) have a virtual object 11 and a virtual object drawing unit 4 for drawing an image of the virtual object 11.

The real space information acquisition unit 31 acquires various information about the real space from the captured image and stores it in the RAM 23 as the real space information 41.
The type of information stored as the real space information 41 differs depending on the situation and application in which the glasses-type terminal 1 is used.
However, by increasing the types of information, it is possible to bring the experience in mixed reality space closer to the experience in real space.
In the case of the present embodiment, the real space information 41 includes not only the information about the real object 12 added in real time but also the information about the real object 12 given or acquired in advance.

The information about the real object 12 may be estimated (calculated) from the captured image, or may be stored in the non-volatile area of the RAM 23 as known information for each real object 12.
The information estimated from the captured image includes information that can be directly acquired from the captured image such as color information, and information estimated by using a method described later.
In the non-volatile region of the RAM 23, for example, information applied to all of the reflective portions of the real object 12 (including a representative value of the reflection information and a formula for calculating the reflection information) is stored. In the non-volatile region of the RAM 23, information for each portion having reflexivity may be stored.
Then, the real space information acquisition unit 31 in the present embodiment acquires information about each real object 12 specified by image recognition from the RAM 23.

Further, the information stored in the RAM 23 may include information of several types of filters that reproduce the appearance when an image of another real object 12 is reflected on a certain real object 12. Each filter is a combination of one or more items of reflection characteristics (eg, reflectance, normal reflectance, diffuse reflectance, glossiness, information on polarization of reflected light, etc.) described below. Given.
Further, the real space information acquisition unit 31 according to the present embodiment may be provided with a function of acquiring these filters. The filter is acquired, for example, by imaging a portion of a real object 12 in which an image of another real object 12 is reflected. The filter here is an example of reflection information.
Then, if the virtual object 11 is within the range in which the reflection image is reflected on the real object 12 due to the positional relationship between the real object 12 and the user (eyeball 5), the virtual object 11 reflecting the filter of the real object 12 is reflected. By drawing the image of, it is drawn in the same way as a real object is reflected as an image.

The information about the real object 12 includes, for example, information about an individual object (including a person), information about the real space where the user is located, information about the distance from the user's position to each position in the image, information about the light source, and imaging. Information etc. are included.
Here, the information of an individual object includes, for example, a shape, a color tone, a material, a pattern, information for specifying a position in a real space, and reflection information for giving various information regarding reflectivity. Existing techniques are used to recognize objects. For example, a method of detecting an edge or a color region as a feature amount is used. Artificial intelligence may be used for recognizing an object.
The information related to imaging includes information on the positions of the cameras 24L and 24R in the actual space, the direction of movement of the cameras 24L and 24R in the actual space, information on the direction in which the cameras 24L and 24R are imaged in the actual space, and the like. Is included. Information regarding the date and time of imaging is also attached to the images captured by the cameras 24L and 24R.

Further, the reflection information includes, for example, the reflection characteristics of the real object 12, information on the portion having and not having reflection in the real object 12, information on the color tone, pattern, shape, and polarization of the reflected light of the portion having reflection. , Information that identifies the position in the real space, etc. are included.
Here, the reflection generally includes specular reflection and diffuse reflection. Specular reflection is reflection without diffusion, and incident light is reflected in a certain direction. Diffuse reflection is reflection that does not have specular reflection as seen by the naked eye, and incident light is reflected in various directions. For example, in the case of a mirror, specular reflection occurs at all positions on an extremely smooth mirror surface, so that the image of the original object can be seen accurately. On the other hand, diffuse reflection occurs in an object with a rough surface such as paper or cloth, and the incident light is reflected in various directions due to the fine irregularities on the surface, so that the image is distorted and the original shape of the object is changed. I don't know.

Therefore, in the present embodiment, the reflective real object 12 is a reality having a reflectivity such that an image of another real object is reflected on the surface, in other words, a mirror surface reflectivity in which normal reflection occurs. It shall refer to the object 12. Further, the reflection characteristics included in the reflection information include, for example, information on the reflectance of the real object 12, the normal reflectance, the diffuse reflectance, the glossiness, the polarization of the reflected light, and the like. Reflectance is the ratio of the reflected radiant flux or luminous flux to the radiant flux or luminous flux incident on the object. The normal reflectance is the ratio of the normal reflection component of the reflected radiant flux or the luminous flux to the incident radiant flux or the luminous flux. Diffuse reflectance is the ratio of the diffuse reflection component of the reflected radiant flux or the luminous flux to the incident radiant flux or the luminous flux. The glossiness is an index that one-dimensionally expresses the degree of gloss on the surface of an object by paying attention to the ratio of specularly reflected light and the directional distribution of diffusely reflected light.

The positive reflectance and the glossiness can be regarded as the degree of reflectivity (specular reflectivity in which normal reflection occurs) such that an image of another real object is reflected on the surface of the real object 12. For example, if the value of the positive reflectance of the real object 12 is 100%, the image reflected on the surface of the real object 12 can be seen more accurately, but as the value of the positive reflectance decreases (that is, diffusion). (As the reflectance increases), the image collapses and looks blurry.
In addition, the reflective real object 12 is, for example, a real object 12 whose reflectance (and specular reflectance, diffuse reflectance, glossiness, etc.) exceeds a predetermined threshold.

The reflection information may be estimated through image processing or may be given in advance. The method of estimating the reflectivity includes a method of discovering the relationship between the pattern appearing on the surface of the real object 12 and another real object 12, and a method of acquiring the reflection information corresponding to the object specified by artificial intelligence from the database. And so on. The database may be stored in a server (not shown) on the cloud network, for example. If the reflection information corresponding to the specified object does not exist in the database, the artificial intelligence may estimate the reflection information corresponding to the specified object based on the information of the similar article existing in the database. ..
The texture of the object changes depending on the combination of individual elements included in the reflection information.
The real space information 41 may be stored in, for example, a server (not shown) on the cloud network.

The real space information acquisition unit 31 in the present embodiment is also provided with a function of generating or updating a three-dimensional model imitating the real space (that is, a function of virtualizing the real space).
The real space information acquisition unit 31 consistently integrates a plurality of information acquired from the real space in the virtual space, and generates or updates a three-dimensional model. The three-dimensional model here is stored in the RAM 23 as the real space virtualization information 42.
A mixed reality space is a space (three-dimensional model) in which a virtual object 11 is arranged in a virtual space (three-dimensional model).

The real object reflectivity estimation unit 32 in the present embodiment has, for example, an image of the surface of the real object 12 defined as an estimation target and an image of an object existing ahead of the direction reflected by the real object 12. By comparison, the reflectivity of the real object 12 is estimated. The real object reflectivity estimation unit 32 also has a function of estimating the reflection characteristics such as the reflectance, the normal reflectance, the glossiness, and the information on the polarization of the reflected light of the real object 12.
The real object reflectivity estimation unit 32 here is an example of the estimation means.
The estimation result is stored as a part of the real space information 41 associated with the real object 12 defined as the estimation target.
The accuracy of estimation improves with the accumulation of real space information 41 and real space virtualization information 42.

The real object reflection information acquisition unit 33 in the present embodiment acquires reflection information from the real space information 41 of the real object 12.
In the case of the present embodiment, the real object reflection information acquisition unit 33 targets the real object 12 estimated to have reflectivity by the real object reflectivity estimation unit 32 to acquire the reflection information.

The real object reflection area determination unit 34 in the present embodiment determines a reflection area, which is a region in which the image of the virtual object 11 is reflected in the real object 12 having reflectivity. In addition, when it is determined that the real object 12 has a reflection region, the virtual object 11 is within the range in which the reflection image is reflected on the real object 12 due to the positional relationship between the real object 12 and the user (eyeball 5). Can be regarded as something.
Then, the virtual object drawing unit 4 associates the information of the virtual object 11 that reflects the reflection information of the real object 12 with the reflection region where the image of the virtual object 11 is determined to be reflected in the real object 12. The information of the virtual object 11 here is the information of the virtual object 11 drawn as an image of the virtual object 11. That is, it is information such as the shape, color tone, and material of the portion of the virtual object 11 that is reflected in the real object 12.

Information such as the position (position in the three-dimensional model) where the virtual object 11 is arranged, the shape, the color tone, and the material is stored as the virtual object information 43. Further, the position of the user's eyeball 5 is not actually measured, but is given in relation to the terminal 1.
In the present embodiment, the information of the virtual object 11 that reflects the reflection information of the real object 12 is referred to as virtual object image information 44. The content of the virtual object image information 44 also changes depending on the movement of the user wearing the terminal 1, the movement of the object in the real space, and the position where the virtual object 11 is arranged.

The virtual object drawing unit 4 uses the real space virtualization information 42, the virtual object information 43, and the virtual object image information 44, and uses an image of the virtual object 11 and an image of the virtual object 11 for the display unit 3L (see FIG. 2). And the image of the virtual object 11 for the display unit 3R (see FIG. 2) and the image of the image of the virtual object 11 are drawn.
The virtual object drawing unit 4 in the present embodiment also includes the reflection area of the real object 12, which is the area where the image of the virtual object 11 is reflected, in the drawing target. That is, in addition to the virtual object 11, the image of the virtual object 11 is drawn.
By displaying the image of the virtual object 11 on the real object 12 having reflectivity in this way, the user can perceive that the virtual object 11 is reflected in the real object 12. As a result, the reality of mixed reality can be enhanced as compared with the conventional technique.
Further, the virtual object drawing unit 4 in the present embodiment has a reflectance, a normal reflectance, and a color tone of the real object 12 in order to make the appearance of the image of the virtual object 11 reflected on the reflective real object 12 closer to reality. , Pattern, shape, etc. are reflected in the drawing of the image of the virtual object 11.

<Processing operation executed by the glasses-type terminal 1>
FIG. 4 is a flowchart illustrating an example of a processing operation executed when drawing an image of a virtual object 11 on a glasses-type terminal 1.
The processing operation shown in FIG. 4 is realized through the execution of the program by the CPU 21. In the figure, the step is represented by the symbol S.

First, the CPU 21 acquires information in the real space (step 1). By this process, the CPU 21 recognizes the real object 12 that the user wearing the terminal 1 sees through the light guide plate 2.
Next, the CPU 21 estimates the reflectivity of the recognized real object 12 (step 2).
The reflectivity estimation process is not limited to the real object 12 whose information on the reflectivity is unknown, and may include the real object 12 for which the estimation process has been performed before.
This is because even the real object 12 previously targeted for the estimation process may be able to acquire new information regarding the reflectivity due to the difference in the environment when the image is acquired.
The process of estimating the reflectivity may be executed independently of the process of drawing the virtual object 11.

FIG. 5 is a diagram illustrating an example of a method used for estimating reflectivity. (A) shows the positional relationship between the real objects 12A and 12B, and (B) shows the positional relationship between the real objects 12A and 12B and the eyeball 5 (see FIG. 1).
In the example of FIG. 5, the CPU 21 performs processing with the real object 12A as the estimation target of the reflectivity. The real object 12A is a mirror, and the real object 12B is an information mobile terminal. Further, the image 12C of the real object 12B is reflected in the real object 12A.
In the case of (A), the eyeball 5 of the user who wears the terminal 1 is located on a normal line extending toward the front from the paper surface.

In estimating the reflectivity, as shown in (B), the CPU 21 has the eyeball 5 to the real object 12A based on the positional relationship between the position of the eyeball 5 (or the terminal 1) and the real object 12A which is the object of estimation of the reflectivity. The reflection angle, which is the angle between the incident light directed toward the object and the reflected light reflected by the real object 12A, is calculated.
Here, the CPU 21 uses the information of the real space information 41 and the real space virtualization information 42 (for example, the information of the position stored for each real object 12), and the position of the real object 12A and the real object 12A are placed. Calculate the reflection angle by grasping the angle of the object. Further, the calculation of the reflection angle is performed on the reflection surface 51 of the real object 12A to which the incident light from the eyeball 5 is reflected. Further, the calculation of the reflection angle is performed at each point on the reflection surface 51.

Next, the CPU 21 traces from the real object 12A (reflection surface 51) and identifies an object existing at a position projected on the surface of the real object 12A. Here, the CPU 21 identifies an object existing in the direction of the reflection angle from each point on the reflection surface 51 of the real object 12A. Here, too, the CPU 21 identifies the object by using the information of the real space information 41 and the real space virtualization information 42.
The example shown in (A) shows a case of tracing from five points on the reflection surface 51 of the real object 12A, and the real object 12B is specified as an object existing at a position projected on the real object 12A.

Next, the CPU 21 analyzes the image captured by the reflective surface 51 of the real object 12A and the image captured by the real object 12, and determines whether or not the image of the reflective surface 51 matches the image of the real object 12B. do. Here, by analyzing the image of the reflection surface 51, the image of the image 12C of the real object 12B is extracted. Further, by tracing from the image 12C, the portion 52 of the real object 12B corresponding to the image 12C is specified. Then, when a high similarity is observed between the shape obtained from the image of the reflecting surface 51 (the image of the image 12C) and the shape obtained from the image of the portion 52, it is determined that the two match. It should be noted that the reason why the matching of the shapes of the two is not required is that the appearance of the real object 12B directly seen from the eyeball 5 and the real object 12B reflected in the real object 12A are different. For example, when the front surface of the real object 12B is directly visible from the eyeball 5, the back surface of the real object 12B is reflected in the real object 12A.

In the example of FIG. 5, the image of the image 12C is a rectangle, and the shape of the real object 12B (part 52) identified by tracing from each point of the rectangle of this image is also a rectangle. Therefore, the CPU 21 estimates that the image of the image 12C matches the image of the portion 52 of the real object 12B, and that the real object 12A has reflectivity. Here, for example, by comparing the elements obtained from the image, such as comparing the ratio between the long side and the short side of the image of the image 12C and the ratio between the long side and the short side of the portion 52, the matching is more accurate. The sex may be determined.
Then, in the case of the present embodiment, it is estimated that the entire reflecting surface 51 of the real object 12A has reflectivity. However, it is estimated that the image 12C has reflectivity only in the region where the image exists, and the presence or absence of reflectivity in the remaining region may be estimated at another opportunity.

The CPU 21 also estimates reflection characteristics such as reflectance, specular reflectance, glossiness, and information on the polarization of reflected light.
For example, the CPU 21 processes the image of the image 12C, and calculates the reflectance, specular reflectance, glossiness, etc. of the reflecting surface 51 based on the brightness, color depth, degree of blurring, etc. of this image. Can be done. For example, the CPU 21 compares the brightness of the two images, the color depth, the degree of blurring, etc. with respect to the image of the image 12C and the image of the real object 12B, thereby determining the reflectance and the normal reflectance of the reflecting surface 51. Calculate glossiness and so on. Further, for example, the reflectance can be calculated by substituting the refractive index obtained for the reflecting surface 51 into a known formula. The CPU 21 may calculate an index of the reflection characteristic that summarizes the reflectance, the regular reflectance, and the glossiness, instead of individually outputting the reflectance, the regular reflectance, and the glossiness.
Further, for example, as a method for estimating information regarding the polarization of the reflected light in the real object 12A, for example, a polarizing filter is provided in front of and behind the light guide plate 2 (see FIG. 1) to image the real object 12A, and the image is obtained from the image. An example is a method of estimating the intensity, direction, etc. of polarization based on the change in the amount of received light (the amount of reflected light). For example, the real object 12A is imaged by the cameras 24L and 24R (see FIG. 2) at each angle at which the polarization filter is rotated. Then, the CPU 21 determines the change in the amount of reflected light in the real object 12A by comparing the images for each angle, and estimates the intensity, direction, and the like of the polarization of the reflected light in the real object 12A.

Further, the user wears the glasses-type terminal 1 and moves, and the real space information 41 is stored in accordance with the movement, so that the real space information 41 is accumulated. Therefore, in the case of FIG. 5, the portion of the real object 12B (for example, the real object) reflected in the real object 12A, for example, by the user moving and taking images with the cameras 24L and 24R (see FIG. 2) from various directions. The back surface of 12B) may be imaged and stored as real space information 41. In this case, the CPU 21 can determine whether or not the image of the image 12C and the image of the real object 12B match by pattern matching or the like, and can estimate the reflectivity of the real object 12A more accurately. .. Further, regarding the estimation of the reflection characteristics such as the reflectance, the normal reflectance, and the glossiness, the image of the image 12C and the part of the real object 12B reflected on the real object 12A can be compared, and the difference between the two makes it more accurate. The reflection characteristics can be calculated.

FIG. 6 is a diagram illustrating another example of the method used for estimating the reflectivity. In the estimation method described with reference to FIG. 6, the CPU 21 matches, for example, the content appearing on the surface of the real object 12 with an image in which the left and right sides of the image captured when the image is captured in the direction opposite to the current imaging direction are exchanged. In this case, it is presumed that the content appearing on the surface of the real object 12 is a mirror image, and the real object 12 has reflectivity. In this case, it is necessary to capture images with the cameras 24L and 24R in advance from the direction opposite to the current imaging direction, and save the captured images as real space information 41.
In the example of FIG. 6, the real objects 12A to 12C are present in front of the eyeball 5. Then, the CPU 21 performs processing with the real object 12A as the estimation target of the reflectivity. The image 12D of the real object 12B is reflected in the real object 12A, and the image 12E of the real object 12C is reflected in the real object 12A.

Here, by capturing images in advance with the cameras 24L and 24R in the direction opposite to the current imaging direction, the image of the portion reflected in the real object 12A by the real object 12B and the reality by the real object 12C. The image of the portion reflected on the object 12A is stored as the real space information 41. In such a case, the CPU 21 identifies the real objects 12B and 12C existing at the positions projected on the surface of the real object 12A in the three-dimensional model imitating the real space. Then, the CPU 21 determines whether or not the image of the surface of the real object 12A matches the image of the real objects 12B and 12C obtained by imaging from the direction opposite to the current imaging direction by pattern matching or the like. judge. For example, when it is recognized that the physical structure, color, pattern, etc. match (or the physical structure, color, pattern, etc. have high similarity), the CPU 21 indicates that the real object 12A has reflectivity. Then it can be estimated.
The current imaging direction is an example of one direction.

Further, FIG. 7 is a diagram illustrating another example of the method used for estimating the reflectivity. Three estimation methods will be described with reference to FIG. 7.
In the example of FIG. 7, the CPU 21 performs processing with the real object 12F as the estimation target of the reflectivity. The real object 12F is a mirror, and the real object 12G is an outlet fixedly attached to the wall. Further, the image 12H of the real object 12G is reflected on the real object 12F. Further, a light source 12I is fixedly attached to the ceiling.
The eyeball 5 of the user who wears the terminal 1 is located on a normal line extending toward the front from the paper surface.

First, in the first estimation method, the CPU 21 estimates when a bright portion (highlight) exists at a position estimated (specified) from the relationship between the position of the light source and the inclination of the surface of the real object 12F. It is presumed that the target real object 12F has reflectivity.
For example, when the real object 12F has reflectivity, the light from the light source 12I is combined with the reflected light reflected by the real object 12F and the direct light from the light source 12I to generate a bright portion. In the illustrated example, a bright portion is generated in the region 16A, and the brightness is higher than that of 16B.
The CPU 21 specifies a region (in this example, region 16A) where a bright portion is presumed to occur from the relationship between the position of the light source 12I and the inclination of the surface of the real object 12F. Then, the CPU 21 compares the brightness of the region 16A estimated to generate a bright portion with the brightness of the surrounding region (for example, the region 16B), and the brightness of the region 16A estimated to generate a bright portion is obtained. When is high, it is estimated that the real object 12F has reflectivity.
The reflection characteristic of the real object 12F may be estimated based on the degree of difference between the brightness of the region where the bright portion is estimated to occur and the brightness of the surrounding region. For example, the greater the difference between the brightness of the area where bright areas are estimated to occur and the brightness of the surrounding area, the greater the reflectance (and specular reflectance, diffuse reflectance, glossiness, etc.) of the real object 12F. Presumed.

The brightness of the region where the bright part is estimated to occur and the brightness of the surrounding region are estimated from the image obtained by capturing the target region.
Further, the information regarding the light source 12I may be estimated from the captured image, or may be stored as known information in the non-volatile region of the RAM 23. As a method for estimating the position of the light source 12I, there is a ray tracing method (ray tracing) or the like.

Next, in the second estimation method, the CPU 21 determines that the content (individual image) appearing on the surface of the real object 12F to be estimated and another real object 12 are both present at symmetrical positions. (In other words, another real object 12 that is paired with the content that appears on the surface of the real object 12F that is the object of estimation has a common outer shape and sandwiches the surface of the real object 12F that is the object of estimation. (When they are located at equal distances), it is estimated that the real object 12F has reflectivity.
For example, the CPU 21 extracts the real object 12G as an object existing in the range where the reflection image is reflected on the real object 12F by analyzing the images obtained by capturing the regions 16A, the region 16B, and the like. Further, by analyzing the image of the real object 12F, the image 12H of the real object 12G having the same outer shape as the real object 12G (that is, a high similarity in shape is recognized) is extracted. Then, since it is determined that the real object 12G and the image 12H are equidistant across the surface of the real object 12F, it is estimated that the real object 12 has reflectivity.

Next, in the third estimation method, the CPU 21 causes the real object 12F to be reflective when the brightness of the surface of the real object 12F to be estimated is darker than when it is directly illuminated by the light source. Presumed to have.
For example, the CPU 21 estimates the brightness of the region 16C by analyzing an image of the region 16C adjacent to the real object 12F. Further, the brightness of the region 16D adjacent to the region 16C is estimated by analyzing the image obtained by capturing the image of the real object 12F. Here, since the region 16C and the region 16D are adjacent to each other, there should be no significant difference in brightness when illuminated by the light source 12I. However, the luminance of the region 16D is lower than the luminance of the region 16C by a certain degree (for example, a degree exceeding a predetermined luminance difference). In other words, the brightness of the region 16D is darker than when directly illuminated by the light source 12I.
As described above, although the region 16C and the region 16D are adjacent to each other, it can be said that there is no connection between the brightness of the region 16C and the region 16D. This is because the image of the space far away from the light source 12I is reflected in the area 16D. Therefore, the CPU 21 determines that the brightness of the surface of the real object 12F is darker than that when it is directly illuminated by the light source 12I, based on the difference in brightness between the area 16C and the area 16D, and the real object 12 has a reflectivity. Presumed to have.

Further, it is conceivable that the content appearing on the surface of the real object 12 is not an image due to reflection but a physical structure or pattern, or an image displayed on a display device. Therefore, the CPU 21 may estimate the reflectivity by determining whether or not the content appearing on the surface of the real object 12 is different from the physical structure or pattern and the image displayed on the display device.

In the present embodiment, when the content appearing on the surface of the real object 12 changes continuously in conjunction with the movement of the cameras 24L and 24R (see FIG. 2) that image the real object 12, the surface of the real object 12 It is determined that the content appearing in is different from the physical structure and pattern and the image displayed on the display device. This is because when the content appearing on the surface of the real object 12 is due to a physical structure or pattern, the content does not change even if the cameras 24L and 24R (see FIG. 2) move. Further, when the real object 12 is a display device, the change in the content of the display is irrelevant to the movement of the cameras 24L and 24R that image the real object 12. By making such a determination, the reflectivity is estimated.

Further, as another method used for estimating the reflectivity, for example, by specifying what kind of object the real object 12 is based on the real space information 41 by artificial intelligence, it corresponds to the specified object. The reflection information to be used may be obtained from the database.
As for the various methods for estimating the reflectivity described above, any one method may be used for estimation, or two or more methods may be combined for estimation.
Returning to the description of FIG.

After step 2, the CPU 21 acquires the reflection information of the real object 12 presumed to have reflectivity in step 2 (step 3). In the case of this embodiment, the CPU 21 acquires the reflection characteristic of the real object 12 calculated in step 2. Further, the CPU 21 can acquire reflection information without estimating the reflectivity, for example, the color tone, the pattern, the shape of the real object 12 estimated to have the reflectivity, the information for specifying the position in the real space, and the like. Get the reflection information of.

Subsequently, the CPU 21 selects an unselected one of the one or a plurality of virtual objects 11 to be drawn (step 4).
The CPU 21 uses the selected virtual object 11 as a processing target, and determines whether or not there is a reflection region of the real object 12 on which the image of the virtual object 11 is reflected (step 5).
Here, the CPU 21 reflects the image of the virtual object 11 to be processed on the reflective real object 12 with reference to the position of the eyeball 5 (see FIG. 1) of the user wearing the terminal 1. Judge whether or not.

Whether or not the image of the virtual object 11 is reflected is determined based on the position of the eyeball 5 of the user wearing the terminal 1. For example, when the surface of the real object 12 having reflectivity is a plane, in a three-dimensional model simulating the real space, the surface of the real object 12 having reflectivity is set as a plane of symmetry, and the position is plane-symmetrical to the virtual object 11. , A reflection image of the virtual object 11 is created. It is determined whether or not the virtual straight line connecting the reflected image of the virtual object 11 and the position of the eyeball 5 intersects with the real object 12.
For example, when the virtual straight line and the real object 12 do not intersect, the CPU 21 obtains a negative result and proceeds to step 7.

FIG. 8 is a diagram illustrating a reflection region 14 in which an image of a virtual object 11 is reflected in a real object 12. (A) shows the relationship between the positions of the real object 12 and the virtual object 11, and (B) shows the reflection region 14 of the real object 12 on which the image of the virtual object 11 is reflected.
As shown in (B), the reflection image 13 of the virtual object 11 is formed at a position symmetrical to the surface of the virtual object 11 with the surface of the real object 12 as the plane of symmetry. The range where the virtual straight line connecting the reflected image 13 of the virtual object 11 and the position of the eyeball 5 intersects with the real object 12 is the reflection region 14, which is the range shown by the thick line in the figure.

As described above, how to draw the image of the virtual object 11 with respect to the real object 12 is determined by the positional relationship between the user's eyeball 5, the virtual object 11, and the real object 12. For example, as shown in FIG. 8, a virtual straight line connecting the reflected image 13 of the virtual object 11 and the eyeball 5 with respect to the reflected image 13 of the virtual object 11 when the surface of the real object 12 is the plane of symmetry. An image of the virtual object 11 is drawn at a position (range) where the real object 12 intersects. That is, for each point of the reflection image 13 of the virtual object 11, the position where the virtual straight line facing the eyeball 5 and the real object 12 intersect is derived, and the image of the virtual object 11 is drawn.

However, the example shown in FIG. 8 is an example when the surface of the real object 12 is a flat surface. Since the surface of the real object 12 may be a curved surface such as a concave shape or a convex shape in addition to a flat surface, the image of the virtual object 11 changes according to the shape of the real object 12. In addition, the image of the virtual object 11 is drawn in the same manner as the image reflected on the real object 12 when the virtual object 11 is assumed to exist as a real object.

When there are a plurality of real objects 12 to which the image of the virtual object 11 is reflected, the CPU 21 specifies a reflection region in which the image of the virtual object 11 is reflected for each real object 12.
Further, when there are a plurality of reflective parts in one real object 12, the CPU 21 specifies a reflective region in which the image of the virtual object 11 is reflected in each of the reflective parts.
Therefore, the number of reflection regions specified for one virtual object 11 is not limited to one. When a plurality of reflection areas are specified, the reflection areas do not always match.
Returning to the description of FIG.

When an affirmative result is obtained in step 5, the CPU 21 associates and stores the information of the virtual object 11 reflecting the reflection information of the real object 12 for each reflection region specified in step 5 (step 6). That is, the CPU 21 stores the virtual object image information 44 in association with the reflection region of the real object 12.
Here, for example, in the example shown in FIG. 8, the information of the virtual object 11 is associated so that the reflection image 13 of the virtual object 11 can be seen from the position of the user. More specifically, for example, a portion of the virtual object 11 that is different from the portion directly visible to the user (for example, if the front surface of the virtual object 11 is directly visible from the user's position, the back surface of the virtual object 11). Information such as the shape, color tone, and material of the part is associated so that the part can be seen. At that time, the virtual object 11 associated with the reflection region is smaller than the virtual object 11 directly visible to the user, for example, based on perspective, in order to express the reflection image 13 of the virtual object 11 so that the user can see it. Associated with each other.

After that, the CPU 21 determines whether or not all the virtual objects 11 have been selected (step 7).
If a negative result is obtained in step 7, the CPU 21 returns to step 4. In step 4, one of the unselected virtual objects 11 is selected as the processing target.
On the other hand, when an affirmative result is obtained in step 7, the CPU 21 draws an image of the virtual object 11 for all the reflection regions using the associated virtual object image information 44 (step 8).

<Drawing example>
Hereinafter, an example of drawing an image of the virtual object 11 in the present embodiment will be described with reference to a specific example.

<Drawing example 1>
FIG. 9 is a diagram illustrating a difference between drawing by the conventional technique and drawing by the present embodiment. (A) is a drawing example by the conventional technique, and (B) is a drawing example by the present embodiment.
In FIG. 9, a drawing example by the conventional technique is described as a comparative example.
In FIG. 9, the eyeball 5 (see FIG. 1) of the user who wears the terminal 1 is located on a normal line extending toward the front from the paper surface.

As shown in (A), in the conventional technique, when the cylindrical virtual object 11 is within the range where the image of reflection is reflected on the real object 12 due to the positional relationship between the flat plate-shaped real object 12 and the user. Even if the real object 12 has reflectivity, the image of the virtual object 11 is not drawn on the real object 12.
Therefore, in the reflective real object 12, the image of the virtual object 11 to be reflected is not perceived by the user, and instead, the real object 12 is perceived by the user as it is.
In this way, the reflective real object 12 creates an unnatural state in the landscape perceived by the user.

On the other hand, in the case of the present embodiment, as shown in (B), the cylindrical shape is within the range in which the image of reflection is reflected on the real object 12 due to the positional relationship between the flat plate-shaped real object 12 and the user. If there is a virtual object 11 of the above, the user can perceive the image 15 of the cylindrical virtual object 11.
As described above, by using the technique according to the present embodiment, the real object 12 having reflexivity does not cause an unnatural state in the landscape, and the virtual object 11 can be experienced as if it actually exists. ..

Further, by using the technique according to the present embodiment, even when the user moves in the real space, the real object 12 is virtualized by the positional relationship between the user's eyeball 5, the virtual object 11, and the reflective real object 12. The image 15 of the object 11 can be drawn. That is, even if the user moves in the real space, the image 15 of the virtual object 11 can be continuously perceived.
Therefore, the virtual object 11 can be perceived by the user without distinguishing it from the real object.

In addition, even when the user and the virtual object 11 do not move and only the real object 12 having reflectivity moves, or when the real object 12 having reflectivity with the user does not move and only the virtual object 11 moves. Since the same unnatural phenomenon disappears, the sense of reality of the virtual object 11 can be enhanced.
Further, as compared with the conventional technique, it becomes easier for the user to notice the virtual object 11 existing at the position where the image 15 is drawn with respect to the real object 12.

<Drawing example 2>
FIG. 10 is a diagram illustrating the effect of the difference in reflectance of the real object 12 on the drawing of the image 15 of the virtual object 11. (A) is a drawing example of the image 15 of the virtual object 11 when the reflectance of the real object 12 is high, and (B) is a drawing example of the image 15 of the virtual object 11 when the reflectance of the real object 12 is low. be.
Also in the case of FIG. 10, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 (see FIG. 1) is located on a normal line extending toward the front from the paper surface.

In the case of FIG. 10, when the reflectance 1 is relatively high (in the case of (A)), for example, compared to the case where the reflectance 2 is relatively low (in the case of (B)). The image 15 of the virtual object 11 is drawn by increasing the resolution or darkening the color to add the influence of the reflectance of the real object 12. Therefore, the case where the reflectance 1 is relatively high (in the case of (A)) is higher than the case where the reflectance 2 is relatively low (in the case of (B)), the virtual object 11. It is possible to clearly perceive the image 15 of.
Therefore, the user can experience as if the virtual object 11 actually exists.

In the examples of (A) and (B), the influence of the normal reflectance (or diffuse reflectance, glossiness) of the real object 12 is not taken into consideration, but the image 15 of the virtual object 11 is actually drawn. In addition to the reflectance of the object 12, the positive reflectance (or diffuse reflectance, glossiness) of the real object 12 also has an effect. For example, by adding the influence of the specular reflectance of the real object 12, the image 15 of the virtual object 11 becomes clearer when the specular reflectance is relatively high than when the specular reflectance is relatively low. It is possible to perceive.

<Drawing example 3>
FIG. 11 is a diagram illustrating the effect of the difference in color of the real object 12 on the drawing of the image 15 of the virtual object 11. (A) is a drawing example of the image 15 of the virtual object 11 when the real object 12 is light blue, and (B) is the drawing example of the virtual object 11 when the real object 12 is light red. This is a drawing example of the image 15.
Also in the case of FIG. 11, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 (see FIG. 1) is located on a normal line extending toward the front from the paper surface.

Due to drawing restrictions, it is not possible to express light blue or light red in FIG. 11, but when the real object 12 is light blue (in the case of (A)), the virtual object 11 The image 15 is drawn with a light blue color. Further, when the real object 12 is colored with a light red color (in the case of (B)), a process of drawing the image 15 of the virtual object 11 with a light red color is performed. By adding the influence of the color tone of the real object 12 in this way, the appearance of the image 15 of the virtual object 11 reflected in the real object 12 is changed to the appearance of other objects in the real space reflected in the real object 12. It will be possible to get closer.
Therefore, the user can experience as if the virtual object 11 actually exists.

<Drawing example 4>
FIG. 12 is a diagram for explaining the influence of the difference in the pattern attached to the real object 12 on the drawing of the image 15 of the virtual object 11. (A) is a drawing example of the image 15 of the virtual object 11 when a diagonal line extending in the diagonal direction is formed on the surface of the real object 12, and (B) is a drawing example of a mesh pattern on the surface of the real object 12. It is a drawing example of the image 15 of the virtual object 11 in the case where it is done.
Also in the case of FIG. 12, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 (see FIG. 1) is located on a normal line extending toward the front from the paper surface.
As shown in FIG. 12, by reflecting the pattern formed on the surface of the real object 12 in the drawing of the image 15 of the virtual object 11, the appearance of the image 15 of the virtual object 11 reflected in the real object 12 can be realized. It becomes possible to get closer to the appearance of other objects in the real space reflected in the object 12.
Therefore, the user can experience as if the virtual object 11 actually exists.

<Drawing example 5>
FIG. 13 is a diagram illustrating the influence of the difference in the shape of the real object 12 on the drawing of the image 15 of the virtual object 11. (A) is a drawing example of an image 15 of a virtual object 11 in a flat plate-shaped real object 12, and (B) is a drawing example of an image 15 of a virtual object 11 in a cylindrical real object 12.
Also in the case of FIG. 13, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 (see FIG. 1) is located on a normal line extending toward the front from the paper surface.
As shown in FIG. 13, when the real object 12 has a cylindrical shape (in the case of (B)), for example, the virtual object 11 is compared with the case where the real object 12 has a flat plate shape (in the case of (A)). The process of drawing the image 15 is performed so that the image 15 is stretched and distorted.
In this way, by adding the influence of the shape of the real object 12 and changing the image 15 of the virtual object 11 according to the shape of the real object 12, the appearance of the image 15 of the virtual object 11 reflected in the real object 12 Can be brought closer to the appearance of other objects in the real space reflected in the real object 12.
Therefore, the user can experience as if the virtual object 11 actually exists.

<Drawing example 6>
FIG. 14 is a diagram illustrating the effect of the difference in the angle at which the real object 12 is placed on the drawing of the image 15 of the virtual object 11. (A) is a drawing example of the image 15 of the virtual object 11 when the flat plate-shaped real object 12 is placed parallel to the vertical plane, and (B) is the drawing example of the flat plate-shaped real object 12 on the vertical plane. It is a drawing example of the image 15 of the virtual object 11 when it is placed diagonally.
Also in the case of FIG. 14, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 (see FIG. 1) is located on a normal line extending toward the front from the paper surface.
As shown in FIG. 14, when the real object 12 is tilted with respect to the vertical plane (in the case of (B)), the real object 12 is placed parallel to the vertical plane ((A)). In the case of), for example, the process of drawing the image 15 is performed so that the lower side of the virtual object 11 becomes larger near and the upper side of the virtual object 11 becomes far smaller.
In this way, by changing the image 15 of the virtual object 11 according to the angle at which the real object 12 is placed, the appearance of the image 15 of the virtual object 11 reflected on the real object 12 is reflected on the real object 12. It makes it possible to get closer to the appearance of other objects in the real space.
Therefore, the user can experience as if the virtual object 11 actually exists.

<Drawing example 7>
FIG. 15 is a diagram illustrating the effect of the difference in information regarding the polarization of the reflected light in the real object 12 on the drawing of the image 15 of the virtual object 11. (A) shows a case where the influence of the polarization of the reflected light in the real object 12 is not taken into consideration, and (B) shows the case where the influence of the polarization of the reflected light in the real object 12 is taken into consideration.
Also in the case of FIG. 15, the eyeball 5 (see FIG. 1) of the user wearing the terminal 1 (see FIG. 1) is located on a normal line extending toward the front from the paper surface.
In the example of (A), since the influence of the polarization of the reflected light on the real object 12 is not taken into consideration, the process of drawing the image 15 of the virtual object 11 on the surface of the real object 12 when the real object 12 has reflectivity. Is done.

Here, for example, if the light guide plate 2 (see FIG. 1) has a function of a polarizing filter or if a polarizing filter is provided in front of and behind the light guide plate 2, the external light B1 incident from the front of the light guide plate 2 can be specified. The light in the direction passes through the polarizing filter and is guided to the eyeball 5, but the light other than the specific direction is cut by the polarizing filter. That is, the reflected light in the real object 12 passes through the polarizing filter or is cut by the polarizing filter depending on the degree of polarization thereof.
When the reflected light in the real object 12 is cut by the polarizing filter, a process of not drawing the image 15 of the virtual object 11 on the surface of the real object 12 is performed as shown in (B).
In this way, the CPU 21 performs a process of drawing the image 15 of the virtual object 11 and a process of not drawing the image 15 depending on the degree of polarization of the reflected light in the real object 12 and the polarization characteristics (degree of polarization, etc.) of the polarization filter. Further, the image 15 of the virtual object 11 is changed by lowering the resolution or lightening the color depending on the degree of polarization of the reflected light in the real object 12 and the polarization characteristics of the polarization filter. .. For example, the larger the degree of polarization of the polarizing filter, the larger the amount of light cut by the polarizing filter, and the image 15 of the virtual object 11 is processed so as to be difficult to see.
Therefore, the user can experience as if the virtual object 11 actually exists.
It should be noted that the effect due to the difference in information regarding polarization may include the effect due to the difference in optical rotation. By reflecting the difference in optical rotation in the drawing, it is possible to express the difference in the appearance of optical isomers.

<Drawing example 8>
FIG. 16 is a diagram illustrating a case where total reflection occurs and a case where total reflection does not occur in the real object 12. (A) shows the positional relationship between the eyeball 5, the real object 12, and the virtual object 11 when total reflection does not occur, and (B) shows the positions of the eyeball 5, the real object 12, and the virtual object 11 when total reflection occurs. Show the relationship.
As shown in FIG. 5, with respect to the reflected image 13 of the virtual object 11 when the surface of the real object 12 is the plane of symmetry, a virtual straight line connecting the reflected image 13 of the virtual object 11 and the eyeball 5 and the reality. An image of the virtual object 11 is drawn at a position (range) where the object 12 intersects. Here, whether or not total reflection occurs depends on whether or not the incident angle of light exceeds the critical angle.

In the case of (A), since the incident angle does not exceed the critical angle (critical angle> incident angle), total reflection does not occur. In this case, the image 15 of the virtual object 11 is drawn on the real object 12 based on the reflectance of the real object 12 (furthermore, positive reflectance, diffuse reflectance, glossiness, etc., if necessary).
On the other hand, in the case of (B), since the incident angle exceeds the critical angle (critical angle <incident angle), total reflection occurs. In total reflection, the incident light is completely reflected without passing through the real object 12, so that the reflectance is 100%. The CPU 21 adds the influence of total reflection, and based on the reflectance of the real object 12 of 100% (furthermore, if necessary, the regular reflectance, the diffuse reflectance, the glossiness, etc.), the image 15 of the virtual object 11 is displayed. Draw on the real object 12. As a result, it is possible to clearly perceive the image 15 of the virtual object 11 in the case where total reflection occurs (in the case of (B)) than in the case where total reflection does not occur (in the case of (A)). be.
Therefore, the user can experience as if the virtual object 11 actually exists.

Here, as a case where total reflection occurs as in (B), for example, a case where the real object 12 is on the water surface and the virtual object 11 exists in the water is exemplified. More specifically, for example, when the user (eyeball 5) is below the water surface (real object 12) of the water stretched in the water tank (not shown) and the virtual object 11 is placed in the water. Total internal reflection can occur. In other words, when the eyeball 5 exists at a position lower than the water surface (real object 12) and the virtual object 11 is placed in the water, the incident angle is calculated based on the positional relationship between the eyeball 5 and the virtual object 11. Whether or not total reflection occurs is determined.
In addition, total internal reflection can occur even under conditions where a mirage occurs.

The critical angle is calculated based on the refractive index of the incident source substance and the refractive index of the traveling destination substance (here, the real object 12). The refractive index of the substance of the incident source and the refractive index of the real object 12 may be estimated (calculated) from the captured image in the same manner as the reflection information of the real object 12, or the non-volatileity of the RAM 23 is known as known information. It may be stored in the area. As a method for estimating the refractive index, for example, there are a method of comparing a plurality of images captured at a plurality of time points, a method of acquiring a refractive index corresponding to an object specified by artificial intelligence from a database, and the like.

<Drawing example 9>
FIG. 17 is a diagram illustrating a drawing example of an image 15 of a virtual object 11 when there are a plurality of real objects 12. (A) is a diagram showing the positional relationship between the real object 12 and the virtual object 11, and (B) is a diagram showing the mixed reality perceived by the user.
In the case of FIG. 17, the real object 12A is a frame-shaped member having no reflectivity, and the real object 12B is a member having a reflectivity. The real object 12A is, for example, a frame of a mirror, and the real object 12B is, for example, a glass plate of a mirror.

In the case of FIG. 17, if the conventional technique is used, the image 15 of the virtual object 11 is not drawn with respect to the real object 12B. Therefore, an unnatural state occurs in the landscape perceived by the user.
On the other hand, in the present embodiment, as shown in (A), the reflection information of the real object 12B is reflected in the reflection region 14 where the image 15 of the virtual object 11 is determined to be reflected in the real object 12B having reflectivity. The information of the virtual object 11 reflecting the above is associated. As a result, as shown in (B), the image 15 of the virtual object 11 is drawn with respect to the real object 12B. Further, since the real object 12A has no reflectivity, the image 15 of the virtual object 11 is not drawn.
By drawing the image 15 of the virtual object 11 in this way, the user wearing the terminal 1 can easily understand the relationship in which the virtual object 11 is located in front of the real object 12B having reflexivity.
As described above, the user can experience as if the virtual object 11 actually exists.

Further, although not shown as a drawing example, when the virtual object 11 has reflectivity, the CPU 21 draws an image of the virtual object 11 on the real object 12 and also draws an image of the real object 12 on the virtual object 11. do. Here, the image of the real object 12 in which the image of the virtual object 11 is reflected is drawn on the virtual object 11. Further, the image of the virtual object 11 in which the image of such a real object 12 is reflected is drawn on the real object 12. As described above, when the virtual object 11 has reflectivity, a phenomenon such as an infinite loop occurs in the depiction of the image. When the phenomenon of an infinite loop occurs, in the actual processing, for example, by limiting the number of times the reflected image is calculated, it is possible to prevent the calculation processing by the CPU 21 from becoming infinite.

<Embodiment 2>
In this embodiment, a case where a display device mounted on the head is used for the experience of mixed reality will be described.
FIG. 18 is a diagram illustrating a principle that a user who wears a display device 100 that displays an image obtained by synthesizing a virtual object with an image of the outside world captured in real time in the experience of mixed reality can experience the mixed reality. ..

FIG. 18 is shown with reference numerals corresponding to the portions corresponding to FIGS. 1 and 2.
The display device 100 synthesizes the image of the outside world captured by the cameras 24L and 24R and the image of the virtual object 11 drawn by the virtual object drawing unit 4 (and the image of the image of the virtual object 11) by the image synthesizing unit 101. The image is displayed on the display units 3L and 3R arranged in front of the user's eyeball 5.
The display device 100 here is an example of an information processing device as well as an example of an information processing system.
The hardware configuration of the display device 100 is the same as that of the glasses-type terminal 1 (see FIG. 2). Therefore, the description of the hardware configuration of the display device 100 will be omitted.

FIG. 19 is a diagram showing an example of the functional configuration of the display device 100.
FIG. 19 is shown with reference numerals corresponding to the portions corresponding to those in FIG.
The basic functional configuration of the display device 100 is the same as that of the glasses-type terminal 1 (see FIG. 2). The functional configuration peculiar to the display device 100 is the image composition unit 101.
The image synthesizing unit 101 has a function of synthesizing two images so that the image drawn by the virtual object drawing unit 4 and the image of the outside world captured by the cameras 24L and 24R match.
For example, the image synthesizing unit 101 collates the three-dimensional model stored as the real space virtualization information 42 with the image of the outside world captured by the cameras 24L and 24R, and collates the image of the virtual object 11 (and the virtual object 11). Determine the area to synthesize the image of the image).
As described above, the method in which the present embodiment perceives the mixed reality is different from the first embodiment, but the point that the reality of the mixed reality perceived by the user is higher than that of the conventional technique is the embodiment. Same as 1.

<Other embodiments>
Although the embodiments of the present invention have been described above, the technical scope of the present invention is not limited to the scope described in the above-described embodiments. It is clear from the description of the claims that the above-mentioned embodiments with various modifications or improvements are also included in the technical scope of the present invention.
For example, in the above-described embodiment, the display units 3L and 3R for both the left and right eyes are used, but one display unit may be used. For example, in the case of a glasses-type terminal 1 (see FIG. 1), one display unit may be arranged in front of either the left or right side. Further, for example, in the case of the display device 100 (see FIG. 18), one display unit may be arranged in front of both eyes.
Further, in the above-described embodiment, the virtual object drawing unit 4 is realized as one of the functions of the glasses-type terminal 1 (see FIG. 1) and the display device 100 (see FIG. 18), but the external network (for example, see). The function of the virtual object drawing unit 4 may be executed in an information processing device such as a server connected to the cloud network). The server on the external network that executes the functions of the glasses-type terminal 1 and the virtual object drawing unit 4 here is an example of an information processing system.
Further, in the above-described embodiment, the function of the virtual object drawing unit 4 is realized by using the CPU 21 which is a general-purpose arithmetic unit, but the GPU (GPU) which is an arithmetic unit specialized in image processing in real time. It may be realized by using a Graphics Processing Unit).

1 ... Glass-type terminal, 2 ... Light guide plate, 3, 3L, 3R ... Display unit, 4 ... Virtual object drawing unit, 11 ... Virtual object, 12 ... Real object, 13 ... Reflected image, 14 ... Reflection area, 15 ... Image, 31 ... Real space information acquisition unit, 32 ... Real object reflectivity estimation unit, 33 ... Real object reflection information acquisition unit, 34 ... Real object reflection area determination unit, 41 ... Real space information, 42 ... Real space virtualization information , 43 ... Virtual object information, 44 ... Virtual object image information, 100 ... Display device, 101 ... Image synthesizer, B1 ... External light, B2 ... Display light

Claims (3)

An estimation means for estimating the reflectivity information of a real object from an image of a real object,
Estimated reflectivity information when there is a virtual object within the range where the image of reflection is reflected on the real object due to the positional relationship between the real object estimated to have reflectivity and the user. And a generation means to generate an image of the virtual object based on
It has a drawing means for drawing an image of the virtual object with respect to the real object.
In the estimation means, the light from the real light source is reflected by the real object defined as the estimation target from the relationship between the position of the real light source and the inclination of the surface of the real object defined as the estimation target. If there is a bright part at the position specified as the position where the reflected reflected light irradiates, and the position where the light from the actual light source directly irradiates, the reality defined as the estimation target. An information processing system characterized in that an object is presumed to have reflectivity . Claim 1 is characterized in that the estimation means estimates that the greater the difference between the brightness of the specified position and the brightness around the specified position, the greater the reflectance of the actual object. The information processing system described. Computer,
An estimation means for estimating the reflectivity information of a real object from an image of a real object,
Estimated reflectivity information when there is a virtual object within the range where the image of reflection is reflected on the real object due to the positional relationship between the real object estimated to have reflectivity and the user. And a generation means to generate an image of the virtual object based on
It functions as a drawing means for drawing an image of the virtual object on the real object.
In the estimation means, the light from the real light source is reflected by the real object defined as the estimation target from the relationship between the position of the real light source and the inclination of the surface of the real object defined as the estimation target. If there is a bright part at the position specified as the position where the reflected reflected light irradiates, and the position where the light from the actual light source directly irradiates, the reality defined as the estimation target. A program characterized by presuming that an object is reflective .

Images