JP4744536B2 - Information presentation device - Google Patents

Description

  The present invention relates to an information presentation method and apparatus used in the field of virtual reality (VR) or computer visualization (Computer Visualization), and in particular, a virtual object without causing any disturbance in the shielding relationship with a real object. The present invention relates to a method and apparatus used for interactive presentation.

  In the field of artificial reality in recent years, research on mixed engineering and other researches that seek to apply its engineering application by actively merging physical space and information space have become active. However, many of these studies use a head-mounted display (HMD) or a normal projector as a visual presentation device, which causes a flaw in the shielding relationship between a real object and a virtual object. The feeling may be significantly impaired.

Furthermore, in addition to visual presentation, for example, as shown in Japanese Patent Application No. 6-102187, when tactile presentation including interaction with a virtual object is to be performed simultaneously, the tactile presentation device is a visual presentation device. It interferes visually and physically. This is a major problem when building a system that presents visual and tactile sensations simultaneously.
[Fusion of reality] We believe that society will become more highly computerized in the future by seamlessly integrating the physical space in which people live and the information space composed of computers and communication networks. It is done. To realize this, concepts such as "wearable-computing" and "ubiquitous-computing" have been proposed.

  Here, we examine the significance of seamless “fusion”. In fields such as psychology, binocular fusion and other things that have a close relationship with each other and give a new perceptual representation that cannot be born alone for some phenomenon or object The process is called “fusion”.

  In the engineering field as well, research on sensor fusion has been conducted to obtain a new sensing function that cannot be obtained from a single sensor by uniformly processing information obtained from a plurality of sensors.

Similarly, not only simply mixing physical space where humans exist and computer information space, or visual space and tactile space, but also seamlessly merging them to obtain effects that cannot be born alone. Can be expected.
[Fusion of physical space and information space] As the information communication infrastructure is improved, it is considered that on-demand type or bidirectional type communication will become mainstream instead of the conventional broadcast type information distribution. And as the application becomes more interactive, personalization of information advances, and it is considered that personal reality fusion for personal information becomes an essential issue.

  Therefore, as long as it is a personal interface, it is necessary to realize reality fusion compactly at a low cost.

  However, in the conventional display, the device itself occupies a large weight and volume and is not suitable for personal use.

  On the other hand, the HMD used in the field of artificial reality in recent years is suitable for generating a personal information space, but the physical space is shielded by the display itself, and the work in the physical space is extremely difficult. Have difficulty. Naturally, it is impossible to visually recognize one's body image except as a metaphor by CG (Computer Graphics), and the self-projection property, which is an important factor of reality, is low.

There has also been proposed a see-through type HMD using a half mirror or a head-mounted camera and attempting to superimpose and present an information space image on a physical space image. According to this, since the video (image) is synthesized in the vicinity of the observer's eyes or in the computer, the work space part is extremely disadvantageous for the work such as the shielding relationship between the presented objects and the mismatch of the focal position. Problems arise in principle. The problem regarding the mismatch of the shielding relationship is exactly the same in the immersive projection display such as CAVE (Non-Patent Document 1 below) or CABlN (Non-Patent Document 2 below) that has been frequently used in recent years.
[Visual and tactile fusion presentation] It has been reported that the presence of not only visual information but also tactile information improves the sense of reality and workability.

However, when trying to present tactile information and visual information at the same time, the visual presentation device and the tactile presentation device interfere optically and geometrically. This is exactly the same problem as the fusion part of the physical space and the information space, and is nothing but the interface presentation part exists in the physical space. Research on WYSIWYF display (Non-Patent Document 3 below), PDDM (Non-Patent Document 4 below), and haptic screen (Non-Patent Document 5 below) has been conducted to date. In this case, mounting is performed while ignoring the problem of the shielding relation, or mounting by optical superposition using a half mirror.
Carolina Cruz-Neira et al. "Surround-Screen Projection-Based Virtual Reality: The Design and Inplementation of the CAVE" COMPUTER GRAPHICS Proceedings, Annual Conference Series, pp. 135-142, 1993 Michitaka Hirose et al. "Development of Immersive Multi-Display (CABlN)" Proceedings of the 2nd Annual Conference of the Virtual Reality Society of Japan, pp, 46-53, 1996 Y. Yokokohji et al. "What You canSee is What You canFeel -Development of a virtual / Haptic Interface to Virtual Environment" In Proc., VRAIS '96, pp. 46-53, 1996 Haruma Noma et al. “Haptic Interface for Object Manipulation Using a Small Display” Proceedings of the 11th Human Interface Symposium, pp. 491-496, 1995 Hiroo Iwata et al. "Haptic Screen" Proceedings of the 1st Annual Conference of the Virtual Reality Society of Japan, pp. 7-10, 1996

[Realistic Fusion Configuration Theory] Here, many of the problems of the conventional system are that the physical space and the information space are synthesized at a point other than the point where both the physical space and the information space are to interact. Presumed to be due. In other words, it is extremely difficult for the current technology to cover all the fusion of reality from a short distance to a long distance with a single display, and it is an immediate problem to use devices of various distances and sizes depending on the application. Think of it as a solution.

  The inventor of the present application made a proposal regarding an object-oriented display as a display suitable for observing and manipulating a virtual object at hand, in view of this problem (Japanese Patent Application No. 10-172722). In this application, the idea is further expanded to show that “near objects are near displays and distant objects (scenery) are distant displays” as if they were written on the stage. It aims to fuse reality.

  This method is based on a 3D virtual space generation method based on the writing method in the field of Image Based Rendering (Nobuyuki Ozaki et al. "Tele-Gigence by Virtual Viewpoint (Part 3)-Landscape Plane) Virtual Space Configuration Method "Proceedings of the 35th Annual Conference of the Society of Instrument and Control Engineers, pp. 209-210, 1996, Michitaka Hirose et al." Generation of a three-dimensional virtual space using two-dimensional real images "Society of Instrument and Control Engineers Human Interface N & R, Vol 11, No. 2, pp.209-214, 1997).

  However, in order to appropriately combine reality in a wide range, it is necessary to construct a so-called ubiquitous display, in which a display is ubiquitous in a wide space according to a presentation range. However, the conventional display is not practical because the spatial and monetary costs become enormous.

  The conventional display is effective for broadcast-type information presentation that continues to present unidirectional and omnidirectional information to a space that assumes a large number of observers. However, when trying to use it as a personal information presentation device for interactive use, that is, when trying to use it for the purpose of being observed by a specific observer, the conventional display is omnidirectional. Information is also emitted in the direction where there is no observer, which becomes useless. In addition, there is a problem that it is difficult to protect the privacy of personal information because information enters even in the eyes of non-target people.

  In other words, an interactive video information presentation device requires a device that can be observed by a person, that is, a device that should be called an “observee”, rather than a “display” that displays a video to a person.

  As an “observer”, it is only necessary to construct a system that presents an image only in a range that can be seen by an observer. For example, it is necessary to present an image in a portion that is shielded by an object or that does not enter the field of view, such as the back. Absent.

  In order to realize the function of the observable information presentation device “observation”, it is possible to form a projection volume so as to coincide with the viewing volume that is the human viewing range. Such a video presentation system (device) is considered to be promising.

  As a system that performs the above functions, a system that measures the position of the observer and presents the presented video only to the observer using an optical system such as a lens so as to follow the observer, and the direction in which the light is incident A system for projecting an image from a viewpoint conjugate with an observer on a retroreflective screen having the property of reflecting directly.

  However, the former observer-following system requires a cost for the presentation device, and is difficult to be used for simultaneous ubiquitous applications. On the other hand, the latter projection-type video presentation system can easily generate a large amount of lightweight presentation planes by consolidating the spatial and monetary costs for video presentation on the projector side. It has the feature. In addition, by presenting as much information as is necessary for the observer as much as necessary, there is also a feature that leads to effective use of the limited bandwidth of the communication infrastructure. Furthermore, since only the observer can see the video, it is possible to protect the privacy of the observer.

  Further, in a situation where one or more displays are required for one person, it can be said that a system configuration with better cost performance is obtained by adding an information generation function to the human side rather than the space side.

  In the former observer-following system, the problem of the depth of focus due to the change in the screen position, the problem of the amount of light, and the like occur. A “display” that shows images to people is usually aimed at presenting information in a wider range, so widening the field of view is one of the points of research.

  The conventional screen is no exception, and it can be said that the target device is to diffuse the incident light using glass beads or the like. In other words, a screen used for a “display” typified by a movie theater essentially has the property that as the distance increases, the amount of light decreases in inverse proportion to the square.

  Therefore, it is considered that a function necessary as a screen for realizing the “observation” is not to diffuse incident light but to recursively reflect in the direction of the observer.

  In addition, “observation” makes it possible to present completely unique video information to each person, so even if it is physically one device, it is completely different for each person who observes it. Independent information can be presented, and the space can be used more effectively.

  The inventor previously proposed an image display method for realizing “observation” as Japanese Patent Application No. 10-172722, but has the following problems.

  That is, in the virtual space, when presenting sensory information other than visual information, particularly tactile information, to the observer, it is necessary to install a tactile sense presenting device in front of the observer. In addition, a human body state measurement sensor such as a three-dimensional position sensor may exist between the observer's eyes and the presented image.

  If these objects that are not supposed to be visible in the virtual space are seen by the observer, the reality of the virtual environment will be impaired. In particular, when an object that should not be seen is seen in front of the presented video, visual obstruction is generated by the object, and a blind spot is formed in the presented video, so that the reality is significantly impaired.

  A fatal example of this problem is when the depth relationship between a virtual object and a real object is reversed in the image, in which case a transparent interpretation that allows the object to show through occurs and the realism is significantly impaired. End up.

  For example, as shown in FIG. 5 (A), the virtual ball BL1 should be visible in front of the real human hand HD, but as shown in FIG. 5 (B), the ball BL1 and the hand When the HD context is reversed, the shielding relationship becomes inconsistent and the image looks very unnatural.

  Therefore, among the real objects that exist between the viewer and the presented image, the objects that are not originally visible cannot be seen, and the objects that should be visible are visible. It is necessary to prevent this from occurring.

  The present invention has been made in view of the above-described problems. An information presentation method and apparatus capable of maintaining a correct shielding relationship between a real object and a virtual object and presenting a virtual space with a high sense of reality. The purpose is to provide.

In the apparatus according to the present invention, all or a part of the real object JT placed between the background object 11B, which is an actually existing object, and the observer BS is optically hidden using the image of the background object. Thus, an apparatus for presenting information in a virtual space, which is a projector 14B disposed at a position optically conjugate with the position of the eye of the observer BS to project an image, and an image of the background object 11B Photographing means 12 for obtaining the image, and the position and size of the image obtained by the photographing means 12 are adjusted so as to coincide with the actual image of the background object as viewed from the observer, and The image processing apparatus 13B outputs the image adjusted in this way to the projector 14B so as to project the image onto the retroreflecting material disposed on the portion to be hidden of the real object JT. Further, the photographing unit is configured to acquire the image of the background object from the same position as the observer's viewpoint or a position optically conjugate with the observer's viewpoint.

  According to the present invention, a correct occlusion relationship can be maintained between a real object and a virtual object, and a virtual space with high realism can be presented.

Further, according to the first aspect of the present invention, the object can be made transparent even when the background is an actual object.

  FIG. 2 is a diagram showing the configuration of the information presentation apparatus 1A according to the present invention, FIG. 3 is a diagram functionally showing the configuration of the information presentation apparatus 1A, and FIG. 4 is based on the retroreflective material RU used for the screen 11A. It is a figure for demonstrating a retroreflection function.

  2 and 3, the information presentation device 1A includes a screen 11A, a processing device 13A, projectors 14L and 14R, a half mirror 15A, a tactile sense presentation device 16, a head position measurement device 17A, and a background screen 19. The projectors 14L and 14R are for the left eye and the right eye, respectively, are monoculars with a small aperture diameter, and have the same structure. Therefore, the projector 14L or 14R may be referred to as “projector 14A” in some cases.

  The screen 11A projects an image (video) HG projected by the projector 14A, and visually presents an object (virtual object) that is the content of the image HG to the observer BS. The shape of the screen 11A can be an arbitrary shape, but is preferably the same shape as the object presented by the image HG or a shape close thereto. However, since it is inefficient to manufacture the screen 11A according to the shape of each target object, it is not particular about it, and for example, it is assumed to have an arbitrary shape corresponding to the target object such as a flat surface, a curved surface, or a closed surface. Can do. The closed curved surface can be, for example, a spherical shape, an egg shape, a polyhedron, or a donut shape.

  The material used for the main body of the screen 11A is preferably a material that is lightweight and can be easily processed into an arbitrary shape, such as polystyrene foam. A retroreflective material using glass beads or the like is applied to the surface of the screen 11A to form a screen having a retroreflective function (retroreflective screen).

  As shown in FIG. 4B, on the surface of the normal material MU, the reflected light LR is emitted in all directions without much depending on the incident angle of the incident light LC. However, when a retroreflective screen is formed using the retroreflecting material RU, most of the reflected light LR is emitted in the direction in which the incident light LC is incident, as shown in FIG. Therefore, the image projected on the surface of the screen 11 </ b> A has high luminance in the viewing direction of the observer BS. That is, the observer BS can observe the image projected on the screen 11A with extremely high luminance.

  Thus, the retroreflecting material RU is a material that reflects with a strong directivity in the direction of the light source, and is realized using a corner cube array or glass beads.

  The processing device 13A generates and sends the image information DL and DR to the left and right projectors 14L and 14R in the virtual environment video generation unit 13a. For example, for a volleyball-sized egg-shaped screen 11A, an image of an object having an egg shape as a whole is generated. For example, it is an image of a small animal such as a cat, a frog or a squirrel. The generated image is a three-dimensional image having three-dimensional information. Therefore, it is possible to output image information DL and DR that are two-dimensional images from an arbitrary viewpoint.

  That is, the processing device 13A generates the corrected image information DL and DR so that an appropriate image is displayed on the screen 11A according to the viewpoint of the observer BS with respect to the screen 11A. That is, the content, shape, position, and the like of the images projected from the projectors 14L and 14R are changed following the changes in the positions of the screen 11A and the observer BS. For example, if an image of a certain object viewed from the front is shown on the screen 11A, and the observer BS rotates the screen 11A 90 degrees around the vertical line, the object on the screen 11A The content of the image is changed so that the image of the object seen from the side is reflected.

  Note that the image information DL and DR output to the projectors 14L and 14R have parallax so that the image projected on the screen 11A can be stereoscopically viewed by the observer BS with the left and right eyes.

  The processing device 13A calculates a reaction force applied to the operation unit 161 of the tactile sense presentation device 16 and gives a command to the control device 160 in the reaction force calculation unit 13b.

  A microcomputer, personal computer, workstation, or the like is used as the processing device 13A. In the processing device 13A, in order to generate the above-described image information DL and DR, data and programs for computer graphics, rendering software, and other control programs are mounted. Further, the virtual space information storage device DB1 that stores a database such as an object shape model is referred to. Each of the virtual environment video generation unit 13a and the reaction force calculation unit 13b is realized as one of functions based on the execution of these programs.

  In order for the projector 14A to project an image having a deep focal depth, it is preferable that the aperture diameter be as small as possible, for example, the diameter of the aperture is 5 cm or less, 2 cm or less, and further 1 mm or less. By narrowing down the aperture diameter, a pinhole optical system is formed and the depth of focus is increased.

  The projector 14A is provided at a position optically conjugate with the positions of the left and right eyes of the observer BS. In that case, it is preferable to adopt a head-mounted type that can be integrally attached to the head of the observer BS together with the half mirror 15A.

  In order to make the projector 14A a head-mounted type, for example, the projector 14A, the half mirror 15A, and the head position measuring device 17A are attached to a cap or a helmet that the observer BS can wear on the head. Just keep it. Or you may comprise so that it may mount | wear on a head or a face with a band etc. like a head mounted display.

  The half mirror 15A reflects the light projected from the projector 14A, but transmits the light of the image displayed on the screen 11A. The projector 14A is provided at a position optically conjugate with the positions of the left and right eyes of the observer BS by the half mirror 15A. Thereby, the observer BS can observe the object presented on the screen 11A.

  The tactile sense presentation device 16 is a device for presenting a tactile sense to the finger BSF of the observer BS. That is, the tactile sense presentation device 16 is provided with an operation unit 161 that performs an operation by inserting the finger BSF, and is movably supported by the arms 162 to 164 and the actuators 165 and 166. A screen 11A is attached to the operation unit 161, and the screen 11A moves together with the operation unit 161. That is, the position and orientation of the screen 11 </ b> A are determined by the position and orientation of the operation unit 161.

  The actuators 165 and 166 are used to move and drive the operation unit 161 or to generate a reaction force on the operation unit 161. Angle sensors SE1 and SE2 that detect the angles of the arms 162 to 164 are provided at the positions of the actuators 165 and 166, respectively. Based on the output signals of the angle sensors SE1, SE2, the position and orientation of the screen 11A in the three-dimensional space are measured in real time. In this specification, both the position and the posture may be collectively referred to as “position”. The control unit 160 controls the entire operation unit 161 based on the output signals of the angle sensors SE1 and SE2 and the command from the processing device 13A.

  The operation unit 161 of the tactile sense presentation device 16 is usually an object in front of the screen 11A, and thus can be seen by the observer BS, but is an object that should not be seen when the virtual space is presented. Therefore, a retroreflecting material 18A is applied to the operation unit 161 in order to optically hide the operation unit 161 from projection by the projector 14A. In addition, as for other parts of the tactile sense presentation device 16, a retroreflecting material 18 </ b> A is applied to a part that may be seen from the observer BS.

  For example, in order to present a sense of weight as the tactile sensation given to the observer BS, the screen 11A may be pulled downward with a force corresponding to the virtual weight of the object. Examples of tactile sensations other than weight sensation include a sense of recoil when something is fired, a viscosity sensation as if the object was moved in a viscous medium such as water or other liquids, and the object was shaken by the wind Sensation, or the force of bouncing back when a virtual ball is thrown on a virtual wall. The tactile sense presentation device 16 allows the observer BS to experience the sense of force of the object in the image projected on the screen 11A.

  The head position measurement device 17A is attached to the head of the observer BS, and measures the position of the head, that is, the position of the eyes and the direction of the line of sight. A measurement signal from the head position measurement device 17A is fed back to the processing device 13A. In the processing device 13A, the relative position of these is calculated from the position of the screen 11A and the position of the observer, and the viewpoint of the observer BS with respect to the object is obtained.

  The background screen 19 is a retroreflective screen and is provided behind the screen 11A. On the background screen 19, the background of the object to be projected on the screen 11A is projected. The background screen 19 may be a wall surface or a floor surface to which a retroreflective material is applied, or may be a sheet having a retroreflective function.

  Next, the operation of the information presentation device 1A will be described. Here, it is assumed that the initial positions of the tactile sense presentation device 16 and the observer BS have already been aligned with the virtual space.

  Based on the viewpoint position and viewpoint information obtained by the head position measurement device 17A and the virtual space information, the processing device 13A generates an image of the virtual space at the viewpoint and line of sight of the observer BS. The generated video is projected on the screen 11A by the projectors 14L and 14R. The image projected on the screen 11A is observed by the observer BS. The observer BS can observe the images projected by the projectors 14L and 14R separately with the left and right eyes. The observer BS can simultaneously see the virtual object by projection and the real object existing there with the naked eye. In this way, the virtual space is presented to the observer BS along with the real space.

  On the other hand, the position of the tip of the finger BSF that is in contact with the operation unit 161 of the tactile sense presentation device 16 is calculated based on the output signals from the angle sensors SE1 and SE2. The actuators 165 and 166 are driven and controlled to generate a reaction force so that the finger BSF of the observer BS does not enter the inside of the virtual object (video). As a result, the observer BS is presented with a tactile sensation as if there was a real object.

  Since the retroreflecting material 18A is also applied to the tactile presentation device 16, the image projected from the projector 14A is also reflected by the tactile presentation device 16 and enters the eyes of the observer BS. Since the image reflected by the tactile sense presentation device 16 is reflected in the eyes of the observer BS in the same manner as the image reflected by the screen 11A, only the image projected from the projector 14A is presented to the observer BS. Due to the nature of the retroreflecting material 18A, the rays other than the light rays in the line of sight of the observer BS do not return to the eye, so the tactile sense presentation device 16 cannot be seen from the observer BS. That is, the tactile sense presentation device 16 is visually transparent to the observer BS.

  If the tactile presentation device 16 is visible from the observer BS, the tactile presentation device 16 causes visual shielding, and a blind spot is formed in the image presented by the projector 14A. End up.

  For example, as shown in FIG. 6A, since the retroreflecting material 18Z is applied on the surface of the tactile presentation device 16Z, the tactile presentation device 16Z is visually transparent and visible to the observer BS. Absent. The virtual object VT1 is shielded only by the hand HD. If the tactile presentation device 16X to which the retroreflective material is not applied is used, the virtual object VT1 is shielded by the tactile presentation device 16X as shown in FIG. The feeling is significantly impaired.

  In particular, when presenting a tactile sensation as a sense other than vision in a virtual space as in the present embodiment, a device such as a tactile sensation presentation apparatus needs to be installed in front of the user. In the virtual space presentation method using the naked eye or stereo vision goggles using a rear projection type screen, such as conventional CAVE and CABIN, the device can be seen inside the virtual environment, and the virtual environment can be sensed. Significantly damaged. Further, in the virtual space presentation method using HMD or the like, the device cannot be seen, but the hand of oneself is often drawn as CG on the HMD, which is also not realistic.

  As in this embodiment, by applying the retroreflecting material 18A to the tactile presentation device 16 and projecting an image from a position conjugate with the eye position of the observer BS, the tactile presentation device 16 is also part of the screen. It functions and can be viewed visually as if the tactile presentation device 16 was erased from the virtual space. In addition, since the shadow portion by the tactile sense presentation device 16 becomes a blind spot even in the field of view of the observer BS, no shadow is generated.

  And since the observer's BS hand etc. do not have a retroreflection function, the brightness | luminance is remarkably low compared with the image | video projected on the screen 11A. Therefore, no reflection occurs in one's hand or the like, and the hand can be viewed as a hand.

  As described above, by selectively using the retroreflecting material selectively with respect to the real object, it is possible to camouflage the unnecessary real object to make it visually “transparent”. That is, an “optical camouflage” is realized. As a result, visual and tactile sensations are integrated, a correct shielding relationship is maintained between the real object and the virtual object, and a virtual space with a high sense of reality can be presented.

  As described above, the information presentation apparatus 1A according to the present embodiment satisfies the condition of “show only visible objects” required for the information presentation device “observer” to be viewed.

  As described above, the retroreflective material may be applied directly to a real object such as the tactile sense presentation device 16 or may be applied to paper or cloth to have a retroreflective function. You may make a shielding screen and cover the real object you want to hide. Further, not only the tactile sense presentation device 16 but also any real object such as other sensory presentation devices, other devices, wiring, or furniture can be hidden from the virtual space.

Other advantages of the information presentation device 1A according to the present embodiment are as follows.
(1) By using the retroreflective screen as the screen 11A, it is possible to obtain a brightness that is sufficiently observable under indoor light. Further, since all objects that can be coated with a retroreflecting material can be used as a screen, it is possible to realize a display that is very lightweight and has a free shape.
(2) Since the image is projected from a position optically conjugate with the eyes of the observer BS, the image distortion that causes trouble in the shape of the screen 11A does not occur.
(3) By narrowing the aperture diameter of the image projection unit of the projector 14A as much as the amount of light permits, the depth of focus increases, and an image is formed on the screen 11A having an arbitrary shape and position without causing a focus shift in a wide range. It is possible.
(4) Distance dependency of screen luminance is reduced.
(5) Since the binocular images observed by the observer BS are spatially separated, autostereoscopic viewing is possible. By performing the stereoscopic viewing, dirt on the surfaces of the screen 11A, the background screen 19, and the tactile sense presentation device 16, or a joint between the screens becomes inconspicuous.
(6) The projector 14A is integrated with an optical member such as the half mirror 15A so that the projector BS can be mounted on the head of the observer BS. It is possible to move freely within.
(7) Since the configuration is relatively simple, it can be realized at a low cost.

  In the information presentation apparatus 1A of the above-described embodiment, the example in which the screen 11A is arranged has been described, but the screen 11A may be omitted. In that case, images with parallax for the left eye and right eye are projected on the background screen 19, and the surface of the virtual object is displayed at the position (distance) of the screen 11A by the projected left and right images. And it is comprised so that reaction force may be shown by the tactile sense presentation apparatus 16 according to the surface shape of the virtual object. Although the tactile presentation device 16 is disposed in front of the background screen 19, even when the screen 11 </ b> A is not provided, the tactile presentation device 16 is applied from the observer BS by applying a retroreflecting material to the tactile presentation device 16. Never see. Further, the background screen 19 can be omitted instead of omitting the screen 11A.

  By the way, in the embodiment described above, the background of the object that realizes transparency is a projected image.

  That is, in the information presentation apparatus 1 shown in FIG. 1, the background of the real object JT is an image projected from the projector 14 onto the screen 11. By projecting an image that does not contradict or matches the image projected on the screen 11 to the actual object JT, the actual object JT is buried in the image projected on the screen 11 to be transparent.

  Also in the information presentation apparatus 1A shown in FIG. 2, the background of the operation unit 161 of the tactile presentation apparatus 16 that is a real object is an image projected on the screen 11A. Further, when viewing a part of the arms 162 to 164 and the control unit 160 of the tactile sense presentation device 16, the background is an image projected on the background screen 19. Therefore, an image that matches the image projected on the screen 11A is also projected on the operation unit 161, and an image that matches the image projected on the background screen 19 is also projected on a part of the arms 162 to 164 and the control unit 160. By doing so, they are buried in the background image and made transparent.

  However, in the embodiment described below, the background is not a projected image but a real object. That is, even when the background is an actual object, the object can be made transparent by configuring as in the following embodiment.

  FIG. 7 is a diagram showing a configuration of an information presentation device 1B according to another embodiment of the present invention, and FIG. 8 is a flowchart showing a flow of processing or operation in the information presentation device 1B.

  In FIG. 7, the background object 11B is arranged as the background of the real object JT. In this embodiment, the background object 11B is a wall made of brick, but a wall other than a brick, window, room, house, building, garden, other building, book, bookshelf, painting, creation, person Various objects that actually exist, such as animals, landscapes, etc., can be used.

  When these objects are used as the background object 11B, the background object 11B is photographed using the camera 12, and an image of the background object 11B is acquired in advance. Alternatively, the background object 11B is photographed simultaneously with the presentation of the virtual space.

  The image of the background object 11B obtained by shooting is projected on the retroreflecting material 18B for shielding the real object JT. The position and size of the image projected on the retroreflecting material 18B are adjusted by the processing device 13B so as to match the image of the background object 11B that should be seen from the observer BS. Therefore, the observer BS sees the image of the background object 11B projected on the retroreflecting material 18B instead of the real object JT. As a result, the real object JT is made transparent.

  When the background object 11B is photographed by the camera 12, photographing is performed from the same position as the viewpoint of the observer BS or a position optically conjugate with the viewpoint of the observer BS. By doing so, it is possible to superimpose the actual background object 11B and its image so as to coincide with each other without performing any pre-processing on the image obtained by photographing.

  However, photographing from such a position may be difficult or impossible. In addition, the observer BS may move and its viewpoint may move. In such a case, an image obtained by photographing is corrected or deformed using a known image-based rendering method. For example, three-dimensional measurement of the background object 11B is performed, and the image is corrected or deformed according to the relative positional relationship with the viewpoint of the observer BS. Regarding image-based rendering, in addition to the above, it is described in "Generation of three-dimensional virtual space using two-dimensional live-action images" (Hirose, Miyata, Tanikawa, Shingaku Giho, MVE96-18 (1996)). Can be referred to.

  In the flowchart of FIG. 8, a background image is captured by photographing a background object (# 11). If the shooting position of the camera 12 and the viewpoint of the observer BS are optically conjugate (Yes in # 12), the image is enlarged or translated as necessary (# 13) and processed. The image is projected from the projector 14B (# 14).

  When the shooting position of the camera 12 and the viewpoint of the observer BS are not optically conjugate (No in # 12), the shooting position of the camera 12, the viewpoint position of the observer BS, the position of the background object 11B, and the background object Each data of the shape of 11B is acquired (# 15). Based on these data, image-based rendering processing such as affine conversion is performed on the captured image to convert it into an image (background video) from the viewpoint of the observer BS (# 16) and conversion The projected image is projected from the projector 14B (# 14).

  FIG. 9 is a diagram illustrating an example in which information is presented using the information presentation device 1B.

  In FIG. 9, a bookshelf 11Ba is used as a background object. A large number of books are stored in the bookshelf 11Ba according to normal usage, and two books BK1 and BK2 are placed in front of the bookshelf 11Ba. The observer BS is in front of the front of the bookshelf 11Ba (front side in the direction perpendicular to the paper surface of FIG. 9), and the hand HD has a large loupe-like object JTa.

  The object JTa is entirely opaque, and a part of each of the two books BK1 and BK2 is hidden by the object JTa. The object JTa is coated with a retroreflecting material 18Ba on a circular portion corresponding to the lens. An image of the bookshelf 11Ba is taken using the camera 12, and an image VF1 subjected to processing such as alignment is projected onto the retroreflecting material 18Ba of the object JTa. The projected image VF1 shows the books BK1 and BK2 that should have been visible if the part of the retroreflecting material 18Ba is transparent.

  That is, in FIG. 9, the part visible on the outside of the real object JTa is the bookshelf 11Ba itself, and the part visible on the retroreflecting material 18Ba inside the real object JTa is the image VF1 projected by the projector 14B. When the observer BS changes the viewpoint or the position of the hand HD or the real object JTa, an image VF1 corresponding to the change is projected onto the retroreflecting material 18Ba.

  In FIG. 9, the image VF1 projected on the retroreflective member 18Ba is the same size as the images of the books BK1 and BK2 on the bookshelf 11Ba as the background, but the image is enlarged by performing an image enlargement process or the like. VF1 can be enlarged more than the image of the real object JTa. Further, the enlargement ratio can be changed according to the distance between the books BK1 and BK2 and the real object JTa. This makes it feel as if the real object JTa is a real loupe.

  That is, the transparency of the lens-corresponding portion (recursive reflector 18Ba) of the real object JTa is realized with the real object of the bookshelf 11Ba as the background.

  In the information presentation apparatus 1A shown in FIG. 2, the background object 11B is used instead of the background screen 19 and the image is taken and projected onto the parts such as the arms 162 to 164 and the control unit 160, thereby 164 and the like can be made transparent.

  In the above-described embodiment, the processing devices 13, 13A, 13B, the projectors 14, 14L, 14R, 14B, the tactile sense presentation devices 16, 16Z, and the information presentation devices 1, 1A, 1B, or the configuration, shape, and configuration of each part, Dimensions, materials, processing details, and the like can be changed as appropriate in accordance with the spirit of the present invention.

It is a figure for demonstrating the principle of this invention. It is a figure which shows arrangement | positioning of the structure of the information presentation apparatus which concerns on this invention. It is a figure which shows the structure of an information presentation apparatus functionally. It is a figure for demonstrating the retroreflection function by the retroreflection material used for a screen. It is a figure which shows the example of how the correctness of a shielding relationship appears. It is a figure which shows the example of how the correctness of a shielding relationship appears. It is a figure which shows the structure of the information presentation apparatus of other embodiment which concerns on this invention. It is a flowchart which shows the flow of operation | movement in the information presentation apparatus shown in FIG. It is a figure which shows the example which performed information presentation using the information presentation apparatus shown in FIG.

Explanation of symbols

1, 1A, 1B Information presentation device 11, 11A Screen 11B, 11Ba Background object 13 Image generation device 13A, 13B Processing device (image generation device)
13a Virtual environment video generation unit (image generation device)
14, 14A, 14B Projector 15, 15A, 15B Half mirror (optical member)
16, 16Z Tactile presentation device (real object)
18 Shielding screen (shielding material)
RU, 18A, 18B, 18Ba, 18Z Retroreflective material (shielding material)
BS observer JT, JTa real object

Claims (1)

Presents virtual space information by optically hiding all or part of the real object placed between the background object and the observer, which is an actual object , using an image that matches the background object. A device that performs
A projector disposed at a position optically conjugate with the position of the observer's eye to project an image;
Photographing means for obtaining an image of the background object;
The position and / or size of the image acquired by the photographing means is made to coincide with the image of the background object that should be seen when viewed from the observer by an enlargement or translation process of the image, and A processing device that outputs to the projector such that the matched images are projected toward a retroreflecting material disposed on a portion to be hidden of the real object;
Have
The photographing means is configured to acquire an image of the background object from the same position as the observer's viewpoint or a position optically conjugate with the observer's viewpoint ,
The projector projects at least a part of the real object placed between the background object and the observer by projecting an image matched with the image of the background object onto the retroreflecting material. An information presentation device characterized by being transparent .

Images