JP2009288613A - Information presentation device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To present an image with less incompatibility to an external environment on a screen, without the need of directly measuring the position and direction of user's eyeballs. <P>SOLUTION: A viewpoint camera 1 is arranged in a position conjugate to the viewpoint 101 of the user 10. The viewpoint camera 1 is configured to capture an image in the direction of the screen 3 from the viewpoint 101 as a viewpoint image. A background camera 2 is configured to capture an environmental image lying behind the screen 3 as a background image. A projector 4 is arranged in a position conjugate to the viewpoint 101 and the viewpoint camera 1. The projector is configured to project a projection image formed by a processing part 5 on the screen 3. The processing part 5 is configured to make a correspondence between the background image and the viewpoint image and convert the background image to an image in the rear side of the screen 3 from the viewpoint 101. Thus, the projection image is formed. Further, the formed projection image is transmitted by the processing part 5 to the projector 4. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an information presentation apparatus. More particularly, the present invention relates to an apparatus capable of presenting information about an environment hidden from view by the screen on the screen.

  Technologies that provide Augmented Reality (AR) or Mixed Reality (MR) by projecting an image on a relatively small screen are disclosed in Patent Document 1 and Non-Patent Document 1 below. It is described in Patent Document 1.

  In the techniques described in these documents, AR and MR can be provided by projecting various images onto a small screen. Also, with these technologies, it is possible to make the presence of the screen difficult to be recognized by the user by projecting an image of the environment that is supposed to be behind the screen onto the screen. In other words, in this technique, objects constituting the screen can be recognized as if they are part of the environment, not as a screen.

  Specifically, in these techniques, an image with less discomfort with respect to an image of the environment (external environment) around the screen is presented on the screen. As a result, the screen can be buried in the external environment, and as a result, the presence of the screen can be made thin. In addition, in the field of AR or MR, the presence of a screen having a small presence as described above may be referred to as “transparency”.

  By the way, in the conventional technique, the position of the eyeball and the direction of the line of sight of the user are always measured in order to project an image that does not feel uncomfortable with the external environment on the screen. Then, an image consistent with the image that should be seen from the measured position and orientation is generated and projected onto the screen. Thereby, transparency of a screen can be performed appropriately.

Here, in order to always measure the direction of the user's line of sight, for example, a measuring device for detecting the eyeball position and the line of sight needs to be attached to the user's head. This measuring apparatus can measure a three-dimensional position and orientation (posture). Such a measuring device can be configured using, for example, a displacement sensor or an acceleration sensor.
JP 2000-122176 A INAMI, M., KAWAKAMI, N., AND TACHI, S. 2003. Optical camouflage using retro-reflective projection technology.In Proc. Of the 2nd ISMAR '03.

  However, when a measuring device is attached to the user's head, the movement of the user is restricted by the measuring device itself and the wiring therefor. In addition, the conventional measuring apparatus has a disadvantage that it is difficult to accurately measure the position and orientation of the user's eyeball. When the measurement accuracy of the position and orientation of the eyeball deteriorates, a sense of incongruity between the projected image and the external environment increases, and as a result, there is a possibility that the transparency of the screen may be hindered.

  In order to avoid this problem, it is conceivable to measure the position and orientation of the eyeball by attaching a measuring device in the environment (for example, indoor ceiling) and observing the movement of the user. However, with such a technique, the range in which the user can move is limited to a measurable range (for example, indoors). This technique also has a problem that it is still difficult to accurately measure the position and orientation of the eyeball. In addition, when the measuring device is installed outside, there is a problem that the entire device becomes larger or complicated, and the device cost and installation cost increase.

  The present invention has been made in view of such a situation. A main object of the present invention is to provide a technique that makes it possible to present an image with less discomfort to the external environment on the screen without directly measuring the position and orientation of the user's eyeball. That is.

  The present invention has a configuration described in any of the following items.

(Item 1)
It has a viewpoint camera, a background camera, a screen, a projector, and a processing unit.
The viewpoint camera is arranged at a position conjugate with the user's viewpoint,
And the viewpoint camera is configured to acquire an image of the screen and its surroundings as a viewpoint image when the direction of the screen is viewed from the viewpoint.
The background camera is configured to acquire, as a background image, an image of an environment existing on the back side of the screen when the screen is viewed from the viewpoint.
The screen is small enough to be visually recognized by the user together with the background within the user's field of view.
And the screen is configured to reflect at least a part of the image projected from the projector toward the user's viewpoint,
The projector is disposed at a position conjugate with the viewpoint and the viewpoint camera,
And the projector is configured to project the projection image generated by the processing unit onto the screen,
The processor is
By associating the background image acquired by the background camera with the viewpoint image acquired by the viewpoint camera, the background image is converted into an image when the back side of the screen is viewed from the viewpoint, As a result, a process for generating a projection image is performed.
Furthermore, the processing unit is configured to send the generated projection image to the projector.

  By arranging the viewpoint camera at a position conjugate with the user's viewpoint, an image viewed from the viewpoint can be acquired by the viewpoint camera. In this specification, an image obtained by a viewpoint camera (this image corresponds to an image viewed from the viewpoint) is referred to as a viewpoint image.

  In the present invention, the projector is disposed at a position conjugate with the user's viewpoint and the viewpoint camera. Then, if the same image as the viewpoint image acquired by the viewpoint camera is projected onto the screen from the projector, the user can observe the image without a sense of incongruity.

  In the present invention, it is possible to convert a background image on the basis of a viewpoint image acquired by a viewpoint camera, thereby generating a projection image and presenting it on a screen. Since the generated projection image is converted on the basis of the viewpoint image, it is possible to reduce discomfort with respect to the viewpoint image. On the other hand, the user can simultaneously observe the projected image presented on the screen and the environment around the screen. At this time, the user does not feel a sense of incongruity between the projected image and the surrounding environment, and can feel as if the screen is buried in the surrounding environment.

  In addition, in the present invention, since the background image is converted based on the viewpoint image, it is not necessary to measure the position and posture of the user's eyeball. For this reason, it becomes possible to omit the installation of a device for measuring the position and posture of the eyeball, and the movement of the user can be facilitated.

  Further, in the present invention, since the background image is converted based on the viewpoint image, it is possible to generate a more appropriate projection image as compared with the case where the measured position and orientation of the eyeball are used as a reference. Become.

(Item 2)
The information presentation apparatus according to item 1, wherein the background camera is disposed at a position on a back side of the screen.

  By installing a background camera on the back side of the screen, acquisition of a background image is facilitated.

(Item 3)
Item 3. The information presentation device according to Item 2, wherein the background camera is arranged in substantially the same direction as the viewpoint camera.

  When the orientations of the background camera and the viewpoint camera are greatly different, there is a high possibility of inconsistency between the background image and the viewpoint image in the occlusion relationship (so-called occlusion). On the other hand, in the invention of this item, avoiding such a problem can be facilitated by making the directions of both cameras substantially the same.

(Item 4)
The information presentation device according to any one of items 1 to 3, wherein the screen is a retroreflective screen.

  When the screen is a diffuse reflection screen, the image projected on the screen may be difficult for the user to see due to the influence of disturbance light (for example, light existing in the environment). On the other hand, in the invention of this item, by using the retroreflective screen, the brightness of the image projected on the screen is sufficiently strong against disturbance light. For this reason, the visibility of the projected image presented on the screen can be improved.

(Item 5)
The information display device according to any one of items 1 to 4, wherein the screen includes an arc surface that is convex outward.

  Examples of the screen having an arc surface protruding outward are a spherical screen and a cylindrical screen, but are not limited thereto.

  By presenting the projection image on the arc surface, the projection image can be observed even if the user moves. When the viewpoint image and the background image change due to the movement of the user, it is possible to present the projection image with a little uncomfortable feeling to the user by regenerating the projection image at an appropriate cycle.

(Item 6)
The background camera is a plurality,
And the said background camera is arrange | positioned so that the image of the environment which exists in the back side of the said circular arc surface may be acquired,
6. The information presentation device according to item 5, wherein directions of the plurality of background cameras are different from each other.

  By using a plurality of cameras having different directions, it is easy to obtain an appropriate background image corresponding to the moving position even if the user moves. For example, when a cylindrical screen is used, it is preferable to install the camera from the outer peripheral surface of the screen toward the outside direction (radius extension direction).

(Item 7)
The information presentation device according to any one of items 1 to 6, wherein the processing unit generates the projection image by the following processing:
(1) Processing for obtaining corresponding points between the background image and the viewpoint image;
(2) A process for projective transformation of the background image based on the corresponding point information.

  By performing projective transformation on the background image based on the corresponding point information, it is possible to generate a projected image with less discomfort with respect to the environment image viewed from the viewpoint. In addition, in the invention of this item, a projection image can be generated by a so-called image-based rendering (IBR) technique without generating a three-dimensional structure model.

(Item 8)
The information presentation device according to any one of items 1 to 7, wherein the projection image projected from the projector is configured to be projected across the screen and the outside of the screen.

  The projected image can be projected across both the screen and the periphery of the screen. In this case, even when the screen is moved or the user moves, if the screen is within the projection range of the projector, a projected image with a little uncomfortable feeling can be presented on the screen.

  In the invention of this item, the projection range (projection angle) of the projector can be set wide in advance. If an attempt is made to limit the projection range of the projector only to the screen, an apparatus configuration for that purpose is required. On the other hand, in the invention of this item, such an apparatus configuration becomes unnecessary, and the apparatus cost and the installation cost can be reduced. Further, in the present invention, a projection image having no contradiction with respect to the viewpoint image is generated and projected. Therefore, even if an image projected over a wide range is visually recognized by the user, it is considered that there is little trouble.

(Item 9)
The information presentation device according to any one of items 1 to 8, wherein the viewpoint camera and / or the projector is configured to be mounted on a user's head.

  By installing the camera and / or projector on the user's head, it is easy to keep the camera and projector in a conjugate position with the user's viewpoint even when the user moves. In addition, by mounting the viewpoint camera and / or projector on the user's head, the user's viewpoint and the conjugate point of the camera or projector can be physically connected (for example, by a fixture). For this reason, in the invention of this item, the viewpoint image can be generated with high accuracy. Furthermore, it becomes easy for the user to visually recognize the projected image accurately.

(Item 10)
An information presentation method comprising the following steps:
(A) a step of acquiring images of the screen and its surroundings as viewpoint images when the direction of the screen is viewed from the viewpoint of the user;
(B) obtaining an image of an environment existing behind the screen as the background image when the screen is viewed from the viewpoint;
(C) By associating the background image acquired by the background camera with the viewpoint image acquired by the viewpoint camera, the background image is changed to an image when the back side of the screen is viewed from the viewpoint. Converting and thereby generating a projection image;
(D) Projecting the projection image generated by the processing unit onto the screen from the viewpoint position.

(Item 11)
A computer program for causing a computer to execute each step according to item 10.

  This computer program can be recorded on various recording media such as a magnetic recording medium (such as a hard disk), an electrical recording medium (such as a flash memory or DRAM), an optical recording medium (such as a CD or DVD), or a magneto-optical recording medium (such as a hard disk). MO etc.). The type of the recording medium is not limited to these. The program can be transmitted as a signal via various media (such as an optical fiber and a copper wire).

  This program can be described in various languages (for example, C language or assembly language). The program may be described in a language that requires compilation before execution. In this case, the program may be a pre-compiled program or a compiled program.

  According to the present invention, it is possible to present on the screen an image that is less uncomfortable with respect to the external environment while eliminating the need to directly measure the position and orientation of the user's eyeball.

(First embodiment)
Hereinafter, the configuration of the information presentation apparatus according to the first embodiment of the present invention will be described with reference to FIGS. 1 and 2.

(Configuration of the first embodiment)
The information presentation apparatus according to the present embodiment includes a viewpoint camera 1, a background camera 2, a screen 3, a projector 4, a processing unit 5, a first half mirror 6, a second half mirror 7, and a casing 8. As a main configuration (see FIG. 1).

  The viewpoint camera 1 is disposed at a position conjugate with the viewpoint 101 of the user 10. Specifically, the first half mirror 6 and the second half mirror 7 are arranged between the viewpoint camera 1 and the viewpoint 101, and thereby the viewpoint camera 1 and the viewpoint 101 become conjugate. Yes.

  The viewpoint camera 1 is configured to acquire images about the screen 3 and its surroundings when the direction of the screen 3 is viewed from the viewpoint 101. The image acquired in this way is referred to as a viewpoint image in this specification.

  The viewpoint camera 1 is housed in a casing 8 together with the projector 4 and the first half mirror 6 (the casing 8 will be described later).

  In this embodiment, the angle of view of the viewpoint camera 1 is set so that not only the screen 3 but also an image over the periphery (outside) of the screen 3 can be taken. That is, the viewpoint camera 1 in the present embodiment is configured to be able to capture an image of the environment around the screen 3.

  The viewpoint camera 1 is configured to acquire image data and send it to the processing unit 5. The viewpoint camera 1 is not limited to a still image, and may be a camera that acquires a moving image. Since a moving image can be grasped as a sequence of still images, the description in this specification will be based on the assumption of still images. The viewpoint camera 1 acquires viewpoint images at a predetermined cycle. The predetermined period can be set as appropriate. For example, it is preferable to acquire about 10 to 60 images per second, but the upper limit is determined by constraints such as the capability of the processing unit 5.

  The background camera 2 is configured to acquire an image of the environment existing behind the screen 3 when the screen 3 is viewed from the viewpoint 101. The image acquired in this way is referred to as a background image in this specification.

  In this embodiment, the background camera 2 is attached to a screen 3 (described later) formed in a cylindrical shape, and is set toward the outside of the screen 3. Further, the background camera 2 is installed at a position (a position on the back side) that is separated from the user 10.

  The background camera 2 is configured to acquire image data about the background and send it to the processing unit 5. The background camera 2 may acquire a moving image as in the case of the viewpoint camera 1, but will be described on the premise of a still image here. Further, the background camera 2 acquires a background image in a predetermined cycle in synchronization with the viewpoint camera 1.

  The screen 3 has a size small enough to be visually recognized by the user together with the background within the visual field of the user 10. That is, the size of the screen 3 is such that “when the screen 3 is viewed, an image of an environment existing outside the screen 3 can be observed around the screen 3”. It is considered that it is difficult to apply the technique of the present embodiment under the condition that the entire visual field is covered with the screen and the external environment cannot be observed.

  The screen 3 is configured to reflect at least part of the image projected from the projector 4 toward the user's viewpoint.

  Specifically, in this embodiment, the screen 3 is a retroreflective screen. As the retroreflective screen, various configurations having a retroreflective function can be used. For example, the thing of the structure which apply | coated the retroreflection material (for example, glass bead) to the surface of a base material (not shown) can be utilized. Of course, there is no need to be constrained by this configuration. The basic configuration of the retroreflective screen may be the same as that described in Patent Document 1 and Non-Patent Document 1 described above, and thus detailed description thereof is omitted.

  Further, the screen 3 in the present embodiment is provided with an arc surface that is convex outward. Specifically, the screen 3 of the present embodiment is formed in a cylindrical shape. And the outer peripheral surface becomes a screen provided with the circular arc surface which becomes convex outside. However, the shape of the screen 3 is not limited to a cylindrical shape, and for example, a spherical shape can be used. The shape of the screen 3 is not particularly limited as long as it is a size within the projection range of the projector 4. The shape of the screen 3 may be not only a simple shape such as a plane, a sphere, and a cylinder, but also a complicated shape such as a three-dimensional shape of a person. Even in such a case, if the projector 4 projects an image from a point conjugate with the viewpoint 101, projection without distortion becomes possible.

  The projector 4 is disposed at a position conjugate with the viewpoint 101 and the viewpoint camera 1. Specifically, the projector 4 is conjugate with the viewpoint 101 by the action of the first half mirror 6. Further, the projector 4 is conjugate with the viewpoint camera 1 by the action of the second half mirror 7.

  The projector 4 is configured to project a projection image (described later) generated by the processing unit 5 onto the screen 3 via the first and second half mirrors 6 and 7.

  In this embodiment, the projection image projected from the projector 4 is configured to be projected not only on the screen 3 but also around the screen 3 (outside). That is, the projection angle of the projector 4 in the present embodiment is set to a size that can project an image up to the periphery of the screen 3.

  As described above, the projector 4 and the viewpoint camera 1 are accommodated in the casing 8 (see FIG. 2). The casing 8 is fixed to the user's head (for example, the temporal region) by a fixture (not shown). Of course, if the fixing is released, the casing 8 can be removed from the head.

  Therefore, in this embodiment, the viewpoint camera 1 and the projector 4 are configured to be mounted on the user's 10 head via the casing 8.

  The processing unit 5 first associates the background image acquired by the background camera 2 with the viewpoint image acquired by the viewpoint camera 1. As a result, the background image can be converted into an image when the back side of the screen 3 is viewed from the viewpoint 101. The processing unit 5 generates a projection image by this processing. Further, the processing unit 5 is configured to send the generated projection image to the projector 4.

More specifically, in the processing unit 5 of the present embodiment, the process of generating a projection image is performed by the following process:
(1) Processing for acquiring corresponding points between the background image and the viewpoint image;
(2) A process for projective transformation of the background image based on the corresponding point information.

  The processing unit 5 can be configured by, for example, appropriate hardware such as a personal computer or a workstation and an appropriate program for operating the hardware. Specific operations in the processing unit 5 will be described later.

(Operation of the first embodiment)
Next, the operation of the information presentation apparatus according to the present embodiment will be described with reference mainly to the flowchart shown in FIG. An example of the positional relationship among the user 10, the screen 3, and the environment is schematically shown in FIG. In FIG. 4, reference numeral 9 indicates an external environment (exactly part thereof). Furthermore, the positional relationship between the user 10 and the screen 3 is conceptually shown in FIG. Note that wireless or wired communication can be used as a communication path for performing data transmission between the processing unit 5 and each camera and projector. As a protocol for configuring the wireless communication path, for example, Bluetooth can be used, but is not limited thereto.

(Steps SA-1 and SA-2)
A background image is taken by the background camera 2. An example of the photographed image is shown in FIG. Thereafter, the lens distortion in the acquired background image is corrected. The background image obtained as a result is shown in FIG. Since the existing technique can be used as the technique for correcting the lens distortion, a detailed description thereof will be omitted.

(Steps SA-3 and SA-4)
A viewpoint image is taken by the viewpoint camera 1 before or after the above steps or in parallel with these steps. An example of the photographed image is shown in FIG. Thereafter, as in the case of step SA-2 described above, lens distortion in the acquired viewpoint image is corrected. FIG. 6D shows the viewpoint image obtained as a result.

(Step SA-5)
Next, a corresponding point between the corrected background image and the viewpoint image is searched. An example of the corresponding point search algorithm will be described in detail below.

  Obtaining correspondence between images taken by cameras at different positions is generally a difficult problem. In the present embodiment, it is assumed that the background object is a plane or is sufficiently away from each camera. In addition, a method called Scale-Invariant Feature Transform (SIFT) is used to obtain the correspondence between the background image and the viewpoint image. SIFT is an algorithm for detecting feature points and describing feature quantities. SIFT describes features that are robust to image rotation, scale changes, lighting changes, and so on. For this reason, SIFT is suitable for obtaining correspondence between images obtained from different viewpoints. SIFT can detect correspondence between images even when linear transformation of images is performed, so if the condition that the background object is a plane or is sufficiently far from the camera is satisfied, it will be very accurate. You can get a good response. In addition, even if the background object is not a flat surface or the like, it is considered to be a technique that can generate a projection image that is close to optimal when viewed by a human because a characteristic point is associated between images. It is done.

  The SIFT algorithm is outlined below. SIFT processing consists of two stages: feature point detection and feature quantity description.

(Detection of feature points)
In the detection of feature points, first, an extreme value is searched from a Difference-of-Gaussian (DOG) image that is a difference between smoothed images having different scales. Thereby, the position and scale of the candidate point are determined. Next, the detected candidate points are narrowed down by the main curvature and contrast. Further, the position and scale of the feature point are calculated by subpixel estimation.

(Description of features)
Next, the feature amount is described. First, orientation is obtained from the brightness gradient in the vicinity of the detected feature point. By describing the feature value by rotating in the orientation direction, the feature value becomes invariant to the rotation. Then, luminance gradient information in a circular area centered on the feature point and having the radius of the scale of the feature point is described as a feature vector.

  As SIFT algorithms, those described in the literature (eg, DG Lowe. Object recognition from local scaleinvariant features. In Proc. Of IEEE International Conference on Computer Vision (ICCV), pp. 1150.1157, 1999.) can be used. Detailed description will be omitted.

  An example of the result of detecting the feature points is shown in FIG. In this figure, reference numeral 11 denotes a viewpoint image, and reference numeral 21 denotes a background image. In addition, a large number of lines connecting the two images represent the correspondence between feature points.

(Step SA-6)
Next, a projective transformation matrix is calculated based on the corresponding point search result. This calculation algorithm will be described in detail below.

  When one plane is photographed by two cameras, the correspondence between the two obtained images is related by plane projective transformation. This correspondence is conceptually shown in FIG. A symbol H in FIG. 8 is a projective transformation matrix.

Point on 2D plane
x = [x, y] ^T
From the same point on the two-dimensional plane
x ′ = [x ′, y ′] ^T
Projective transformation to the, [x1, x2, x3] T ~ [x, y, 1] T becomes homogeneous coordinates as equivalence relation is established
x tilde = [x1, x2, x3] ^T
If you use
Can be written. Here, s is an arbitrary real number, and H is a 3 × 3 matrix as follows.

H is called a projective transformation matrix. A matrix H is obtained for each image (that is, each frame) taken periodically. An image taken from the viewpoint 101 of the user 10 is obtained by projective transformation of the image taken by the background camera 2. The number of elements of the projective transformation matrix H exemplified above is 9, but the degree of freedom is 8 because all the elements are multiplied by a constant to represent the same transformation. Therefore, h ₃₃ = 1 may be set. Given a pair of corresponding points, from Equation 1,
Is obtained. Organize
The following two equations are obtained. As a result, if four or more sets of correspondences are known, the projective transformation matrix H can be obtained by the least square method.

  However, there is a possibility that the corresponding points detected using the SIFT feature amount include incorrect correspondence. If there is an incorrect correspondence, an image after projective transformation may be greatly distorted due to the influence of an outlier. Therefore, a projective transformation matrix is obtained using RANSAC (RANdom SAmple Consensus). RANSAC is a method for performing fitting by removing outliers from data including outliers.

The RANSAC algorithm is shown below.
1. Select several corresponding points at random.
2. Calculate the parameters using the least squares method.
3. When the obtained parameters are applied, the number of points within the set error range is taken as the evaluation value.
4. If the evaluation value is sufficiently large, adopt the obtained parameter and finish.
5. Repeat steps 1 to 4 a finite number of times.
6. If a sufficiently large evaluation value is not obtained, it is considered a failure and the process is terminated.

  In this way, a correct parameter (that is, a projective transformation matrix) can be estimated. The RANSAC algorithm is described in literature (for example, MA Fischler and RC Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. In Commun. ACM, Vol. 6, pp. 381.395, 1981.) Since the described ones can be used, further details will be omitted.

  In this embodiment, the correspondence between images can be made very robust by using the corresponding point search based on the SIFT feature amount and the calculation of the projective transformation matrix by RANSAC in combination.

(Step SA-7)
Next, projective transformation is applied to the background image using the calculated projective transformation matrix. Thereby, an image to be projected (projected image) can be generated. An example of the conversion is shown in FIG. FIG. 9A shows an example of a background image. A portion indicated by a frame L in FIG. 9B shows a projection image generated by projective transformation.

(Step SA-8)
Next, the generated projection image can be projected from the projector 4 toward the screen 3.

  An example of an image projected toward the screen 3 is shown in FIG. FIG. 10A shows a state in which a projection image is irradiated on a screen (a circular planar object in this example) and is visually recognized together with the external environment. FIG.10 (b) is an enlarged view of the principal part in Fig.10 (a).

  In the example of FIG. 10, the image around the screen is an image of the actual environment, and the image on the screen is a projected image. However, as shown in this figure, since the projection image is aligned with the environment image, the screen is recognized as if it were buried in the environment.

  FIG. 11 shows an example in which the viewpoint is moved and projection images obtained from the respective viewpoints are projected. FIG. 11A shows a case where the viewpoint is away from the screen. FIG. 11B shows a case where the viewpoint approaches the screen. FIG.11 (c) shows the case where a screen is seen from the diagonal direction. FIG. 11D shows a case where the viewpoint is rotated.

  In the apparatus of the present embodiment, the viewpoint camera 1 can be acquired by the viewpoint camera 1 by arranging the viewpoint camera 1 at a position conjugate with the viewpoint 101 of the user 10.

  In this embodiment, the projector 4 is disposed at a position conjugate with the viewpoint 101 of the user 10 and the viewpoint camera 1. In this case, when the same image as the viewpoint image acquired by the viewpoint camera 1 is projected from the projector 4 onto the screen, the user 10 can observe the image without a sense of incongruity.

  In the present embodiment, the background image is converted using the viewpoint image acquired by the viewpoint camera 1 as a reference, thereby generating a projection image and presenting it on the screen 3. Since the generated projection image is converted on the basis of the viewpoint image, it is possible to reduce discomfort with respect to the viewpoint image. On the other hand, the user can observe the projected image presented on the screen 3 and the environment around the screen at the same time. At this time, the user does not feel much discomfort between the projected image and the surrounding environment, and can feel as if the screen 3 is buried in the surrounding environment. That is, according to this technique, the screen 3 can be made transparent.

  In addition, in the present embodiment, since the background image is converted based on the viewpoint image, there is no need to measure the position and posture of the user's eyeball. For this reason, it becomes possible to omit the installation of a device for measuring the position and posture of the eyeball, and the movement of the user can be facilitated. Furthermore, by omitting the installation of a device that measures the position and orientation of the eyeball, this embodiment also has an advantage that the manufacturing cost and installation cost of the information presentation device can be reduced.

  That is, according to the present embodiment, it is possible to present an image with a little uncomfortable feeling to the external environment on the screen without directly measuring the position and orientation of the user's eyeball.

  In addition, the accuracy of the position and posture of the eyeball measured by the measuring device is not sufficient for accurately estimating where the user is gazing, and this can improve the accuracy of the projection image. There was a problem that it was difficult. In contrast, in the present embodiment, since the background image is converted based on the viewpoint image, an appropriate projection image can be obtained with higher accuracy than in the case where the measured position or posture of the eyeball is used as a reference. Can be generated. For this reason, there exists an advantage that a projection image with little discomfort with respect to an environment can be projected on a screen.

  Furthermore, in this embodiment, since the background camera 2 is installed on the back side of the screen 3, there is an advantage that acquisition of a background image is easy.

  Furthermore, in the present embodiment, it is preferable that the background camera 2 is arranged in substantially the same direction as the viewpoint camera 1. If the orientations of the background camera 2 and the viewpoint camera 1 are greatly different, there is a high possibility of inconsistency between the background image and the viewpoint image in the occlusion relationship (so-called occlusion). On the other hand, avoiding such a problem can be facilitated by making both directions substantially coincide.

  When the screen 3 is a diffuse reflection screen, the image projected on the screen 3 may be difficult for the user 10 to see due to the influence of disturbance light (for example, light existing in the environment). On the other hand, in the present embodiment, by using the retroreflective screen, the brightness of the image projected on the screen 3 is sufficiently strong against disturbance light. For this reason, the visibility of the projected image presented on the screen can be improved.

By making the screen 3 a retroreflective screen, the following advantages can be further exhibited.
-Since the light projected from the viewpoint position returns efficiently to the viewpoint, the amount of projection light can be reduced. Therefore, the projector can be downsized and the amount of heat generated can be kept low. This makes it easier to attach the projector to the head.
-Since the amount of projection light can be reduced, the depth of focus of the projector can be increased. For this reason, the image on the screen can be viewed with high precision without strictly adjusting the focal position of the projector.
-Even if the shape of the screen is distorted, basically, the user can visually recognize the same image as the projected image. Accordingly, the degree of freedom of the shape of the screen is increased, and various shapes can be adopted.
-Since light projected from different locations is reflected in the direction of the light source, a number of images can be presented on one screen. For this reason, an image corresponding to each viewpoint can be presented to a plurality of users who are simultaneously viewing the same screen.

  Furthermore, the screen 3 of the present embodiment includes an arc surface that is convex outward. By presenting the projection image on the arcuate surface portion of the screen 3, the projection image can be observed even if the user 10 moves. Even when the user 10 moves, it is possible to present a projected image with less discomfort to the user by continuously generating the projected image at an appropriate cycle.

In this embodiment, the process of generating a projection image includes (1) a process of acquiring corresponding points between the background image and the viewpoint image; and (2) a background image based on the information of the corresponding points. It is implemented by the process of projective transformation. In this way, by performing projective transformation of the background image based on the corresponding point information, it is possible to generate a projection image that is less uncomfortable with respect to the environment image viewed from the viewpoint. Moreover, in the present embodiment, a projection image can be generated by a so-called image-based rendering method without generating a three-dimensional structure model. Therefore, the burden on the processing unit 5 is reduced, and a background image can be generated in a short cycle.

  In the present embodiment, the projected image is projected so as to cover both the screen 3 and the periphery of the screen. Therefore, in the present embodiment, even when the screen 3 is moved or the user 10 moves, if the screen 3 is within the projection range of the projector 4, a projected image with less discomfort is presented on the screen. be able to.

  Further, in the apparatus of the present embodiment, the projection angle of the projector 4 can be set widely. With respect to an image projected on an object that is outside the screen 3 and is not a retroreflective material, most of the light is not reflected in the direction of the viewpoint, so that it is substantially invisible to the user.

  Further, if it is attempted to limit the projection range of the projector 4 only to the screen, an apparatus configuration for that purpose is required. On the other hand, in this embodiment, such an apparatus configuration becomes unnecessary, and the apparatus cost and the installation cost can be reduced. Further, in this embodiment, a projection image having no contradiction with respect to the viewpoint image is generated and projected, so that even if an image projected over a wide range is visually recognized by the user, it is considered that there is little trouble.

  Furthermore, in this embodiment, the viewpoint camera 1 and the projector 4 are mounted on the user's 10 head. Thereby, even if the user 10 moves, it becomes easy to keep the viewpoint camera 1 and the projector 4 in a conjugate position with the viewpoint of the user 10. Further, by attaching the viewpoint camera 1 and the projector 4 to the head of the user 10, the viewpoint of the user 10 and a conjugate point in the camera 1 and the projector 4 are physically connected (for example, by a fixture). be able to. For this reason, the apparatus of this embodiment can generate the viewpoint image with high accuracy. Furthermore, since the projector 4 can be accurately attached at a position conjugate with the viewpoint, it is easy for the user 10 to visually recognize the projection image accurately.

  Various processes in the present embodiment can be executed on a computer by an appropriate computer program. For example, the processing in the processing unit 5 is appropriately performed by a computer program.

(Second Embodiment)
Next, an information presentation apparatus according to a second embodiment of the present invention will be described based on FIG. In the description of the second embodiment, elements that are basically the same as the components described in the first embodiment are denoted by the same reference numerals, and the description is simplified.

  In the second embodiment, the screen 3 is formed in a flat plate shape. The front surface (user side surface) of the screen 3 is a retroreflecting surface. The background camera 2 is attached to the back side of the screen 3.

  In the apparatus of the second embodiment, the screen 3 can be moved to an appropriate position while holding the screen 3 by hand. At that time, an appropriate projection image corresponding to the positional relationship between the screen 3 and the user 10 can be projected onto the screen 3. When the user 10 visually recognizes the projection image in the environment, the user 10 can feel as if the screen 3 is buried in the environment. Further, by enlarging or reducing the scale of the projected image, the screen 3 can be recognized as if it were a convex or concave lens.

  Other configurations and advantages of the second embodiment are basically the same as those of the first embodiment, and thus detailed description thereof is omitted.

(Third embodiment)
Next, an information presentation apparatus according to a third embodiment of the present invention will be described based on FIG. In the description of the third embodiment, elements that are basically the same as the components described in the first embodiment are denoted by the same reference numerals, and the description is simplified.

  In the third embodiment, the screen 3 is formed in a spherical shape. The entire surface of the screen 3 is a retroreflecting surface. There are a plurality of background cameras 2. The plurality of background cameras 2 are attached around the screen 3 so as to face outward.

  Even in the apparatus of the third embodiment, the screen 3 can be moved to an appropriate position while holding the screen 3 by hand. At that time, an appropriate projection image corresponding to the positional relationship between the screen 3 and the user 10 can be projected onto the screen 3.

  Furthermore, in the apparatus according to the third embodiment, the background in a wide range can be acquired by the plurality of background cameras 2. Therefore, even when the user 10 looks into the side surface or the back surface of the screen 3, an appropriate projection image corresponding to the positional relationship between the screen 3 and the user 10 can be projected onto the screen 3.

  Other configurations and advantages of the third embodiment are basically the same as those of the first embodiment, and thus detailed description thereof is omitted.

(Fourth embodiment)
Next, an information presentation apparatus according to a fourth embodiment of the present invention will be described with reference to FIG. In the description of the fourth embodiment, elements that are basically the same as the components described in the first embodiment described above are simplified by using the same reference numerals.

  In the fourth embodiment, the image projected on the screen 3 is acquired again, and the projected image is fed back. Hereinafter, the process of applying feedback will be described with reference to FIG.

(Step SB-1)
The image projected on the screen 3 according to the first embodiment is acquired again by the viewpoint camera 2.

(Step SB-2)
Next, the processing unit 5 compares the image acquired by the viewpoint camera 2 (that is, the image displayed on the screen) with the image projected from the projector 4 (projected image).

(Step SB-3)
The projected image is corrected so that the displayed image is the same as the projected image. At this time, as the feature amount for comparing the images, appropriate ones such as color and luminance can be used. If the display image and the projected image are sufficiently close, correction can be omitted.

(Step SB-4)
The corrected projection image is projected onto the screen 3. Thereafter, by further comparing the image on the screen with the initial projection image by the viewpoint camera 2, the projection image can be corrected again.

  Other configurations and advantages of the fourth embodiment are basically the same as those of the first embodiment, and thus detailed description thereof is omitted.

  Note that the description of the embodiment and the examples is merely an example, and does not indicate a configuration essential to the present invention. The configuration of each part is not limited to the above as long as the gist of the present invention can be achieved.

  For example, each functional element described above (for example, the processing unit 5) only needs to exist as a functional block, and does not need to exist as independent hardware. As a mounting method, hardware or computer software may be used. Furthermore, one functional element in the present invention may be realized by a set of a plurality of functional elements, and a plurality of functional elements in the present invention may be realized by one functional element.

  Moreover, the functional element may be arrange | positioned in the position physically separated. In this case, the functional elements may be connected by a network. It is also possible to realize functions or configure functional elements by grid computing.

It is a block diagram for demonstrating the schematic structure of the information presentation apparatus which concerns on 1st Embodiment of this invention. It is an enlarged view of the principal part in the apparatus of FIG. It is a flowchart for demonstrating an example of the information presentation method using the apparatus shown by FIG. It is explanatory drawing for demonstrating the positional relationship of a user, a screen, and external environment. It is explanatory drawing for demonstrating the positional relationship of a user and a screen. It is a figure for demonstrating an example of distortion correction of the lens acquired with the camera. FIG. 1A shows a background image before correction. FIG. (B) shows a background image after correction. FIG. 3C shows a viewpoint image before correction. FIG. 4D shows a corrected viewpoint image. It is explanatory drawing for demonstrating the corresponding point between a background image and a viewpoint image. It is explanatory drawing for demonstrating the principle of projective transformation. It is a figure which shows an example of the projective transformation using a projective transformation matrix. FIG. 1A shows a background image before conversion. FIG. (B) shows the background image after conversion. In FIG. 5B, a viewpoint image is shown around a background image surrounded by a frame for reference. FIG. 4A is a diagram for explaining a state in which a projection image is projected toward the screen. Fig. (B) is a partially enlarged view of Fig. (A). It is a figure explaining the state which produced | generated the projection image in the state which changed the viewpoint variously, and was projected. The figure (a) shows a case where the viewpoint is away from the screen. FIG. (B) shows a case where the viewpoint approaches the screen. FIG. 3C shows a case where the screen is viewed from an oblique direction. FIG. 4D shows a case where the viewpoint is rotated. It is explanatory drawing for demonstrating the information presentation apparatus which concerns on 2nd Embodiment of this invention. It is explanatory drawing for demonstrating the information presentation apparatus which concerns on 3rd Embodiment of this invention. It is a flowchart for demonstrating the information presentation apparatus which concerns on 4th Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Viewpoint camera 11 Viewpoint image 2 Background camera 21 Background image 3 Screen 4 Projector 5 Processing part 6 1st half mirror 7 2nd half mirror 8 Casing 9 External environment 10 User 101 Viewpoint of user

Claims (11)

It has a viewpoint camera, a background camera, a screen, a projector, and a processing unit.
The viewpoint camera is arranged at a position conjugate with the user's viewpoint,
And the viewpoint camera is configured to acquire an image of the screen and its surroundings as a viewpoint image when the direction of the screen is viewed from the viewpoint.
The background camera is configured to acquire, as a background image, an image of an environment existing on the back side of the screen when the screen is viewed from the viewpoint.
The screen is small enough to be visually recognized by the user together with the background within the user's field of view.
And the screen is configured to reflect at least a part of the image projected from the projector toward the user's viewpoint,
The projector is disposed at a position conjugate with the viewpoint and the viewpoint camera,
And the projector is configured to project the projection image generated by the processing unit onto the screen,
The processor is
By associating the background image acquired by the background camera with the viewpoint image acquired by the viewpoint camera, the background image is converted into an image when the back side of the screen is viewed from the viewpoint. Thus, it is configured to perform a process of generating a projection image,
Furthermore, the processing unit is configured to send the generated projection image to the projector.   The information presentation apparatus according to claim 1, wherein the background camera is disposed at a position on a back side of the screen.   The information presentation apparatus according to claim 2, wherein the background camera is arranged in substantially the same direction as the viewpoint camera.   The information presentation device according to claim 1, wherein the screen is a retroreflective screen.   The information presentation device according to claim 1, wherein the screen includes an arc surface that is convex outward. The background camera is a plurality,
And the said background camera is arrange | positioned so that the image of the environment which exists in the back side of the said circular arc surface may be acquired,
The information presentation device according to claim 5, wherein directions of the plurality of background cameras are different from each other. The information presentation device according to claim 1, wherein the processing unit generates the projection image by the following processing.
(1) Processing for obtaining corresponding points between the background image and the viewpoint image;
(2) A process for projective transformation of the background image based on the corresponding point information. The information presentation apparatus according to claim 1, wherein the projection image projected from the projector is configured to be projected across the screen and the outside of the screen. The information presentation device according to any one of claims 1 to 8, wherein the viewpoint camera and / or the projector is configured to be mounted on a head of the user. An information presentation method comprising the following steps:
(A) a step of acquiring images of the screen and its surroundings as viewpoint images when the direction of the screen is viewed from the viewpoint of the user;
(B) obtaining an image of an environment existing behind the screen as the background image when the screen is viewed from the viewpoint;
(C) By associating the background image acquired by the background camera with the viewpoint image acquired by the viewpoint camera, the background image is changed to an image when the back side of the screen is viewed from the viewpoint. Converting and thereby generating a projection image;
(D) Projecting the projection image generated by the processing unit onto the screen from the viewpoint position.   The computer program for making a computer perform each step of Claim 10.