JP5799776B2 - Information display device and program - Google Patents

Description

  The present invention relates to an information display device and a program for displaying information on a transmissive display.

  In recent years, augmented reality (AR) technology has become widely used. In the field of augmented reality, the user views the real space through a transmissive display such as an HMD (Head Mounted Display). When a comment text or the like is added to an object or the like in the real space for display, the comment text is displayed at the position where the user is looking at the object in the transmissive display.

  In this case, it is necessary to match at least the coordinate system of the real space with the coordinate system of the virtual space in the transmissive display. Patent Document 1 lists a number of methods for aligning the real space and the virtual space in the HMD.

JP 2008-046750 A

  However, in the conventional HMD or the like, the coordinate system of the virtual space is set based on the coordinate system of a sensor such as a camera that recognizes the real space, and the direction of the user's line of sight is not considered. For this reason, even if the virtual space and the real space are aligned, the display position of the information displayed in the virtual space appears to be shifted by the user in relation to the object in the real space. It was supposed to be.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an information display device and a program in consideration of the user's line-of-sight direction.

  The present invention for solving the problems of the above-described conventional example is an information display device, a transmissive display arranged in front of a user's eyes, means for detecting a user's line-of-sight vector V, and the detection Means for defining the user's visual field coordinate system Ep using the line-of-sight vector V, and a position for obtaining a transformation matrix for converting the coordinates in the defined visual field coordinate system Ep of the user into coordinates in the external world coordinate system Wp Matching means, means for converting coordinates representing the position of the object in the external world coordinate system Wp into coordinates representing the position in the visual field coordinate system Ep by the conversion matrix, and coordinates of the visual field coordinate system Ep And means for determining the display position of the information on the object by projecting on the transmissive display, and means for displaying the information on the object at the determined display position.

  In addition, the sensor further includes a sensor that detects that the head of the user is worn, and the alignment unit detects that the information display device is worn by the sensor when the conversion is performed. A matrix may be obtained, or means for detecting a relative position variation between the information processing device and the viewpoint position after the sensor detects that the information display device is mounted on the user's head The alignment unit may correct the conversion matrix based on the detected relative position fluctuation. Furthermore, there may be provided means for obtaining a transformation matrix between the external world coordinate system Wp and the coordinate system Dp on the transmissive display.

  A program according to an aspect of the present invention uses a computer including a transmissive display to detect a user's line-of-sight vector V, and uses the detected line-of-sight vector V to determine a user's visual field coordinate system Ep. Defining means; positioning means for obtaining a transformation matrix for converting coordinates in the visual field coordinate system Ep of the defined user into coordinates in the external world coordinate system Wp; and position of the object in the external world coordinate system Wp Means for converting coordinates representing the position into coordinates representing the position in the visual field coordinate system Ep by the transformation matrix, and projecting the coordinates of the visual field coordinate system Ep onto the transmissive display, The information display position is functioned as means for displaying information on the object at the determined display position.

  According to the present invention, it is possible to display information in consideration of the user's line-of-sight direction.

It is a block diagram showing the structural example of the information display apparatus which concerns on embodiment of this invention. It is a schematic diagram showing one of the implementation examples of the information display apparatus which concerns on embodiment of this invention. It is a functional block diagram showing the example of the information display apparatus which concerns on embodiment of this invention. It is a flowchart figure showing the example of the calibration process of the information display device concerning an embodiment of the invention. It is explanatory drawing showing the example of the user's visual field coordinate system which the information display apparatus which concerns on embodiment of this invention utilizes. It is explanatory drawing showing the example of the database which the information display apparatus which concerns on embodiment of this invention uses. It is a flowchart figure showing the example of the process of the information display by the information display apparatus which concerns on embodiment of this invention. It is a schematic diagram showing another implementation example of the information display apparatus according to the embodiment of the present invention.

  An information display device 1 according to an embodiment of the present invention includes, for example, a transmissive display, and a user wears like a pair of glasses, and displays an information display object in the line-of-sight direction via the transmissive display. In addition to visual recognition, information related to the object is displayed near the position where the line segment connecting the user and the object intersects the surface of the transmissive display. Thereby, the information regarding a target object is shown with respect to a user.

  An example of the information display device 1 according to the present embodiment includes a control unit 11, a storage unit 12, an imaging unit 13, an eyeball detection unit 14, and a transmissive display 15, as illustrated in FIG. . As will be described later, the information display device 1 may include a mounting sensor 16. Further, the information display device 1 may include a displacement sensor 17.

  The information display device 1 according to one aspect of the present embodiment has a substantially glasses shape as illustrated in FIG. That is, the information display device 1 of this example is provided on a pair of temples 21, a pair of rims 22 connected to one end side of each temple 21 via hinges, and a part of each rim 22. It is mounted on a frame 20 having a nose pad portion 23 for supporting the rim 22 by hitting the user's nose and a bridge 24 connecting the pair of rims 22.

  In the example illustrated in FIG. 2, the transmissive display 15 is supported on at least one rim 22. When the transmissive display 15 is supported on one rim 22, the other rim 22 is made of a glass or plastic lens (which may or may not be included for correction).

  Here, the control unit 11 is a program control device such as a CPU, and operates according to a program stored in the storage unit 12. In the present embodiment, the control unit 11 detects the user's viewpoint position, processes to estimate the user's line-of-sight vector V from the detected user's viewpoint position, and the estimated user's line-of-sight vector. Alignment processing for obtaining a transformation matrix for transforming V into a vector in the external world coordinate system Wp, the coordinates of the object in the external world coordinate system Wp, and the line segment in the direction of the line-of-sight vector transformed by the transformation matrix are transmission type A process for obtaining the display reference position from the coordinates of the positions intersecting on the display 15 and a process for determining the information display position based on the obtained display reference position are performed. A detailed description of the processing in the control unit 11 will be given later.

  The storage unit 12 stores a program executed by the control unit 11. This program may be provided by being stored in a computer-readable recording medium such as a DVD-ROM and copied to the storage unit 12. The storage unit 12 also operates as a work memory for the control unit 11.

  The imaging unit 13 is, for example, a camera, and sequentially captures images in the direction of an external object (real object), and outputs the captured images to the control unit 11 as digital image data. In the case of the example shown in FIG. 2, when the user wears the imaging range of the imaging unit 13, the imaging direction is substantially the same as the user's viewing direction, but the viewing angle is different, so the imaging unit The image data output by 13 is different from the image viewed by the user.

In the present embodiment, the camera of the imaging unit 13 is fixed to a part of the bridge 24 and is arranged so as to capture the direction of the line of sight when the user wears the information display device 1.
The eyeball detection unit 14 includes, for example, a camera C independent of the imaging unit 13 and a near-infrared light projector P. The eyeball detection unit 14 is also fixed to a part of the bridge 24, for example. The camera C and the near-infrared light projector P are arranged so as to capture the direction of the face of the user wearing the information display device 1 and project the light in the direction of the face. The eyeball detection unit 14 outputs image data captured by the camera C to the control unit 11. As described above, the camera C of the eyeball detection unit 14 and the camera that is the imaging unit 13 are both fixed to the bridge 24 and fixed so that their relative coordinates do not change. Thereby, the coordinate system Cc2 of the camera C of the eyeball detection unit 14 and the coordinate system Cc1 of the camera which is the imaging unit 13 can be converted to each other by the conversion matrix [Tc1,2]. That is,
Cc2 = [Tc1,2] Cc1, Cc1 = [Tc1,2] -1Cc1 = [Tc2,1] Cc1
It is.

  The transmissive display 15 is supported by the rim 22. The transmissive display 15 is a liquid crystal display panel, an organic EL panel or the like that can see an object on the back side through the display surface, and has a transparent or translucent display surface that functions as a display. Prepare. Since such a transmissive display is also widely known, a description thereof is omitted here.

  The transmissive display 15 displays information on the display surface according to an instruction input from the control unit 11. In this example, since the transmissive display 15 is also supported by the rim 22 connected by the bridge 24, the coordinate system of the camera as the imaging unit 13 (hereinafter referred to as the coordinate system of the imaging unit 13) Cc1 and The transformation matrix [Tc1, Dp] with respect to the coordinate system Dp of the transmissive display 15 is also known.

  Next, the specific content of the process in the control part 11 is demonstrated. Functionally, the control unit 11 includes a calibration instruction unit 31, a calibration processing unit 32, and an information display unit 33, as shown in FIG. The calibration instruction unit 31 instructs the calibration processing unit 32 to execute calibration processing at a time point that is determined in advance as a timing at which calibration should be performed. This timing may be, for example, a timing that comes regularly every predetermined time after power-on.

  When the calibration processing unit 32 receives a calibration processing execution instruction from the calibration instruction unit 31, the calibration processing unit 32 executes the next calibration processing. In the following description, the control unit 11 uses the so-called Thai camera calibration algorithm for each of the camera of the imaging unit 13 and the camera C of the eyeball detection unit 14, that is, the optical axis center. Information including the position, the aspect ratio of the pixel, and the camera position and rotation angle (transformation matrix between the world coordinate system and the camera coordinate system) with respect to the world coordinate system Cw with the imaged object as a reference is obtained ( RYTsai, "A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses", IEEE Journal of Robotics and Automation, Vol. RA-3, No.4, pp. 323- 344, 1987).

Here, the world coordinate system Cw is generally different for each of the camera of the imaging unit 13 and the camera C of the eyeball detection unit 14. Here, a world coordinate system obtained from image data (image data in the user's face direction) captured by the camera C of the eyeball detection unit 14 is defined as a user coordinate system Wu.
In the present embodiment, the external world coordinate system Wp may be, for example, the coordinate system Cc1 itself of the imaging unit 13, or may be the world coordinate system for the coordinate system Cc1 of the imaging unit 13.

  Further, as an example, the control unit 11 forms a virtual rectangular area on the road surface, floor surface, or surface constituting the wall surface imaged in the image data based on the image data obtained from the imaging unit 13. This rectangular area can be formed by adopting a method called PTAM (Parallel Tracking and Mapping) if the user is moving in translation with respect to the external coordinate system Wp, for example, during vehicle operation. This PTAM is described in detail in Georg Klein and David Murray, "Parallel Tracking and Mapping for Small AR Workspaces", In Proc. International Symposium on Mixed and Augmented Reality (ISMAR '07), 2007, etc. Detailed description is omitted. The control unit 11 displays, for example, predesignated information in this rectangular area.

  As illustrated in FIG. 4, the calibration processing unit 32 acquires image data output from the eyeball detection unit 14 (S1). Then, the user's line-of-sight vector V and the pupil center are estimated (S2). Here, the pupil center corresponds to the center coordinates of the movement of the eyeball when the user does not move and the face does not move.

  In an example of this embodiment, the calibration processing unit 32 uses the method of Takehiko Ohno et al., “Gaze measurement method based on eyeball shape model”, 8th Image Sensing Symposium, pp.307-312,2002. A coordinate system Ep of the user's viewpoint as the line-of-sight direction is estimated. In this method, the position of the pupil (pupil center) and the corneal surface reflected light (Purkinje image) are used.

  Next, the calibration processing unit 32 detects the user's pupil from the image data acquired in step S1. For such pupil detection technology, for example, Hideki Kashima et al., "Study on pupil detection methods that are resistant to face orientation changes", IPSJ Research Report. CVIM, 2001 (36), 139-144, 2001-05 Since there are various methods such as -10, detailed explanation is omitted.

  The calibration processing unit 32 also extracts a bright spot (a pixel having a brightness higher than a predetermined brightness) from the image data acquired in the process S1, and determines the bright spot closest to the position where the user's pupil is detected. The center is determined to be a Purkinje statue. Hereinafter, the coordinates of this Purkinje image are set as u0. Further, the distance from the camera C to the Purkinje image is obtained from the focal length of the eyeball detection unit 14 at the position where the camera C is most focused. Then, the position vector u of the Purkinje image in the user coordinate system Wu can be obtained from the known method using the coordinates u0 of the Purkinje image, the distance from the camera C to the Purkinje image, and the internal and external parameters of the camera C.

  Further, the calibration processing unit 32 calculates the position vector of the pupil center in the user coordinate system Wu from the coordinate position pu on the image data of the pupil center, the distance from the camera C to the pupil center, and the internal and external parameters of the camera C. Find P. Here, the distance from the camera C to the pupil center may be approximately calculated as follows. That is, for a person assumed as a user, a distance l from the corneal surface to the position of the pupil (which may be different between a child and an adult) is set as a model in advance. A value obtained by adding this distance l to the distance from the camera C to the Purkinje image is used as the approximate distance from the camera C to the pupil center.

  Next, the calibration processing unit 32 obtains the corneal curvature center from the Purkinje image. Here, an average corneal curvature radius R of a person assumed as a user is determined in advance. Then, the vector r (viewpoint position) of the corneal curvature center in the user coordinate system Wu is given by the following equation (1).

In (1), || u || represents the norm of u.
The calibration processing unit 32 uses the position vector P of the pupil center in the user coordinate system Wu and the vector c of the corneal curvature center as V = Pr, and the user's line-of-sight vector V in the user coordinate system Wu. Ask for.

Next, the calibration processing unit 32 converts the transformation matrix [Twu, c2] (= [Tc2, wu) from the user coordinate system Wu to the camera coordinate system Cc2 of the camera C, which is obtained for the camera C of the eyeball detection unit 14. ], Where [Tc2, wu] is a transformation matrix from the camera coordinate system Cc2 of the camera C to the user coordinate system Wu) and the user coordinate system Wu in the camera coordinate system Cc2 of the camera C. A user's line-of-sight vector Vc2 and a pupil center position vector Pc2 are obtained (S3). That is,
Vc2 = [Twu, c2] V
Pc2 = [Twu, c2] P
And

The calibration processing unit 32 converts the user's line-of-sight vector Vc2 and pupil center position vector Pc2 in the camera coordinate system Cc2 into the user's line-of-sight vector Vc1 and pupil center position vector Pc1 in the coordinate system Cc1 of the imaging unit 13. Conversion is performed (S4). This conversion is
Vc1 = [Tc1,2] -1Vc2
Pc1 = [Tc1,2] -1Pc2
Done in

Thus, as illustrated in FIG. 5, a coordinate system is defined in which the position vector Pc1 is the origin and the line-of-sight vector Vc1 is one coordinate axis (ξ axis). The calibration processing unit 32 further determines two coordinate axes (η axis and ζ axis) orthogonal to each other in a plane having the line-of-sight vector Vc1 as a normal vector, thereby defining the user's visual field coordinate system E ( S5). Specifically, the calibration processing unit 32 converts the position vector u of the Purkinje image (by the camera coordinate system Cc2) into a vector in the coordinate system Cc1 of the imaging unit 13. That is,
uc1 = [Tc1,2] -1u
And

Then, the calibration processing unit 32 generates a vector uc1−Vc1 in a plane having the line-of-sight vector Vc1 as a normal vector.
Is a vector orthogonal to both the ξ-axis and the η-axis, for example, u × Vc1 in a plane having the direction of the projection vector u when projected as the η-axis and the line-of-sight vector Vc1 as the normal vector.
Let the direction of the vector obtained in step be the ζ axis. In addition,
uc1−Vc1
Is zero vector (when the user happens to look in the direction of the near-infrared light projector P), the calibration processing unit 32 may return to the processing S1 and repeat the processing.

  The calibration processing unit 32 obtains a conversion matrix between the user's visual field coordinate system E and the external world coordinate system Wp obtained as described above, for example (S6). Since the user's visual field coordinate system is already defined by a vector in the coordinate system Cc1 of the imaging unit 13, a transformation matrix [TE, c1] from the coordinate system Cc1 of the imaging unit 13 to the visual field coordinate system E of the user Can be computed.

  When the external coordinate system Wp is the coordinate system Cc1 itself of the imaging unit 13, this conversion matrix [TE, c1] becomes the conversion matrix [TE, wp] from the external world coordinate system Wp to the visual field coordinate system E as it is.

  If the external coordinate system Wp is not the coordinate system Cc1 itself of the imaging unit 13, the calibration processing unit 32 uses the transformation matrix [TE, c1] to convert the external coordinate system Wp, which is the imaging unit 13, to the camera coordinate system. Multiply the transformation matrix [Tc1, wp] to Cc1 (= [Twp, c1] -1, where [Twp, c1] is the transformation matrix from the camera coordinate system Cc1 to the outer world coordinate system Wp) by multiplying it from the right. A transformation matrix [TE, wp] = [TE, c1] [Tc1, wp] from the system Wp to the visual field coordinate system E may be obtained.

Note that the transformation matrix from the visual field coordinate system E to the external world coordinate system Wp is given as an inverse matrix [TE, wp] -1 of the transformation matrix [TE, c1]. The calibration processing unit 32 also multiplies the transformation matrix [Tc1, Dp] with the coordinate system Dp of the transmissive display 15 by the transformation matrix [TE, c1], and calculates the inverse matrix to obtain the visual field coordinates. The transformation matrix [TDp, E] = [TE, Dp] -1 = [[TE, c1] [Tc1, Dp]]-1 for projecting the system E onto the coordinate system Dp (two-dimensional coordinates) of the transmissive display 15
Get.

  The transformation matrix [TE, wp] (or its inverse matrix [TE, wp] -1) from the external world coordinate system Wp to the visual field coordinate system E obtained here, and the coordinate system Dp of the transmissive display 15 from the visual field coordinate system E The conversion matrix [TDp, E] into is stored in the storage unit 12 as a calibration parameter (S7), and the calibration process is terminated. And the calibration process part 32 outputs the information showing that the process was complete | finished.

  The calibration processing unit 32 uses two cameras whose transformation matrices of the respective coordinate systems are known, detects a user's line-of-sight vector on one side, and coordinates of the outside world visually recognized by the user on the other side. Any method can be used as long as the transformation matrix between the user's visual field coordinate system that can be defined using the line-of-sight vector and the external field coordinate system can be obtained. Not only the method but also other methods may be used.

  When the information display unit 33 of the control unit 11 receives information indicating that the processing is completed from the calibration processing unit 32, the information display unit 33 starts the information display processing. In an example of the present embodiment, as shown in FIG. 6, a rectangular image portion (I) that should be in the outside world and information (character string) to be displayed corresponding to the image portion in advance in the storage unit 12. And a bitmap image or the like; R) and a database including entries that associate information indicating the actual size of the image portion (size information; for example, the diagonal length of the rectangle of the actual object) It shall be.

  As illustrated in FIG. 7, the information display unit 33 is a partial image in the range of an object detected in the external coordinate system Wp from the image data output from the imaging unit 13 by a markerless augmented reality method such as PTAM. To extract. Here, for example, a rectangular range (each of which has a plurality of rectangles) is detected, a partial image included in the rectangular range is extracted (S11), and the detected rectangular partial image is converted into a rectangular shape. Execute the process. For this process, a process widely known as an image process for correcting projection distortion can be used. The information display unit 33 further generates at least one image for collation from the obtained rectangular partial image (S12). As an example, the information display unit 33 generates an image obtained by rotating the obtained rectangular partial image by 90 degrees, 180 degrees, and 270 degrees, and four rectangular parts for each detected rectangular shape. Obtain a set of images (set i for the i-th rectangle).

  The information display unit 33 refers to the database stored in the storage unit 12 in advance, and an entry including an image that matches one of the four images included in the obtained set Si (i = 1, 2,...). It is checked whether there is an entry (whether there is an entry including an image for verification) (S13). If there is an entry including an image that matches any of the four images included in the set Sj (j = 1, 2,...) (If Yes), the information display unit 33 displays the display target included in the entry. Information and size information are taken out (S14). Then, in the image data picked up by the image pickup unit 13, the coordinates of the position where the rectangular shape that is the source of the partial image included in the set Sj is found, the size of the found rectangular shape, and the size information extracted from the database Are used to obtain the rectangular coordinate information σj (a set of coordinates of four vertices, etc.) in the coordinate system Cc1 of the imaging unit 13 (S15). Since the calculation of this information is also widely known, detailed description thereof is omitted here.

  When the external coordinate system Wp is the coordinate system Cc1 itself of the imaging unit 13, the information display unit 33 converts the transformation matrix [TE, wp] as a calibration parameter for the coordinate information included in σj, that is, The transformation matrix [TE, c1] is multiplied to transform into coordinate information σ ″ j in the user's visual field coordinate system E. Further, the information display unit 33 has an external coordinate system Wp that is different from the coordinate system Cc1 of the imaging unit 13. If they are different, the coordinate information contained in this σj is multiplied by a conversion matrix [TWp, c1] to convert the information of these coordinates into coordinate information σ′j in the external coordinate system Wp, and further calibrated to this. Multiplying by the conversion matrix [TE, wp] as an operation parameter, it is converted into coordinate information σ ″ j in the user's visual field coordinate system E (S16). According to the latter method, the information display unit 33 is provided with coordinate information in the visual field coordinate system E of the user if given the coordinate information σj of the object in the external coordinate system Wp different from the coordinate system Cc1 of the imaging unit 13. σ ″ j can be obtained. In other words, the coordinate information σj of the object can be obtained by a device other than the imaging unit 13 and information processing can be performed.

  Further, the information display unit 33 uses the transformation matrix [TDp, E] as the calibration parameter to convert the coordinate information included in the coordinate information σ ″ j into the point coordinates Q1 (x1, y1) on the transmissive display 15. , Q2 (x2, y2), etc. (S17) The information display unit 33 will now calculate, for example, the average value Qave (x, x2) of these point coordinates Q1, (x1, y1), Q2 (x2, y2),. y) = ((x 1 + x 2 +... + xn / n), (y 1 + y 2 +... + yn / n)) is calculated from the database in the vicinity of Qave (x, y) (position separated by a predetermined number of pixels). The retrieved display target information is displayed (S18).

  Note that steps S12 to S18 are repeatedly performed in a loop for each rectangle extracted in step S11. If there is no corresponding entry in the process S13 (if No), the process for the next rectangle is performed. If the process has been performed for all rectangles, the information display unit 33 ends the process.

  In this way, in this embodiment, the user's line-of-sight vector is detected, and the user's visual field coordinate system is set with the detected line-of-sight vector as one of the coordinate axes. The position of the object in the user's field of view is also expressed in this field of view coordinate system, and then the coordinates are converted from the field of view coordinate system to the coordinate system in the display surface of the transmissive display 15 to obtain information on the object. Get the display position. Then, by arranging the information regarding the object at the obtained position, the information is displayed at the position in consideration of the direction of the user's line of sight.

  In the above description of the present embodiment, the user periodically performs calibration after wearing the information display device 1 of the present embodiment. However, the present embodiment is not limited to this. That is, when the information display device 1 has the shape of eyeglasses, the displacement after wearing is considered to be slight, so that the wearing sensor 16 detects that the user has worn the information display device 1 of the present embodiment on the head. The calibration instruction unit 31 gives an instruction to perform the calibration process when the mounting sensor 16 detects that the information display device 1 is mounted on the user's head. It is good also as outputting.

  Specifically, the mounting sensor 16 may be, for example, a proximity sensor that radiates infrared rays and detects that an object is approaching by the reflected light, and may be attached to the temple 21. The output of the wearing sensor 16 is L while the object is not close, and the output is H when the object is close.

  When the control unit 11 functions as the calibration instruction unit 31, when the output of the mounting sensor 16 changes from L to H and then the state of H continues for a predetermined time, the user receives information Assuming that the display device 1 is attached and the timing for performing calibration has arrived, an instruction to perform calibration processing is output.

  In addition, the information processing apparatus 1 according to the present embodiment further includes a displacement sensor 17 as means for detecting a relative position variation between the information processing apparatus 1 and the viewpoint position after the information processing apparatus 1 is mounted. Good. The displacement sensor 17 is, for example, as follows. That is, in this example, the information processing apparatus 1 has the shape of glasses. However, in the glasses, in general, the positional deviation after wearing is mainly information about the portion where the temple 21 is on the user's ear. This occurs when the processing device 1 rotates in the vertical direction (that is, in the elevation direction) with respect to the user's face. Therefore, the displacement sensor 17 may be an inclination sensor that is attached to the temple 21 and detects the elevation angle of the temple 21 with respect to the vertically downward direction. As an example, the displacement sensor 17 outputs a voltage signal proportional to the elevation angle.

  When the control unit 11 functions as the calibration processing unit 32, it generates elevation angle information θ 0 at the time of calibration processing from the voltage signal output from the displacement sensor 17, and stores it in the storage unit 12 as a calibration parameter. Keep it.

  Thereafter, the control unit 11 repeatedly generates information on the elevation angle θ from the voltage signal output from the displacement sensor 17 at every predetermined timing. Then, the generated elevation angle information θ is compared with the elevation angle information θ0 included in the calibration parameter stored in the storage unit 12, and the absolute value of the difference, Δθ = | θ−θ0 | Is generated. Here, | * | means the absolute value of *.

  The control unit 11 performs the function as the calibration instruction unit 31 and performs calibration when Δθ exceeds a predetermined threshold θth (when Δθ> θth or Δθ ≧ θth). Is output, an instruction to perform the calibration process is output.

  Further, the control unit 11 may perform the following process instead of performing the calibration process again when the threshold value is exceeded. That is, the transformation matrix is corrected by using the elevation angle information θ obtained from the output of the displacement sensor 17 repeatedly at predetermined timings and the elevation angle information θ0 at the time of calibration processing included in the calibration parameter. It is good.

Specifically, the control unit 11 corrects rotation of the external coordinate system in the elevation direction (angle direction in the vertical plane in the external coordinate system Wp) by θ−θ0 with respect to the transformation matrix [TE, wp] as the calibration parameter. Transformation matrix [TE, wp] ′ = [TE, wp] [Tθ] corrected by multiplying by the matrix [Tθ]
Is generated. In the example here, the transmissive display 15 as well as the imaging unit 13 is subjected to the same displacement. Therefore, the coordinate system of the transmissive display is similarly set to the elevation angle for the transformation matrix [TDp, E] as the calibration parameter. The transformation matrix [TDp, E] ′ = [Tθ] [TDp, E corrected by multiplying the direction (angle direction in the vertical plane in the coordinate system Dp of the transmissive display) by the rotation correction matrix [Tθ] by θ−θ0 ]
Is generated. Thereafter, the processing as the information display unit 33 is executed using the corrected conversion matrices [TE, wp] ′ and [TDp, E] ′.

  That is, as a process of the information display unit 33, the control unit 11 converts the corrected transformation matrix [TE, TE, into rectangular coordinate information σj (a set of coordinates of four vertices, etc.) detected in the external coordinate system Wp. is multiplied by wp] ′ to convert to coordinate information σ ″ j in the user's visual field coordinate system E. Also, the coordinates included in the coordinate information σ ″ j using the corrected conversion matrix [TDp, E] ′ Is converted into values of point coordinates Q1 (x1, y1), Q2 (x2, y2)... On the transmissive display 15. From now on, the information display unit 33, for example, the average value Qave (x, y) = ((x1 + x2 + ... + xn / n), (y1 + y2 + ... + yn /) of these point coordinates Q1 (x1, y1), Q2 (x2, y2). n)) is calculated, and display target information extracted from the database is displayed in the vicinity of Qave (x, y) (a position separated by a predetermined number of pixels).

  In this way, by setting the calibration timing to the attachment time point, the number of calibration processes can be reduced, and the processing load can be reduced. Such processing can be effectively applied when the direction of the user's eyeball does not change extremely, for example, when the vehicle is running.

  The information display device 1 of the present embodiment has the above configuration and operates as follows. In an example of the information display device 1 according to the present embodiment, an entry that associates an image of a signboard that can be visually recognized by the driver with information set in advance for each signboard when traveling on a road with a vehicle. A database including the data is stored in the storage unit 12.

  When the user wears this information display device 1, the information display device 1 then periodically performs a calibration process, detects the direction of the user's line of sight, and converts the transformation matrix [TE, wp as a calibration parameter. ] And a transformation matrix [TDp, E].

  In addition, the information display device 1 has a rectangular shape (a plurality of rectangles) detected in the external coordinate system Wp by using a markerless augmented reality method such as PTAM from image data obtained by imaging an object in the direction visually recognized by the user. In this case, the partial image included in each of them is extracted, and the detected rectangular partial image is converted into a rectangular shape. Further, an image obtained by rotating the obtained rectangular partial image by 90 degrees, 180 degrees, and 270 degrees is generated, and a set of four rectangular partial images for each detected rectangular shape (i th A set Si is obtained for a rectangular shape.

  The information display apparatus 1 refers to a database stored in the storage unit 12 in advance, and an entry including an image that matches one of the four images included in the obtained set Si (i = 1, 2,...) Check if there is any. For example, since the direction signboard on the road is rectangular, it is detected by this processing. Information that should be displayed in association with the image of the direction sign (for example, information on a store in the vicinity of the direction sign) is stored in the storage unit 12 of the information display device 1 in advance.

  When the information display unit 33 finds an entry including an image that matches one of the four images included in the set Sj (j = 1, 2,...), The information to be displayed, the size information, Take out. Then, in the image data picked up by the image pickup unit 13, the coordinates of the position where the rectangular shape that is the source of the partial image included in the set Sj is found, the size of the found rectangular shape, and the size information extracted from the database Are used to obtain the rectangular coordinate information σj (a set of coordinates of four vertices, etc.) in the external coordinate system Wp. That is, for example, information on a store near the direction signboard can be obtained.

  The information display device 1 multiplies the rectangular coordinate information σj by this conversion matrix [TE, wp] to convert it into coordinate information σ ″ j in the user's visual field coordinate system E. Also, the information display device 1. Uses the corrected transformation matrix [TDp, E] to convert the coordinate information included in the coordinate information σ ″ j into point coordinates Q1 (x1, y1), Q2 (x2, y2) on the transmissive display 15. Convert to the value of…. From now on, the information display device 1, for example, average values Qave (x, y) = ((x1 + x2 + ... + xn / n), (y1 + y2 + ... + yn /) of these point coordinates Q1 (x1, y1), Q2 (x2, y2). n)) is calculated, and display target information (such as information on nearby stores) extracted from the database is displayed in the vicinity of Qave (x, y) (a position separated by a predetermined number of pixels). .

[Modification]
In the above description of the present embodiment, as illustrated in FIG. 2, the case where the shape of glasses is formed, that is, the case where the information display device 1 is mounted as a head mounted display has been described as an example. The embodiment is not limited to this.

  For example, the transmissive display 15 of the present embodiment may be a vehicle windshield or the like. In this case, the imaging unit 13 and the eyeball detection unit 14 may be arranged on the windshield or a position where the relative position with respect to the windshield does not change (a windshield frame, a dashboard, or the like). Also by doing this, the transformation matrix [Tc1, Dp] between the coordinate system Cc1 of the imaging unit 13 and the coordinate system Dp of the transmissive display 15 is also known, and the coordinate system of the imaging unit 13 is also known. The transformation matrix [Tc1, Dp] between Cc1 and the coordinate system Dp of the transmissive display 15 is also known. Therefore, also in this case, by performing processing in the same manner as the example already described, it is possible to display information in consideration of the user's line-of-sight direction.

  Furthermore, in an example of the embodiment of the present invention, the transmissive display 15 may be arranged on a glass surface of a show window of a product or a showcase. In this case, as illustrated in FIG. 8, the imaging unit 13 and the eyeball detection unit 14 are arranged behind the stage where the product is displayed (a position opposite to the glass surface with respect to the product). In this example, the imaging unit 13 may also serve as the camera C of the eyeball detection unit 14. As shown here, when the display is arranged on the glass surface of a show window of a product or a showcase, the transmissive display 15 can project an image on the glass surface as well as the liquid crystal display panel and the organic EL panel described above. The projector may include a projector disposed at a position and a screen disposed on a glass surface. The screen in this case is a transparent or translucent screen on which an image projected by the projector is projected and an image on the opposite side is visible from the user side.

  The control unit 11 performs the calibration process by the same method as described above. Further, in this example, the coordinates of the product that is the object of information display can be determined in advance within the coordinate system Cc1 of the imaging unit 13 (here, the coordinate system Cc1 is directly the external world coordinate system Wp). Therefore, it is assumed that the coordinates in the coordinate system Cc1 of each vertex of the rectangular parallelepiped circumscribing these products are known. It is also assumed that the coordinates of the four vertices of the transmissive display 15 in the coordinate system Cc1 of the imaging unit 13 are known. Therefore, the transformation matrix [TDp, wp] from the coordinate system Cc1 of the imaging unit 13 to the coordinate system Dp of the transmissive display 15 is also known.

The control unit 11 detects the user's line-of-sight vector V in the calibration process, and obtains a transformation matrix [TE, wp] therefrom. And this and the known [TDp, wp]
[TDp, E] = [TDp, wp] [TE, wp] -1
Is calculated.

  In addition, the storage unit 12 stores, for each product, an entry for each product in which the coordinates of each vertex of a rectangular parallelepiped circumscribing the product and information to be displayed regarding the product are associated with each other as a database.

  Hereinafter, the control unit 11 reads one entry for each product stored in the database. Each time each entry is read, the coordinate information in the visual field coordinate system E of the user is obtained by multiplying the coordinates of each vertex of the circumscribed rectangular parallelepiped contained in the read entry by the transformation matrix [TE, wp]. Convert. Further, the information display device 1 uses the transformation matrix [TDp, E] to convert the coordinate information included in the coordinate information after the transformation into the point coordinates Q1 (x1, y1), Q2 (x2) on the transmissive display 15. , Y2)...

  Further, the control unit 11 obtains a rectangular area circumscribing a polygon connecting these points Q1 (x1, y1), Q2 (x2, y2)..., And in this rectangular area on the transmissive display 15, Information to be displayed regarding the product included in the read entry is displayed.

  According to this example, even if the user moves in front of the glass surface, the product information is displayed superimposed on the product for the user.

  Furthermore, the positions of the transmissive display 15 and the imaging unit 13 are not necessarily fixed. In this case, the control unit 11 detects the coordinates of the vertices on the outer periphery of the transmissive display 15 from the image data obtained by the imaging unit 13. The detection of the coordinates may be performed by attaching markers to two points on the diagonal line of the rectangular transmissive display 15 and detecting the markers in the coordinate system Cc1 of the imaging unit 13, for example. Accordingly, the control unit 11 obtains a conversion matrix [TDp, wp] from the coordinate system Cc1 of the imaging unit 13, that is, the external world coordinate system Wp, to the coordinate system Dp of the transmissive display 15.

  Further, in the description so far, the information to be displayed is held as a database in the storage unit 12, but is not limited thereto. For example, the database is stored in an external server, and the control unit 11 accesses the database via a wired or wireless LAN, or a cellular phone network, and searches for or reads the contents. I do not care.

  DESCRIPTION OF SYMBOLS 1 Information display apparatus, 11 Control part, 12 Storage part, 13 Imaging part, 14 Eyeball detection part, 15 Transmission type display, 16 Wear sensor, 17 Displacement sensor, 20 Frame, 21 Temple, 22 Rim, 23 Nasal pad part, 24 Bridge, 31 Calibration instruction section, 32 Calibration processing section, 33 Information display section.

Claims (5)

A transmissive display placed in front of the user's eyes;
An imaging unit;
Means for detecting a user's line-of-sight vector V;
Means for defining a user's visual field coordinate system Ep using the detected line-of-sight vector V;
Means for obtaining a first transformation matrix for transforming the coordinate system Cc of the imaging unit into coordinates in the external coordinate system Wp;
Positioning means for obtaining a second transformation matrix for transforming the coordinates in the visual field coordinate system Ep of the defined user into coordinates in the external world coordinate system Wp ;
The external world coordinate system including the coordinates of the object imaged by the imaging unit, using the first conversion matrix, into coordinates in the external world coordinate system Wp and including the coordinates of the converted object. Means for converting coordinates representing the position of the object in Wp into coordinates representing the position in the visual field coordinate system Ep by the second transformation matrix;
Means for projecting the coordinates of the visual field coordinate system Ep onto the transmissive display to obtain a display position of information on the object;
Means for displaying information on the object at the obtained display position;
An information display device. The information display device according to claim 1,
The information processing apparatus is mounted on the user's head,
A sensor for detecting that the head is mounted on the user's head;
The position alignment means is an information display device for obtaining the second transformation matrix when the sensor detects that the information display device is mounted by the sensor. The information display device according to claim 2,
And further comprising means for detecting a relative position variation, which is a positional shift after mounting the information processing apparatus and the viewpoint position, after detecting that the sensor is mounted on a user's head,
The information alignment apparatus that corrects the second transformation matrix based on the detected relative position fluctuation. In the information display device according to any one of claims 1 to 3,
Further, an information display device comprising means for obtaining a transformation matrix between the external world coordinate system Wp and the coordinate system Dp on the transmissive display. A computer having a transmissive display arranged in front of the user's eyes and an imaging unit ,
Means for detecting a user's line-of-sight vector V;
Means for defining a user's visual field coordinate system Ep using the detected line-of-sight vector V;
Means for obtaining a first transformation matrix for transforming the coordinate system Cc of the imaging unit into coordinates in the external coordinate system Wp;
Alignment means for obtaining a second transformation matrix for transforming coordinates in the external world coordinate system Wp into coordinates in the specified user visual field coordinate system Ep ;
The external world coordinate system including the coordinates of the object imaged by the imaging unit, using the first conversion matrix, into coordinates in the external world coordinate system Wp and including the coordinates of the converted object. Means for converting coordinates representing the position of the object in Wp into coordinates representing the position in the visual field coordinate system Ep by the second transformation matrix;
Means for projecting the coordinates of the visual field coordinate system Ep onto the transmissive display to obtain a display position of information on the object;
Means for displaying information on the object at the obtained display position;
Program to function as.

Images