JP5215211B2 - Related information display position specifying system and related information display position specifying program - Google Patents

Description

The present invention relates to a related information display position specifying system and related information display position specifying program, electronically by transmission head-mounted display additional information about the object relative to the object in the real environment to be viewed when adding displayed, their display position of the relevant information can be suitably identify, related information display position specifying system and related information display position specifying program.

  Conventionally, there is a non-transparent head mounted display (hereinafter referred to as HMD) in which a display is placed in front of the user when worn and covers the field of view and allows a user to view only the display screen. According to this non-transparent HMD, the user can confirm the electronic information presented without having to consciously turn his gaze, and realize a virtually large screen display using a small display. it can. Non-transmissive HMD (Vuzix iWear vr920) is described in Non-Patent Document 1.

  In recent years, a transmissive HMD capable of additionally displaying electronic information while securing a field of view has also been realized. Since the transmissive HMD can secure a field of view, it is expected to be applied to an augmented reality service that additionally displays related information about an object existing in the field of view on the HMD. The transmissive HMD (LUMUS) is described in Non-Patent Document 2.

http://www.vuzix.com/iwear/ products_vr920.html http://www.lumusvision.com/

  Appropriate information on the HMD so that it can be easily recognized that the related information added to the object is easily displayed when the related information about the object existing in the field of view is electronically added and displayed by the transmission HMD. It is necessary to additionally display related information at various positions. For example, when additional information related to an object is additionally displayed using a balloon, it is desirable that the start point of the balloon overlaps the position of the object. On the other hand, if the start point of the balloon does not overlap the position of the target object, it cannot be easily recognized which object the related information additionally displayed is related to.

  When displaying related information for an object on an electronic image, the object on the electronic image is recognized by object recognition technology or the like, and the position where the object is recognized is added as it is. A position suitable for display can be obtained.

  However, the object in the transmissive HMD is not on the electronic image, but is recognized in the user's field of view in the real environment. Therefore, when the related information about the object is additionally displayed, the above method is used. Thus, a suitable position for additionally displaying the related information cannot be obtained.

  FIG. 3 is an explanatory diagram showing a situation where an object in the real environment is visually recognized through the transmission type HMD. FIGS. 9A and 9B show a case where the object O is at a short distance and a case where the object O is at a long distance, respectively. As shown in the figure, the object O is visually recognized at a position relatively far from the center line of the eye on the HMD screen when it is at a short distance, and the center line of the eye when it is at a long distance. It is visually recognized at a relatively close position. Thus, the position of the object O on the HMD screen differs depending on whether the object O is at a short distance or a long distance, and a suitable position for adding related information to the object O. Is also different.

The object of the present invention is to solve the above-mentioned problems, and to display a related information about an object in a real environment visually recognized by the transmission HMD at a suitable position on the transmission HMD screen. it can be added display relevant information, and to provide a related information display position specifying system and related information display position specifying program.

In order to solve the above problems, a related information display position specifying system according to the present invention includes a two-dimensional image acquisition unit that acquires two-dimensional image data, and an object from the two-dimensional image data acquired by the two-dimensional image acquisition unit. Recognizing and identifying the specific information and the two-dimensional position information in the two-dimensional image, the depth image obtaining means for obtaining the depth image data, and the object identified by the image recognition means. Three-dimensional position acquisition means for acquiring three-dimensional position information of the object from the three-dimensional position information and the depth image data acquired by the depth image acquisition means, and unique information of the object specified by the image recognition means related information acquisition means, the display position of the relevant information acquired by the related information obtaining unit, before Symbol dimensional position acquiring means for acquiring relevant information about the object based on the Three-dimensional position information of the acquired object, the two-dimensional image acquisition unit and the user is not a position of the user's eye when viewing the object through the transmission head mounting display, and two-dimensional position information of the user's eye the display position specifying means for specifying with Re not a position in the depth direction of the eye of the user when viewing the transmission type head mounting display of the display screen and the user object through the transmissive head mounting display, wherein A transmissive head mounting display for displaying the related information acquired by the related information acquiring means at the display position specified by the display position specifying means, and the transmissive head mounting display can visually recognize the real environment through the display screen. Related information is displayed, and the display position specifying means is visible in an actual environment. It is characterized by identifying the balloon start of display of the display position or related information of the related information in the area of the object that.

In the related information display position specifying system according to the present invention, the display position specifying unit specifies the display positions of the related information acquired by the related information acquiring unit separately for the right eye and the left eye, and the transmission The type head mounting display is characterized in that the related information acquired by the related information acquiring means is displayed at the display position specified by the display position specifying means on the right-eye and left-eye screens, respectively. .

Further, in the related information display position specifying system of the present invention, the display position specifying means determines the display position of the related information acquired by the related information acquiring means based on the position of the center of both eyes or the position of the dominant eye. The transmissive head mounting display displays the related information acquired by the related information acquisition unit at the display position specified by the display position specifying unit on the right-eye and left-eye screens. It is characterized by.

  Furthermore, in the related information display position specifying system of the present invention, the display position specifying unit displays the related information display position so that the start point of the related information balloon or the display center of the related information overlaps an object visually recognized in the real environment. It is characterized by specifying.

The present invention is not only a related information display location system can also be implemented as a related information display position specifying program.

  ADVANTAGE OF THE INVENTION According to this invention, according to the position of the target object in the real environment which a user visually recognizes through transmission type | mold HMD, the relevant information of the target object can be displayed on the suitable position on a HMD screen. Therefore, the user can easily and correctly recognize the correspondence between the related information additionally displayed on the HMD screen and the object viewed in the real environment.

It is a functional block diagram which shows one Embodiment of this invention. It is explanatory drawing which shows the identification method of the display position of related information. It is explanatory drawing which shows the condition where the target object of a real environment is visually recognized through transmissive HMD.

  The present invention will be described below with reference to the drawings. FIG. 1 is a functional block diagram showing an embodiment of the present invention. In this embodiment, the present invention is realized as a related information display position specifying system, and a two-dimensional image acquisition unit 11, an image recognition unit 12, a depth image acquisition unit 13, a three-dimensional position acquisition unit 14, and a related information acquisition Means 15, display position specifying means 16, display control means 17, and display means 18 are provided. The image recognition unit 12, the three-dimensional position acquisition unit 14, the related information acquisition unit 15, the display position specification unit 16, and the display control unit 17 function as a related information processing apparatus, and are provided in, for example, a mobile phone. Each of these means can be realized by hardware or software.

  The two-dimensional image acquisition means 11 acquires two-dimensional image data with a camera. The camera may be mounted on a transmissive HMD (hereinafter simply referred to as HMD) or may be mounted on a related information processing apparatus. Moreover, it is good also as an independent apparatus.

  The image recognizing unit 12 specifies the unique information of the object and the two-dimensional position information in the two-dimensional image from the two-dimensional image data acquired by the two-dimensional image acquiring unit 11. The unique information and the two-dimensional position information of the object can be specified based on, for example, a comparison between the two-dimensional image data of a previously registered object and the acquired two-dimensional image data and the similarity between the two image patterns. it can. Note that the two-dimensional position of the target object specified here may be an appropriate position in the target object region, such as the center or the edge of the target object.

  The depth image acquisition means 13 acquires depth image data. The depth image data is, for example, the depth (Z value) for each pixel acquired by the depth recognition camera. The depth recognition camera is a camera capable of recognizing depth information. For example, ZCAM (http://www.3dvsystems.com/) can be used.

  The three-dimensional position acquisition unit 14 acquires a Z value corresponding to the two-dimensional position among the two-dimensional position information of the object identified by the image recognition unit 12 and the Z value acquired by the depth image acquisition unit 13. Thus, the three-dimensional position information of the object is acquired. Here, if the camera that acquires the 2D image data and the depth recognition camera are located at different positions, or if their skewness is different, the Z value is acquired by associating the 2D positions with each other. To do.

  It should be noted that the Z value is first acquired by the depth image acquisition means 13, and the image recognition means 12 performs image recognition only when the number of pixels at a two-dimensional position within a certain range (fixed depth) is greater than a certain number. It is good also as a structure to perform. In this case, the Z value acquired by the depth image acquisition means 13 is given to the image recognition means 12.

  The related information acquisition means 15 acquires related information about the object based on the unique information of the object. For example, the related information of the object is stored in advance in a database (DB) in association with the unique information of the object, and the related information is acquired from the DB using the unique information of the object as a search key. The related information is, for example, a name or detailed information about the object. Further, it may be information indicating the acquisition destination of detailed information about the object, for example, a pointer (reference destination information) such as a URL of a Web site. The DB may be provided in the related information acquisition unit 15 or may be referred to via a network.

  The display position specifying means 16 is an HMD screen that displays related information about the object based on the position of the eye of the user wearing the HMD and the 3D position information of the object acquired by the 3D position acquisition means 14. Identify the top position. This position may be the area of the object visually recognized in the real environment or the vicinity thereof. A method for specifying the display position of the related information will be described in detail later.

  The display control means 17 performs display control so that related information about the object is displayed at the position specified by the display position specifying means 16.

  The display means 18 is configured as a transmissive HMD, and displays related information about the object according to the display control of the display control means 17. Thereby, the relevant information about the object is electronically displayed in front of the user. The additional display of the related information about the object can be performed according to the user instruction, the setting of the related information additional display mode, or the like. Of course, the related information is not additionally displayed for an object whose related information is not stored in the DB. The display means 18 is connected to the display control means 17 with an image output cable. This connection method may be wired connection or wireless connection such as Bluetooth, IEEE802.11, Wireless HD.

  FIG. 2 is an explanatory diagram illustrating a method for specifying a display position of related information. Here, the positions related to the Y and Z axes are shown, but the positions related to the X and Z axes are also the same. The display position specifying means 16 specifies the position specified by the calculation method shown below as the display position of the related information on the HMD screen.

  The three-dimensional position of the object O acquired by the three-dimensional position acquisition means 14 in the X-axis, Y-axis, and Z-axis directions is (x, y, z), and one eye of the user wearing the HMD (for example, right Let (u, v) be the two-dimensional position of the intersection point on the HMD screen when a perpendicular line is drawn from the eye) to the HMD screen (lens). The distance between the position of the user's right eye and the HMD screen is r, and the position of the two-dimensional image acquisition means (camera) 11 when the position of the user's right eye is the origin is (dx, dy, dz). The two-dimensional position (u, v) of the user's right eye is known, and the position (dx, dy, dz) of the two-dimensional image acquisition means 11 and the distance r between the position of the user's right eye and the HMD screen are also included. It is decided uniquely.

  When the position of the right eye of the user is the origin, the three-dimensional position of the object O when the user visually recognizes the real environment through the HMD is (x + dx, y + dy, z + dz), and the object O is a point P where the line segment connecting the position of the right eye of the user and the three-dimensional position of the object O intersects the HMD screen on the HMD screen (lens: image projection plane). The X coordinate and Y coordinate of the point P are calculated by equations (1) and (2), respectively. Note that * represents multiplication. If the user's face and the camera direction do not match, it is necessary to further correct the deviation of the two-dimensional position (x, y) due to the mismatch.

  X coordinate of point P: u + (x + dx) * r / (z + dz) (1)

  Y coordinate of point P: v + (y + dy) * r / (z + dz) (2)

  Therefore, the display center of the related information about the object O or the start point of the related information display balloon is specified as the point P. That is, the related information may be displayed at the position of (u + (x + dx) * r / (z + dz), v + (y + dy) * r / (z + dz)) (length: unit m) on the HMD screen. In the case of processing on a computer, coordinates obtained by multiplying the length into the number of pixels are set as electronic display coordinates. For example, if the size of the HMD screen is 0.02 m in the X-axis direction and 0.015 m in the Y-axis direction and the number of pixels is 800x600, the electronic display coordinates are ((u + (x + dx) * r / ( z + dz)) / 0.02 * 800, (v + (y + dy) * r / (z + dz)) / 0.015 * 600) (pixels).

  The above is the specification of the display position of the related information for the user's right eye, but the display position of the related information is also specified for the left eye. In this way, the display position of the related information for the right eye and the left eye of the user is calculated separately and specified for the right eye display and the left eye display of the HMD, respectively. Can be made. In order to simplify the configuration and processing, the user's eye position is considered as the middle position of both eyes or the dominant eye position, the display position of related information is calculated, and this is the right eye display of the HMD. You may make it specify with respect to both left-eye displays.

  By calculating and specifying the display position of the related information of the object as described above, the related information is displayed in the area of the object visually recognized in the real environment or in the vicinity thereof. The correspondence can be easily understood.

Although the embodiment has been described above, the present invention is not limited to the above embodiment and can be variously modified. For example, the above-described embodiment, the present invention but which was realized as related information display locating system, the present invention is, in addition to the system, it was mounted on a mobile phone as related information display location system It can also be realized as a functioning application program.

11 ... 2D image acquisition means, 12 ... Image recognition means, 13 ... Depth image acquisition means, 14 ... 3D position acquisition means, 15 ... Related information acquisition means, 16 ... .Display position specifying means, 17 ... display control means, 18 ... display means

Claims (5)

2D image acquisition means for acquiring 2D image data;
An image recognition means for recognizing an object from the two-dimensional image data acquired by the two-dimensional image acquisition means, and identifying the unique information and the two-dimensional position information in the two-dimensional image;
Depth image acquisition means for acquiring depth image data;
Three-dimensional position acquisition means for acquiring three-dimensional position information of the object from the two-dimensional position information of the object specified by the image recognition means and the depth image data acquired by the depth image acquisition means;
Related information acquisition means for acquiring related information about the object based on the unique information of the object specified by the image recognition means;
The display position of the relevant information acquired by the related information obtaining unit, three-dimensional position information of the object acquired by the previous SL three-dimensional position acquiring means, the two-dimensional image acquisition unit and the user through the transmission head mounting display shift position of the user's eye when viewing the object, the user when viewing the object two-dimensional position information and the display screen and the user of the transmission head mounting display of the user's eyes through the transparent head mounting display Display position specifying means for specifying using a positional shift in the depth direction of the eye;
A transmissive head mounting display for displaying the related information acquired by the related information acquiring means at the display position specified by the display position specifying means;
The transmission type head mounting display displays related information so that the real environment can be visually recognized through a display screen, and the display position specifying means displays the display position of the related information or the related information in the region of the object visually recognized in the real environment. A related information display position specifying system characterized by specifying a balloon start point of information display. The display position specifying unit specifies the display position of the related information acquired by the related information acquiring unit separately for the right eye and the left eye, and the transmissive head mounting display is acquired by the related information acquiring unit. 2. The related information display position specifying system according to claim 1, wherein the related information is displayed at the display positions specified by the display position specifying means on the right-eye screen and the left-eye screen, respectively. The display position specifying means specifies the display position of the related information acquired by the related information acquisition means based on the position of the center of both eyes or the position of the dominant eye, and the transmission type head mounting display 2. The related information display according to claim 1, wherein the related information acquired by the information acquiring means is displayed at a display position specified by the display position specifying means on the right-eye screen and the left-eye screen. Location system.   4. The display position specifying unit specifies a display position of related information so that a start point of a related information balloon or a display center of related information overlaps an object visually recognized in a real environment. The related information display position specifying system according to any one of the above. On the computer,
A first function for acquiring two-dimensional image data;
A second function for recognizing an object from the two-dimensional image data acquired by the first function, and specifying the specific information and the two-dimensional position information in the two-dimensional image;
A third function for acquiring depth image data;
A fourth function for acquiring the three-dimensional position information of the target object from the two-dimensional position information of the target object specified by the second function and the depth image data acquired by the third function;
A fifth function of acquiring related information about the object based on the unique information of the object specified by the second function;
Wherein the display position of the fifth related information acquired by the function of the previous SL fourth three-dimensional position information of the object acquired by the function, location and user transmission head the two-dimensional image data is acquired viewing the object shift position of the user's eye when viewing the object, the display screen and the user of the two-dimensional position information of the user's eye and said transmissive head mounting display through a transmission type head mounting display through mounting display a sixth function of identified using Re not a position in the depth direction of the eye of the user when,
Realizing a seventh function of displaying related information acquired by the fifth function at a display position of a transmissive head mounting display specified by the sixth function;
The seventh function displays related information so that the real environment can be visually recognized through the transmissive head mounting display, and the sixth function displays a display position of the related information or an area of the object visually recognized in the real environment. A related information display position specifying program for specifying a balloon start point for displaying related information.