JP2017216522A - Head-mounted augmented reality display device and augmented reality display system using the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a head-mounted augmented reality display device for allowing a wearer to freely walk in a real space and an augmented reality display system capable of automatically recognizing a marker disposed at each location before converting information or image/video desired to be presented at the location to AR to supply the AR to the real space.SOLUTION: An augmented reality display system includes: a head-mounted augmented reality display device (10) that provides an open state without a cover for one eye so that the eye can directly view a real space, and displays augmented reality only for the other eye using a portable terminal (20) including an integrated camera, arithmetic processing unit, and surface screen to be used for displaying the augmented reality for the other eye; a marker (30) disposed on an object; and signal processing means for automatically recognizing the marker (30) displayed on the camera of the portable terminal (20) and displaying added and modified information related to the marker (30) at a corresponding position on video as augmented reality to enable a wearer 8 to view the information superposed on the object in the real space or positioned at any peripheral position.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a head-mounted augmented reality display device and an augmented reality display system using the head-mounted augmented reality display device, and more particularly, to a head-mounted augmented reality display device having a very simple and inexpensive configuration, and using the same Related augmented reality display system.

  Various devices and systems for displaying augmented reality (referred to as augmented reality: AR) and virtual reality (referred to as virtual reality: VR) have been proposed (Patent Documents 1 to 5).

  In recent years, an inexpensive box-type scope (product name “Hakosco”) using a smartphone as a display device is also sold as a head-mounted display (head-mounted display: HMD) for AR / VR display.

JP 2012-73920 A JP 2014-164482 A JP 2014-170374 A Japanese Patent Application Laid-Open No. 2015-177397 JP 2006-5125 A

  However, in the case of an eyeglass-type HMD developed exclusively for AR, a dedicated display area occupies a part of one eye field of view, and the displayed virtual screen is smaller than the real field of view and can be presented. The amount is small. In addition, since the display position is fixed within the field of view, it cannot be displayed around the object. Furthermore, there is a problem that the visual field blocked by the virtual screen cannot be stereoscopically viewed.

  On the other hand, in the case of an HMD that completely blocks the visual field, developed for VR that does not require direct viewing of the real space, the virtual screen can be taken large in the forward visual field, but both eyes are covered and the peripheral visual field. Is blocked, so the sense of distance cannot be grasped and the real space cannot be grasped. Therefore, it is fearful to move around and cannot move, and there is a danger in walking. In this type of HMD, an inexpensive box-type scope using a smartphone as a display device is also sold. In this case, since the image taken with a single-lens camera of a smartphone is viewed with both eyes, There is a problem that the parallax must be created by processing the video in real time.

  The present invention has been made to solve the above-described conventional problems, and a first object thereof is to provide a head-mounted augmented reality display device that allows a wearer to freely walk around a real space. .

  The present invention also uses this head-mounted augmented reality display device to automatically recognize markers placed in various places and superimpose information or images / videos to be presented in those places as an AR in real space. A second problem is to provide an augmented reality display system that can perform the above-described process.

  The present invention is a head-mounted augmented reality display device that is in an open state without a shield so that one eye can directly see the real space and displays augmented reality only for the other eye. The head-mounted augmented reality display device solves the first problem by using a portable terminal in which a camera, an arithmetic processing unit, and a display screen are integrated to display augmented reality for the other eye. It is a thing.

  Here, an object on the line of sight of an eye looking directly into the real space is photographed with the camera of the mobile terminal with an appropriate parallax, and the raw video or the video modified to the real video to realize augmented reality It can be displayed in real time on the screen of the portable terminal, and stereoscopic viewing with both eyes is possible.

  Further, a lens is disposed between the mobile terminal and the eye, and the lens and the mobile terminal are set so that the position of the virtual image on the display screen can be set to a distance similar to that of the object of the eye looking directly at the real space. The distance can be adjusted.

  In addition, it is possible to place a horizontally long mobile terminal tilted with respect to the front so that the distance between the lens and the mobile terminal screen does not change greatly between the front direction and the peripheral direction of the eye viewing the display screen. Can do.

  In addition, the position of the portable terminal with respect to the wearer can be adjusted.

  Further, when the position of the virtual image is adjusted to the same distance as the object of the eye looking directly in the real space, the image displayed on the screen of the mobile terminal is enlarged or reduced so that the direct view of one eye and the other It is possible to have a function of adjusting the size of the virtual image of the object to be approximately the same as the size of the direct view so that the virtual images of the eyes of the subject are integrated in the brain and stereoscopic viewing by binocular vision is possible.

  The present invention also automatically recognizes the head-mounted augmented reality display device, the marker disposed on the object, and the marker reflected on the camera of the mobile terminal, and sets the marker as an augmented reality at the corresponding location of the image. And signal processing means for displaying the information related to the item with additional modification, so that the wearer can see the information in an arbitrary surrounding position, overlapping the object in the real space. The augmented reality display system using the head-mounted augmented reality display device solves the second problem.

  According to the present invention, a head-mounted augmented reality display device can be realized at a low cost with a simple configuration by using a mobile terminal (for example, a smartphone) in which a camera, an arithmetic processing unit, and a display screen are integrated as a display device. . Here, since one eye is open for real vision, it becomes possible to walk around in a real space with an obstacle without feeling danger. For example, at a HEMS certification support center where a large number of home appliances are placed, a marker such as a color bit is affixed to each home appliance, and the device is viewed while the user is working or the visitor is willing The related information can be read using a balloon or the entire screen. In addition, it can be used by being incorporated into a visual inspection process of a production line that is increasingly demanding. Further, for example, in a residential exhibition hall, a visitor wearing an HMD can walk around without feeling dangerous even if he / she does not have his hand taken by a guide. Also, for example, in concert venues where AR is introduced, in the past, there was no choice but to use HMD with both eyes closed in order to see the AR effect. Because you can see the real space, you can maintain a sense of reality. Furthermore, it is also effective when viewing constellations and the like, and smartphone operations can be performed while walking.

The perspective view which shows the structure of the one eye type | mold AR scope which is embodiment of the head mounted augmented reality display apparatus which concerns on this invention. The perspective view which similarly shows the display state of a screen The figure which shows the relationship between the virtual image position and size when using the same lens Figure for examining how the display screen looks when the mobile device is also placed at an angle Similarly, a plan view showing when the screen position of the mobile terminal is moved away Plan view showing the same approach The figure which shows the example of the marker which can be used with the augmented reality display system which concerns on this invention The block diagram which shows the structure of embodiment of the augmented reality display system which concerns on this invention. Drawing substitute photograph showing a state using the augmented reality display system

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, this invention is not limited by the content described in the following embodiment and an Example. In addition, the constituent elements in the embodiments and examples described below include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those in the so-called equivalent range. Furthermore, the constituent elements disclosed in the embodiments and examples described below may be appropriately combined or may be appropriately selected and used.

  FIG. 1 shows a one-eye AR scope 10 that is an embodiment of a head-mounted augmented reality display device according to the present invention.

  In this embodiment, since the smartphone 20 is held in a position corresponding to one of the eyes (the left eye in the figure) of the frame 12 that is worn on the head of the wearer and the barrel 12 is extended. Unit 13 and smartphone holder 14, a magnifying glass 22 having a lens 23 disposed between the smartphone 20 of the frame 12 and one eye (left eye) of the wearer, and the frame 12 on the wearer's head And a belt 16 for fixing to the belt.

  The frame body 12 is formed into a complete cylindrical shape by, for example, cutting and attaching a plastic plate. Here, by making the frame body 12 transparent, it is possible to widen the real field of view and increase the safety of movement.

  The smartphone holder 14 can be adjusted in position and angle with screws (not shown) using two grooves 13A formed in the frame extension 13.

  As the lens 23, for example, a commercially available magnifying glass 22 of about 2.5 times can be used.

  As the smartphone 20, for example, an Android terminal having a screen size of 5 inches class can be used.

  The one-eye AR scope 10 is worn in front of the wearer's eyes using the smartphone 20 as a display device in the same manner as the box-type scope, but the main difference is that one eye (for example, the right eye) is completely open. The real space is actually viewed, and the visual field of the other eye (for example, the left eye) is closed with the display device of the smartphone 20.

  Since the right eye does not block the peripheral visual field, the surroundings such as the feet and the right foot are visible as a real visual field, and there is no problem with walking around the real space where there is an obstacle. For the left eye, the field of view captured by the camera of the smartphone 20 placed on the left side is displayed through the lens 23, and if the display is enlarged or reduced so that the virtual image is the same as the actual size, the left and right images are displayed. It is integrated in the brain and can be viewed stereoscopically without a sense of incongruity. That is, there is no need to create a stereoscopic parallax by calculation. In addition, since the smartphone 20 has a horizontally long screen, the peripheral visual field at the left end can be put into the left eye by an image captured by the smart phone 20, and the peripheral visual field can be covered with both eyes without any problem.

  When viewing a virtual image, the one that gives a wide field of view without vignetting is a single lens configuration (same as a magnifying glass), but for a field that deviates from the optical axis (center direction), the virtual image distance and virtual image magnification increase toward the periphery. There is also an optical problem of becoming. For this, the focal length f of the lens 23 may be made slightly longer to reduce the amount of change in the virtual image distance and the virtual image magnification.

  Specifically, as shown in FIG. 3, when the position a where the screen 20A of the smartphone 20 is placed is brought close to the focal distance f of the lens 23, the distance d to the virtual image 21 increases and the virtual image magnification also increases.

The distance d from the lens 23 to the virtual image 21 is given by the following equation (1).

The size h ′ of the virtual image 21 is given by the following equation (2).

When the distance d from the lens 23 to the virtual image 21 is determined, the distance a from the lens 23 to which the screen 20A of the smartphone 20 should be installed is as the lens formula of the following equation (3).

  Assume that the distance d to the virtual image 21 is 1 m, which is easy to see by direct viewing. In a commercially available box scope, the focal length f of the lens 23 was short, and there was almost no margin. That is, as shown by the slanted arrow line A in FIG. 3, when the eye 9 is viewed at an angle away from the center line, the distance a to the screen 20A increases to a ′. The distance d to 21 and the size h ′ of the virtual image 21 change abruptly and are very difficult to see.

  Therefore, in this embodiment, the focal length f of the lens 23 is set to be a little longer, 100 mm, in order to increase the AR field of view and smoothly overlap the real field of view of the right eye. That is, if the virtual image distance d is made equal to the actual viewing distance (for example) 1 m, since d = 1000 mm, the distance a from the lens 23 to the screen 20A when the eye 9 sees the smartphone 20 horizontally is (3 ) From the equation, 91 mm, and the margin to the focal length f is 9 mm.

On the other hand, when the upper end 20U of the screen 20A is observed as shown by an arrow line A extending obliquely from the eye 9 to the screen 20A in FIG. 3, the distance a ′ from the lens 23 to the gazing point (20U) is, for example, the screen size. When using a smartphone with 4.7 inches (screen height h = 59 mm when placed horizontally, screen width w = 104 mm), a point that is h / 2≈30 mm up at the horizontal distance a is the two sides of a right triangle. Therefore, the following formula is obtained.

Therefore, the distance d ′ from the lens 23 to the virtual image 21 when the upper end 20U is watched is expressed by the following equation (3).

  That is, when the screen 20A is viewed horizontally (right front of the eye 9 = center axis), the virtual image distance is d = 1 m, whereas when the upper end 20U of the screen 20A is viewed, the virtual image distance is d ′ = 2.4 m. Become. The focus of the eye 9 becomes deeper as the distance to the object increases (that is, the focus is greatly different when the distance to the object is 20 cm and 40 cm, but the focus is not greatly changed between 1 m and 2 m). At 2.4 m, you can focus on your eyes without burden.

  Moreover, regarding the horizontal direction, if the smartphone 20 is installed obliquely, the change in the position a of the smartphone 20 can be reduced.

  That is, when the smartphone 20 is arranged horizontally in the single-eye AR scope 10, the distance to the screen 20A corresponding to the front of the eye 9 (the distance from the lens 23 to the screen 20A on the center axis of one eye) and the lens 23 to the screen Since the distance to the right end and the left end of 20A is greatly different, the range that can be viewed as a virtual image is smaller than the entire screen, and the scenery around the field of view becomes invisible.

  As shown in FIG. 4, for example, a smartphone 20 having a focal length f = 100 mm, a distance a = 85 mm from the lens 23 to the screen 20A at the central axis, and a screen width w = 104 mm is parallel to both eyes (parallel to the face). Assuming that the right end 20R of the screen has reached the center of both eyes 9R and 9L, the distance a′1 = 91 mm from the lens 23 to the right end 20R of the screen and the distance a′2 = 111 mm from the left end 20L of the screen. Thus, the left end 20L exceeds the focal length f = 100 mm, and a virtual image cannot be viewed.

  On the other hand, if the smartphone 20 is also tilted at a tilt angle θ = 12 ° (with the left end 20L closer to the face and the right end 20R moved away) with a = 85 mm, the right end 20R of the screen is at the center of both eyes. Since the distance a′1 = 97.5 mm from the lens 23 to the right end 20R of the screen and the distance a′2 = 98.7 mm from the left end 20L of the screen, both the ends 20R and 20L are within the focal length f = 100 mm. You can see a virtual image. At this time, the virtual image distance at the center axis of one eye is 57 cm, the virtual image distance at the right end 20R of the screen is 388 cm, and the left end 20L is 732 cm, so that the entire screen can be viewed with almost the same (eye) focus centered on the distance 1 m. .

  Since the virtual image distance d is determined by the distance a between the lens 23 and the display screen 20A, the position of the display screen 20A (smart phone 20) can be changed back and forth so that it is the same as the distance of the object to be seen with real vision. It is desirable to keep it.

  Therefore, in this embodiment, as shown in FIG. 5 and FIG. 6, the screen position “a” and the angle of the smartphone 20 can be changed, and the virtual image position is made the same according to the distance from the object that the user wants to see. So that the wearer can adjust.

  When the virtual image position is determined, the size h ′ of the virtual image 21 is also uniquely determined. However, if the enlargement / reduction function of the screen 20A of the smartphone 20 is used, the size h ′ of the real object and the virtual image 21 can be made substantially the same. Therefore, the wearer can view stereoscopically with the left and right eyes without a sense of incongruity.

  The binocular integration function in stereoscopic vision of the brain is extremely plastic, so even if there is a slight difference in size between the real vision and the object of the smartphone image, or it is shifted up and down, it can be stereoscopically viewed immediately without any sense of incongruity It becomes like this.

  Since the AR screen can be superimposed on the real vision with a new one-eye AR scope 10 in which the right eye is real vision and the left eye is AR vision, normal markers can be used when generating the AR screen. . In this embodiment, a color bit marker 30 that performs code conversion by recognizing an image of the arrangement order of cells of three colors of red, blue, and green as shown in FIG. 7 is used as a marker. The color bit marker 30 is easily affixed to a three-dimensional object as (1) a marker. Since there is no restriction on the shape, it can be recognized even if it is bent as shown in FIG. (2) The color bit can be read by the camera of the smartphone 20, and a dedicated device is not necessary for marker recognition. (3) The color bit recognition program runs on the smartphone 20 and can display the AR information on the camera image in real time on the screen 20A of the smartphone 20 in real time. In addition, since a plurality of color bit markers can be simultaneously recognized, it has an advantage that a balloon can be AR-combined for a plurality of target devices. A QR code (registered trademark), bar code, RFID, or the like can also be used as a marker.

  As already mentioned, if the camera image of the smartphone 20 is appropriately enlarged and reduced and displayed on the screen for the left eye, it is integrated and stereoscopically viewed in the brain, so the camera image is analyzed in real time. If so, the software can recognize the color bit marker 30, and information corresponding to the color bit marker 30 can be inserted into the display image by a balloon, or a video effect can be applied to the image. At this time, in order to make the optical axis of the camera of the smartphone 20 coincide with the line of sight of the left eye, an optical system such as a prism can be used as necessary, or image coordinate conversion or the like can be performed.

  In the case of the color bit marker 30, since a plurality of markers can be recognized within the same visual field, a plurality of balloons can be simultaneously given to different objects. When viewing with binocular eyes, the speech balloon appears to overlap the real vision of the right eye, but if you move your consciousness to the left eye, you can read the information in the balloon in detail, and if you move your consciousness to the right eye, the real thing Can be seen in detail, it is practically convenient.

  FIG. 8 shows an embodiment of an augmented reality display system that presents home appliance information according to the present invention. A single board computer (for example, Raspberry Pi) 40 equipped with an ARM processor is used as a database storing information on home appliances (for example, an air conditioner) and can be accessed as a web server. The smartphone 20 attached to the one-eye AR scope 10 can access the database (40) via Wi-Fi and LAN via the wireless router.

  As illustrated in FIG. 9, when the color bit marker 30 enters the visual field of the camera of the smartphone 20 of the one-eye AR scope 10 of the wearer 8, the recognition program displays the index number and position of the color bit marker 30 in real time. To detect. Based on the index number, information on the object (air conditioner 50 in the figure) is extracted from the database of the single board computer 40, and the information is synthesized and displayed as a balloon near the position detected on the screen.

  When viewed with both eyes, the object and landscape are stereoscopically viewed, and the AR information appears to overlap therewith. If you move your consciousness to the left eye, you can read the information in the balloon in detail, and if you move your consciousness to the right eye, you can see the real thing clearly.

  In addition to displaying character information, for example, voice information can be provided via an earphone connected to the smartphone 20.

  In addition, although the structure was very simple in the said embodiment, in addition to the cushion etc. which disperse | distribute pressure to the position where the frame 12 contacts with the wearer's 8 head, the weight of the smart phone 20 is a contact etc. of a face etc. You can concentrate on one point and not feel pain.

  In the above-described embodiment, the right eye is open and the left eye is AR. Moreover, it is also possible to configure so that the left and right can be used interchangeably. Furthermore, the configuration can be devised so that it can be worn only on one eye.

  The mobile terminal is not limited to a smartphone, and a tablet, iPhone (registered trademark), iPad (registered trademark), or the like can also be used.

  The lens is not limited to a magnifying glass.

8 ... Wearer 9 ... Eye 10 ... One eye type AR scope 12 ... Frame body 13 ... Frame body extension 14 ... Smartphone holder 16 ... Belt 20 ... Smartphone 20A ... Screen 21 ... Virtual image 22 ... Magnifying glass 23 ... Lens 30 ... Color bit Marker 40 ... Single board computer (database)
42 ... Wireless router 50 ... Air conditioner

Claims (7)

A head-mounted augmented reality display device that displays an augmented reality only for the other eye, with an open state without a shield so that one eye can directly see the real space,
A head-mounted augmented reality display device characterized in that a mobile terminal in which a camera, an arithmetic processing unit, and a display screen are integrated is used to display augmented reality for the other eye.   An object on the line of sight of an eye looking directly at a real space is photographed with a moderate parallax by the camera of the portable terminal, and the raw video or an image modified to a raw video to realize augmented reality is the portable terminal. The head-mounted augmented reality display device according to claim 1, wherein the head-mounted augmented reality display device according to claim 1, wherein the head-mounted augmented reality display device is displayed in real time on the screen and stereoscopically viewed with both eyes.   A lens is arranged between the mobile terminal and the eye, and the distance between the lens and the mobile terminal is set so that the position of the virtual image on the display screen can be set to the same distance as the object of the eye looking directly into the real space. The head-mounted augmented reality display device according to claim 1, wherein the head-mounted augmented reality display device is adjustable.   The horizontally long portable terminal can be arranged to be tilted with respect to the front so that the change in the distance between the lens and the portable terminal screen does not increase in the front direction and the peripheral direction of the eye viewing the display screen. The head-mounted augmented reality display device according to claim 3.   The head-mounted augmented reality display device according to claim 1, wherein the position of the portable terminal relative to the wearer is adjustable.   When the position of the virtual image is adjusted to the same distance as the object of the eye looking directly in the real space, the image displayed on the screen of the mobile terminal is enlarged or reduced so that the direct view of one eye and the other eye 2. A function of adjusting the size of a virtual image of an object to be approximately the same as a size of direct viewing so that a virtual image of the object is integrated in the brain and stereoscopic viewing by binocular vision is possible. The head-mounted augmented reality display device according to any one of 1 to 5. A head-mounted augmented reality display device according to any one of claims 1 to 6;
A marker disposed on the object;
It automatically recognizes the marker on the camera of the mobile device, displays the information related to the marker as an augmented reality at the corresponding part of the image, and displays it for the wearer, overlapping the object in the real space, or any arbitrary Signal processing means for making information visible in the surrounding position;
An augmented reality display system using a head-mounted augmented reality display device characterized by comprising: