WO2016170717A1 - Wearable display, information processing system, and control method - Google Patents

Abstract

Provided is a wearable display, etc. allowing a user to know, through haptic feedback, an image to which an Augmented Reality (AR) technology is applied and information about the position of an object in a real space relating to the image. The wearable display according to an embodiment of the present technology comprises a display part, a wearable part, and a haptics presentation part. The display part presents an AR displayed object in the visual field of a user. The wearable part is configured to support the display part and to be wearable by the user. The haptics presentation part is provided on the wearable part. When a predetermined change occurs in an AR displayed object candidate which may be presented as the AR displayed object, the haptics presentation part presents, by a haptic feedback pattern and through haptics, information about the relative position of the AR displayed object candidate viewed from the user.

Description

Wearable display, information processing system, and control method

The present technology relates to a wearable display capable of tactilely displaying information related to an image that can be displayed in a display field, an information processing system including the same, and a control method.

A technology called augmented reality (AR) that adds a corresponding image to a real space or its image is known. For example, Patent Document 1 describes a see-through type head-mounted display (HMD: Head Mounted Display) capable of superimposing and displaying an image related to an object existing in a real space.

The AR technology can be applied to, for example, the see-through type head mounted display as described above, an image obtained by imaging a real space, or the like. In any case, the display visual field that the user can visually recognize is limited, and the area in the display visual field that the user can watch closely is also limited.
Therefore, in Patent Document 2, when an image or the like included in mail (AR mail) employing AR technology is displayed superimposed on a corresponding physical object in the real space captured by the camera function of the receiving device, An augmented reality system is described in which a user is guided toward a physical object by a vibrator, audio output, or the like.

International Publication No. 2014/128810 JP 2012-165276 A

However, Patent Document 2 does not disclose a specific method for guiding a user toward a physical object using a vibrator.

In view of the circumstances as described above, the purpose of the present technology is to allow the user to grasp information regarding the image to which the AR technology is applied and the position of the object in the real space related to the image by tactile feedback. An object is to provide a wearable display, an information processing system using the same, and a control method.

In order to achieve the above object, a wearable display according to an embodiment of the present technology includes a display unit, a mounting unit, and a tactile sense presentation unit.
The display unit presents an AR (Augmented Reality) display in the user's field of view.
The attachment unit is configured to be attachable to the user.
The tactile sense presentation unit is provided on the mounting unit.
In addition, the tactile sense providing unit may detect a relative of the AR display candidate viewed from the user by a tactile feedback pattern when there is a predetermined change in the AR display candidate that can be presented as the AR display. Present information about the position by touch.

According to the wearable display, information related to the relative position of the AR display candidate viewed from the user can be presented to the user by a tactile sense using the tactile feedback pattern.

Here, the “AR display object” includes at least an image (AR object) to which the AR technology is applied, and includes an object in the real space related to the image in addition to the AR object. Also good. The “AR display object candidate” is prepared so as to be displayed as an AR display object in the field of view.

Further, the haptic feedback pattern may be generated based on information related to a relative position of the AR display object candidate viewed from the user.

For example, the information regarding the relative position of the AR display object candidate viewed from the user includes the direction of the AR display object candidate with respect to the center direction of the user's visual field, and between the AR display object candidate and the user. Information related to at least one of the distances may be included.

Thereby, for example, the direction of the AR display object candidate based on the center direction of the user's visual field and the distance between the AR display object candidate and the user can be grasped by the user.

Further, the tactile sense presentation unit includes a plurality of vibrators arranged at different positions of the mounting unit,
Each of the plurality of vibrators may vibrate according to a vibration pattern defined based on the information on the relative position and the arrangement of the plurality of vibrators.

Thereby, the relative position of the AR display object candidate viewed from the user and the vibration pattern of the vibrator can be linked, and the user can intuitively grasp the relative position.

More specifically, the mounting portion is
A first mounting member mounted on the right side of the user;
A second mounting member mounted on the left side of the user,
The plurality of vibrators are
A first vibrator disposed on the first mounting member;
A second vibrator disposed on the second mounting member,
The first vibrator vibrates with a stronger intensity than the second vibrator when the direction of the AR display candidate from the user corresponds to the right side of the user, and the AR display candidate When the direction seen from the user corresponds to the left side of the user, the user may vibrate with a lower strength than the second vibrator.

Thereby, the direction of the AR display object candidate viewed from the user can be expressed by the vibration intensity of the vibrator, and the direction can be more intuitively understood by the user.

Alternatively, the plurality of vibrators have a vibrator group arranged in a predetermined direction,
The plurality of transducers included in the transducer group may vibrate in order along the predetermined direction when the direction of the AR display candidate viewed from the user corresponds to the predetermined direction.

Thereby, the direction of the AR display object candidate based on the user can be expressed in the vibration order of the vibrator, and the direction can be more intuitively understood by the user.

In addition, the mounting part is
A modern part worn on the user's ear,
A support part connected to the modern part and supporting the display part;
The tactile sense presentation unit may be disposed in the modern unit.

Since the tactile sense presenting unit is disposed in the modern part away from the display unit, the influence on the display unit due to the driving of the tactile presenting unit can be suppressed.

In this case, the modern part may have higher rigidity than the support part.

This makes it possible to suppress the deformation of the modern part associated with the driving of the tactile sense presenting part, and to more effectively suppress the influence on the display part associated with the driving of the tactile sense presenting part.

On the other hand, the tactile sensation presentation unit may present information regarding the relative position of the AR display object viewed from the user by tactile sensation for the newly acquired AR display object candidate.
Or the said tactile sense presentation part may present the information regarding the relative position of the said AR display thing seen from the user by a tactile sense about the said AR display thing candidate in which the display mode changed.

Thus, more information can be provided to the user via the AR display object candidate.

In addition, the tactile sensation presentation unit may present information regarding the relative position of the AR display object viewed from the user with a tactile sensation for the AR display object candidate in which a predetermined change has occurred in the distance to the user. .

This makes it possible for the user to grasp that a change in distance has occurred with the AR display object candidate.

Alternatively, the AR display object candidate may be associated with a position outside the field of view.

Thus, even when the user cannot visually recognize the AR display object candidate, the information on the relative position can be grasped by tactile feedback.

In addition, the display unit may present an auxiliary display that assists information on the relative position of the AR display object viewed from the user, which is presented by the information pattern on the relative position, in the visual field.

Thereby, information regarding the relative position of the AR display object candidate viewed from the user can be more accurately grasped.

An information processing system according to another embodiment of the present technology includes a wearable display and a control unit.
The wearable display includes a display unit, a mounting unit, and a tactile sense presentation unit.
The display unit presents an AR (Augmented Reality) display in the user's field of view.
The attachment unit is configured to be attachable to the user.
The tactile sense presentation unit is provided on the mounting unit.
In addition, the tactile sense presenting unit presents information related to the relative position of the AR display object candidate viewed from the user to the user by a tactile feedback pattern.
The control unit generates a haptic feedback pattern and outputs the haptic feedback pattern to the haptic presentation unit when there is a predetermined change in an AR display candidate that can be presented as the AR display. .

A control method according to still another embodiment of the present technology includes a display unit that presents an AR display object in a user's field of view, a mounting unit that can be mounted on the user, and a tactile sensation provided on the mounting unit. A wearable display control method including a presentation unit.
When there is a predetermined change in an AR display object candidate that can be presented as the AR display object, information related to the relative position of the AR display object candidate viewed from the user is presented by a tactile sense using a tactile feedback pattern. To do.

As described above, according to the present technology, it is possible to allow the user to grasp information regarding the image to which the AR technology is applied and the position of the object in the real space related to the image by tactile feedback.
Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

1 is a schematic diagram illustrating a configuration of an AR system (information processing system) according to a first embodiment of the present technology. It is a figure which shows the example of the visual field shown by HMD (wearable display) of the said AR system. It is a block diagram which shows the structure of the said AR system. It is a perspective view which shows the external appearance of the said HMD. It is a typical side view of the HMD. It is a block diagram which shows the functional structure of the control unit of the said AR system. It is the schematic which shows the cylindrical coordinate employ | adopted in the said AR system. It is an expanded view which shows the visual field presented in the said HMD. It is an expanded view which shows notionally the relationship between the visual field presented in the said HMD, and an object. It is a flowchart which shows operation | movement of the said AR system whole. It is a flowchart explaining the operation example of the said control unit. It is a schematic diagram which shows the example of a production | generation of the tactile feedback pattern by the said control unit. FIG. 10 is a schematic diagram illustrating a generation example of a haptic feedback pattern by the control unit according to Modification 1-1 of the embodiment. FIG. 10 is a schematic diagram illustrating an example of generation of a haptic feedback pattern by the control unit according to Modification 1-2 of the embodiment. It is a typical side view of HMD of modification 1-3 of the above-mentioned embodiment. It is a typical top view of HMD of modification 1-4 of the above-mentioned embodiment. It is a typical top view of HMD of modification 1-5 of the above-mentioned embodiment. It is a graph explaining the example of a drive of the tactile sense presentation part of HMD shown in FIG. It is a figure showing an example of auxiliary display shown by HMD of AR system concerning a 2nd embodiment of this art. It is a figure which shows the other example of the auxiliary display shown by the said HMD. It is a block diagram which shows the functional structure of the control unit of the said AR system. It is a figure which shows the other example of the auxiliary display shown by the said HMD. It is a figure which shows the other example of the auxiliary display shown by the said HMD.

Hereinafter, embodiments of the present technology will be described with reference to the drawings.

<First Embodiment>
[Schematic configuration of AR system]
FIG. 1 is a schematic diagram illustrating a configuration of an AR system (information processing system) according to the first embodiment of the present technology. As shown in the figure, the AR system 100 includes a wearable display 10, a control unit 20 that controls the wearable display 10, a portable information terminal 30, and an AR server 40. The wearable display 10 is a head mounted display (HMD) in the present embodiment, and is hereinafter referred to as an HMD 10. In addition, HMD10 of FIG. 1 has shown the shape typically.
The AR server 40 is a server device on the Internet 50. The AR server 40 stores information related to the AR object.
The portable information terminal 30 is typically a smartphone, but is configured by an information processing apparatus such as a mobile phone, a tablet terminal, a personal computer (PC), a tablet PC, or a PDA (Personal Digital Assistant).
The portable information terminal 30 can acquire the current position of the user by a GPS (Global Positioning System) function. The portable information terminal 30 is connected to the AR server 40 via the Internet 50, and can acquire information related to the AR display processing and the like from the AR server 40. Further, the portable information terminal 30 is connected to the control unit 20 by a short-range wireless communication system such as Bluetooth (registered trademark), and can transmit information relating to the AR display processing and the like and information relating to the current position of the user to the control unit 20. .
The HMD 10 is configured as a glasses-shaped see-through display.
The control unit 20 is a device for controlling the HMD 10 and controls the operation of the HMD 10 based on an input operation by the user. The control unit 20 is connected to the HMD 10 via a cable corresponding to a predetermined standard, executes processing based on information acquired from the portable information terminal 30, and outputs a processing result to the HMD 10.
Thereby, the HMD 10 can provide an AR display object in which an image associated therewith is superimposed on an object existing in the real space via a see-through display to a user wearing the HMD 10. Hereinafter, an image related to an object is referred to as an AR object or simply an object.

FIG. 2 is a diagram illustrating an example of a visual field presented by the HMD 10.
The visual field V shown in FIG. The AR display object S includes an object A existing in the real space and an object B related thereto. The object A is determined according to the type of application program that executes processing in the AR system 100 (control unit 20). For example, the object A is a building, a person, a tourist spot, a dangerous spot, or the like that the user wants to watch. Can do. The object B is an image that displays information related to the object A, and may be an image including characters, patterns, or an animation image. The object may be a two-dimensional image or a three-dimensional image. Furthermore, the shape of the object may be a rectangle, a circle, or other geometric shape, and can be set as appropriate depending on the type of the object.
A user wearing the HMD 10 can acquire information related to the object A using the object B.
On the other hand, in this embodiment, AR display object candidates are prepared at predetermined positions in a region other than the visual field V. These AR display object candidates can be displayed by changing the field of view due to the movement of the user or the change in the posture of the display unit 11.

FIG. 3 is a block diagram showing the configuration of the AR system 100.
Hereinafter, each element of the AR system 100 will be described with reference to FIG.

[AR server configuration]
As illustrated in FIG. 3, the AR server 40 includes a CPU 401, a memory 402, and a network communication unit 403. Although not shown, the AR server 40 may have a configuration such as an input device, a display device, and a speaker as necessary.
The CPU 401 controls the overall operation of the AR server 40.
The memory 402 includes nonvolatile memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory), an HDD (Hard Disk Drive), and a flash memory (SSD; Solid State Drive). Stores programs for executing control, various parameters, and other necessary data.
The network communication unit 403 communicates with the portable information terminal 30 via the Internet 50. The communication method is not particularly limited, and may be wired communication using a NIC (Network Interface Card) for Ethernet (registered trademark), wireless LAN (IEEE802.11, etc.) such as WiFi (Wireless Fidelity), or mobile Wireless communication using a 3G or 4G network for communication may be used.

The memory 402 also holds an AR database 404.
The AR database 404 stores information such as object image information, target object attributes, target object position, and object position for each AR display object candidate. The position information of the object is typically information on the absolute position (latitude, longitude, etc.) acquired from the portable information terminal 30. For example, a cylindrical coordinate system around the user centered on the user wearing the HMD 10 It may be expressed as the upper coordinates.
In the AR database 404, information on new AR display objects is additionally registered as appropriate by the portable information terminal 30, other portable information terminals, information processing apparatuses, and the like connected to the AR server 40 via the Internet 50.

[Configuration of portable information terminal]
As shown in FIG. 3, the portable information terminal 30 includes a CPU 301, a memory 302, a network communication unit 303, a short-range communication unit 304, a GPS communication unit 305, a display unit 306 on which a touch panel is mounted, And a power source 307.
The CPU 301 controls the operation of the mobile information terminal 30 as a whole.
The memory 302 includes a ROM, a RAM, a non-volatile memory, and the like. An application program and various parameters for executing control of the portable information terminal 30 by the CPU 301, information on an AR display object transmitted to the control unit 20, and other necessary Memorize data.
The network communication unit 303 communicates with the AR server 40 or the like using a wireless LAN (IEEE802.11 or the like) such as WiFi (Wireless Fidelity) or a 3G or 4G network for mobile communication. The portable information terminal 30 downloads information on the AR display object to be transmitted to the control unit 20 from the AR server 40 via the network communication unit 303 and stores it in the memory 302.
The short-range communication unit 304 communicates with the control unit 20 and other portable information terminals using a short-range communication system such as Bluetooth (registered trademark) or infrared communication.
The GPS communication unit 305 acquires the current position of the user carrying the portable information terminal 30 by receiving a signal from a GPS satellite.
The display unit 306 includes, for example, an LCD (Liquid Crystal Display) or an OELD (Organic ElectroLuminescence Display), and displays various menus, GUIs of applications, and the like. Typically, the display unit 306 is equipped with a touch panel and can accept a user's touch operation.
The internal power supply 307 supplies power necessary for driving the portable information terminal 30.

[Control unit configuration]
As shown in FIG. 3, the control unit 20 includes a CPU 201, a memory 202, a communication unit 203, an input operation unit 204, and an internal power source 205.
The CPU 201 controls the operation of the entire control unit 20. The memory 202 includes a ROM, a RAM, and the like, and stores a program for executing control of the control unit 20 by the CPU 201, various parameters, information on an AR display object, and other necessary data. The communication unit 203 constitutes an interface for short-range communication with the portable information terminal 30.
The input operation unit 204 is for controlling an image displayed on the HMD 10 by a user operation. The input operation unit 204 may be configured with a mechanical switch or a touch sensor.
The internal power supply 205 supplies power necessary for driving the HMD 100.

[Configuration of HMD]
FIG. 4 is a perspective view showing the appearance of the HMD 10, and FIG. 5 is a schematic side view of the HMD 10.
As shown in FIGS. 3 and 4, the HMD 10 includes a display unit 11, a mounting unit 12, a tactile sense presentation unit 13, and a detection unit 14.

(Display section)
The display unit 11 presents an AR display object in the user's field of view. In this embodiment, the AR display object includes an AR object and an object in the real space related to the AR object. The display unit 11 includes first and second display surfaces (display surfaces) 111R and 111L, and first and second image generation units (image generation units) 112R and 112L.
The 1st and 2nd display surfaces 111R and 111L are comprised with the optical element which has transparency which can provide real space (external field visual field) to the right eye and left eye of the user U, respectively. The first and second image generation units 112R and 112L are configured to be able to generate images to be presented to the user U via the first and second display surfaces 111R and 111L, respectively.
The display unit 11 configured as described above is configured to be able to provide a user with a visual field in which a predetermined image (or virtual image) is superimposed on the real space via the display surfaces 111R and 111L. .
The display unit 11 may further include a frame 113 that supports the first and second display surfaces 111R and 111L and the first and second image generation units 112R and 112L. The frame 113 is made of a lightweight and relatively high rigidity material, and is made of a metal material such as Mg (magnesium) or Al (aluminum). With the frame 113, the relative arrangement of the first and second display surfaces 111R and 111L with respect to the first and second image generation units 112R and 112L can be fixed, and a clear image can be provided.
In the present embodiment, the display unit 11 is supported by the mounting unit 12.

(Mounting part)
The mounting unit 12 is configured to be mountable by a user.
The mounting unit 12 supports the display surfaces 111R and 111L and the image generation units 112R and 112L, and is mounted on the user's head so that the display surfaces 111L and 111R face the right and left eyes of the user, respectively.
For example, the mounting unit 12 includes a first mounting member 121R mounted on the right side of the user, a second mounting member 121L mounted on the left side of the user, and a support unit 122.
In the present embodiment, the first and second mounting members (mounting members) 121R and 121L are mounted on the user's temporal region and are configured in the shape of temples of glasses.
The support portion 122 is configured along the upper portions of the display surfaces 111R and 111L, and connects the first and second mounting members 121R and 121L. The support part 122 supports the display part 11 by being connected to the center part of the frame 113.
The first and second mounting members 121R and 121L include first and second modern portions (modern portions) 123R and 123L formed in a region extending from the tip portion to the central portion, and the first and second modern portions 123R. , 123L and the support part 122, the first and second connection parts 124R, 124L are included.
The first and second modern parts (modern parts) 123R and 123L have a shape that can be locked to the upper part of the user's auricle, for example, and are attached to the user's ears. Further, the first and second modern portions 123R and 123L may have covers 123Ra and 123La formed of a highly elastic material such as silicone resin or rubber resin as shown in FIGS. Good. Thereby, a user's burden can be reduced.

The support portion 122 and the first and second connection portions 124R and 124L are formed of a material such as a relatively elastic resin or metal as a whole. On the other hand, the first and second modern parts 123 </ b> R and 123 </ b> L may have higher rigidity than the support part 122. The rigidity may be adjusted depending on the material or may be adjusted depending on the shape. Thereby, it can be set as the structure which the drive of the tactile sense presentation part 13 mentioned later cannot transmit easily to the display part 11. FIG.
The connecting portions 124R and 124 may have the same rigidity as the modern portions 123R and 123L, or may have the same rigidity as the support portion 122.

(Tactile presentation part)
The tactile sense providing unit 13 is provided in the mounting unit 12 and presents the user with information related to the relative position of the AR display object candidate viewed from the user by a tactile feedback pattern.
The tactile sense presentation unit 13 includes a plurality of vibrators 131 arranged at different positions on the mounting unit 12. In the present embodiment, the plurality of vibrators 131 include a first vibrator 131R disposed on the first mounting member 121R and a second vibrator 131L disposed on the second mounting member 121L. The first and second vibrators (vibrators) 131R and 131L are disposed in the first and second modern portions 123R and 123L, respectively.
Each vibrator 131R, 131L is constituted by a vibration motor in this embodiment. The vibration motor generates vibration when an unbalanced weight attached to the rotation shaft of the motor rotates around the rotation shaft. For example, in each of the vibrators 131R and 131L, the extending direction of the rotating shaft matches the extending direction of the mounting members 121R and 121R, and the rotating directions of the weights around the rotating shaft are different from each other. Be placed. Thereby, even when the vibrators 131R and 131L vibrate, the balance between the first and second mounting members 121R and 121L can be easily maintained, and the influence of vibration on the display unit 11 can be suppressed.
When there is a predetermined change in the AR display object candidate that can be presented as an AR display object, the tactile sense providing unit 13 obtains information on the relative position of the AR display object candidate viewed from the user by a tactile feedback pattern. Present by touch. Details will be described later.

(Detection unit)
The detection unit 14 can detect a change in posture of the display unit 11. In the present embodiment, the detection unit 14 is configured to detect posture changes around the X, Y, and Z axes, respectively.
The detection unit 14 can be configured by a motion sensor such as an angular velocity sensor or an acceleration sensor, or a combination thereof. In this case, the detection unit 14 may be configured by a sensor unit in which each of the angular velocity sensor and the acceleration sensor is arranged in the three-axis directions, or the sensor to be used may be different depending on each axis. For example, an integrated value of the output of the angular velocity sensor can be used as the posture change of the display unit 11, the direction of the change, the amount of the change, and the like.
Further, a geomagnetic sensor may be employed for detecting the orientation of the display unit 11 around the vertical axis (Z axis). Alternatively, the geomagnetic sensor and the motion sensor may be combined. Thereby, it is possible to detect a change in orientation or posture with high accuracy.
In order to obtain the direction of the AR display object based on the user, it is necessary to know the direction (azimuth) of the user's visual field, and therefore the accuracy can be improved by using a geomagnetic sensor.
The detection unit 14 is disposed at an appropriate position on the display unit 11. The position of the detection unit 14 is not particularly limited. For example, the detection unit 14 may be arranged in one of the image generation units 112R and 112L, or may be arranged in a part of the mounting unit 12.

The operation of the HMD 10 configured as described above is controlled by executing predetermined processing by the control unit 20. Hereinafter, the functional configuration of the control unit 20 will be described.

[Functional configuration of control unit]
FIG. 6 is a block diagram showing a functional configuration of the control unit 20.
The control unit 20 includes an AR display processing unit 21 that executes an AR display process and a haptic feedback processing unit 22 that executes a haptic feedback process.

(AR display processing unit)
The AR display processing unit 21 includes a visual field setting unit 211, an AR information management unit 212, and a display control unit 213.

The visual field setting unit 211 sets a visual field range based on the attitude of the display unit 11 calculated from the detection result of the detection unit 14. The visual field setting unit 211 is executed by the CPU 201 of the control unit 20. In the present embodiment, for setting the visual field range by the visual field setting unit 211, virtual cylindrical coordinates C0 surrounding the user U with the vertical axis Az as the center are used.

FIG. 7 is a schematic diagram illustrating the cylindrical coordinates C0 and the field of view V.
As shown in the figure, the cylindrical coordinate C0 is a coordinate system that defines a position on a virtual circumferential surface arranged at a distance (radius) R from the vertical axis Az. The user U (display unit 11) is arranged on the vertical axis Az. The cylindrical coordinate C0 is a height in the height direction that represents a vertical coordinate axis (θ) that represents an angle around the vertical axis with the north direction being 0 °, and a horizontal line of sight Lh of the user U. Coordinate axis (h). The coordinate axis (θ) has a positive direction around the east, and the coordinate axis (h) has a depression angle as a positive direction and an elevation angle as a negative direction. The radius R and height H of the cylindrical coordinates C0 can be arbitrarily set. Note that the position of the user U that defines the vertical axis Az is defined by the position of the user U acquired by the portable information terminal 30.
As shown in the figure, the visual field V is arranged on a virtual peripheral surface in which the cylindrical coordinates C0 are set.

The visual field setting unit 211 calculates the posture change of the display unit 11 based on the output of the detection unit 14, and determines which region on the cylindrical coordinate C0 the user's U visual field V belongs to. The region to which the visual field V belongs in the cylindrical coordinate C0 is defined by the ranges of θ and h. The visual field V moves on the cylindrical coordinates C0 due to the change in the posture of the user U, and the moving direction and moving amount are calculated based on the output of the detection unit 14.

FIG. 8 is a development view of the cylindrical coordinates C0 indicating the visual field V on the cylindrical coordinates C0. The symbol Oc in the figure indicates the origin of the cylindrical coordinates C0.
The visual field V has a substantially rectangular shape. For example, the circumferential range of the visual field V is expressed by the following formula (3).
θv1 ≦ θv ≦ θv2 (3)
On the other hand, the range of the visual field V in the height direction is represented by the following formula (4).
hv1 ≦ hv ≦ hv2 (4)

The AR information management unit 212 acquires information about a predetermined AR display object candidate from the portable information terminal 30 and manages the acquired information about the AR display object candidate. The AR information management unit 212 is executed by the CPU 201 and the memory 202 of the control unit 20.
The AR display object candidate information includes, for example, object position information, object image information, and object position information.

With reference to FIG. 7 again, position information of the AR display object candidate will be described.
In this embodiment, the AR display object candidate refers to a set of an object that can be arranged on the cylindrical coordinate C0 and an object in the real space related to the object. In FIG. 7, object A1 and object B1 are AR display object candidates S1, object A2 and object B2 are AR display object candidates S2, object A3 and object B3 are AR display object candidates S3, object A4 and object B4 are This is defined as an AR display object candidate S4. Among these AR display object candidates S1 to S4, those displayed in the visual field V are referred to as “AR display objects”.
The objects A1 to A4 exist in the real space, and typically exist at positions farther from the user U than R. Thus, the position information of the objects A1 to A4 may include information on the distance from the user U to the objects A1 to A4, or the positions (latitude, longitude, etc.) of the objects A1 to A4 in the real space. Information may be included.
On the other hand, the positions (coordinates) of the objects B1 to B4 are respectively associated with the intersection positions of the user's eye line L watching the objects A1 to A4 and the cylindrical coordinates C0. That is, the position information of the objects B1 to B4 includes, for example, coordinate information on the cylindrical coordinates C0.
In the illustrated example, the center position of each of the objects B1 to B4 is made coincident with the intersection position. However, the present invention is not limited to this, and a part of the periphery of the object (for example, part of the four corners) may coincide with the intersection position. Good. Alternatively, the coordinate positions of the objects B1 to B4 may be associated with any position away from the intersection position.

The AR information management unit 212 acquires AR display object candidate information corresponding to the current position of the user acquired in advance from the portable information terminal 30 as information about the AR display object candidate. The information acquisition timing by the AR information management unit 212 is determined to be, for example, a timing at which it is determined that the current position of the user has changed by a predetermined distance or more, or a notification from the AR server 40 regarding the state change of the AR display object candidate. Timing can be used.
For example, the AR information management unit 212 can acquire AR display object information related to an object existing at a predetermined distance from the user's current position as AR display object information corresponding to the current position. The AR information management unit 212 may also delete unnecessary AR display object candidate information based on the current position of the user.
The AR information management unit 212 supplies the acquired AR display object candidate information to the display control unit 213.

The display control unit 213 executes a process of displaying (drawing) an object on the cylindrical coordinate C0 corresponding to the orientation of the display unit 11 in the visual field V based on the output of the detection unit 14 (that is, the processing result of the visual field setting unit 211). Configured to do. The display control unit 213 is executed by the CPU 201 of the control unit 20.
FIG. 9 is a developed view of the cylindrical coordinates C0 conceptually showing the relationship between the visual field V on the cylindrical coordinates C0 and the objects B1 to B4.
For example, as shown in the figure, when the current orientation of the visual field V overlaps the display areas of the objects B1 and B2 on the cylindrical coordinates C0, images corresponding to the overlapping areas B10 and B20 are displayed in the visual field V. . The display control unit 213 typically changes the display position of the objects B <b> 1 and B <b> 2 within the field of view V following the change in the orientation or orientation of the display unit 11. This control is continued as long as at least a part of each object B1, B2 exists in the visual field V.
For example, the display control unit 213 may set an xy coordinate (local coordinate) having one point belonging to the field of view V as an origin, and execute a calculation for converting the cylindrical coordinate C0 to the local coordinate.

Since the display area (field of view V) of the HMD 10 is narrower than the visual field of human eyes, the objects displayed in the field of view V are limited. For example, since the objects B3 and B4 shown in FIG. 9 are out of the field of view, the user cannot see these objects B3 and B4 at all. Thus, there is a possibility that useful information cannot be sufficiently provided only by the presentation of the visual field V.
Therefore, according to the present embodiment, the position information of the AR display object candidate outside the field of view can be grasped by the user by tactile feedback.

(Tactile feedback processing unit)
The tactile feedback processing unit 22 includes an AR information management unit 221, a tactile processing determination unit 222, and a feedback generation unit 223. In the present embodiment, the tactile feedback pattern is generated based on information on the relative position of the AR display object candidate viewed from the user.

The AR information management unit 221 acquires information on the AR display object candidate corresponding to the current position of the user from the portable information terminal 30, and manages the acquired information on the AR display object candidate. The AR information management unit 212 is executed by the CPU 201 and the memory 202 of the control unit 20. The AR information management unit 221 may be executed by the AR information management unit 212 of the AR display processing unit 21.
The AR display object candidate information includes, for example, object position information, object image information, and object position information. The position information of the objects A1 to A4 may include information about the distance from the user U to the objects A1 to A4, or may include information about the positions of the objects A1 to A4 in the real space. The position information of the objects B1 to B4 may include coordinate information on the cylindrical coordinates C0.
The AR information management unit 221 acquires information on an AR display object candidate corresponding to the current position of the user acquired in advance from the portable information terminal 30 as information on the AR display object candidate. The information acquisition timing by the AR information management unit 221 can be, for example, a timing when it is determined that the current position of the user has changed by a predetermined distance or more.
The AR information management unit 212 may also delete unnecessary AR display object candidate information based on the current position of the user.
The AR information management unit 221 supplies the acquired AR display object candidate information to the tactile sense processing determination unit 222.

The haptic process determination unit 222 determines whether there is an AR display object candidate to be subjected to the haptic feedback process among the AR display object candidates acquired by the AR information management unit 221. The tactile processing determination unit 222 is executed by the CPU 201 of the control unit 20.
The criteria for tactile feedback processing targets are, for example, whether or not there is a newly acquired AR display object candidate or whether or not there is an AR display object candidate whose display mode has changed. can do. Examples of “newly acquired” AR display object candidates include, for example, a newly acquired AR display object candidate that has been newly registered in the AR database 404, and a user's movement between the user and the target object. The AR display object candidate newly acquired when the distance is equal to or less than the predetermined distance may be used. Examples of the AR display object candidate “changed in the display mode” include an AR display object candidate in which the display mode of the image of the object registered in the database 404 of the AR server 40 has changed.
The tactile processing determination unit 222 supplies the determination result to the feedback generation unit 223.

The feedback generation unit 223 generates a haptic feedback pattern for tactilely presenting information regarding the relative position of the AR display object candidate viewed from the user of the AR display object candidate determined as the target of the tactile feedback process. The feedback generation unit 223 is executed by the CPU 201 of the control unit 20.
The information regarding the relative position of the AR display object candidate viewed from the user is, for example, at least one of the direction of the AR display object candidate based on the center direction of the user's visual field and the distance between the AR display object candidate and the user. Contains information related to one. The information related to the direction of the AR display candidate based on the center direction of the user's visual field is, for example, information on the rotation angle of the object based on the central direction of the visual field (the front direction of the user), Information such as the height of an object based on the height may be included.
In the present embodiment, the tactile feedback pattern is a vibration pattern of the first vibrator 131R and the second vibrator 131L, and the vibration intensity and vibration timing of each of the first vibrator 131R and the second vibrator 131L. Including patterns. As the vibration pattern, for example, when the direction of the AR display object candidate based on the center direction of the user's visual field corresponds to the right side of the user, the first vibrator 131R is stronger than the second vibrator 131L. If the direction of the AR display object candidate with respect to the center direction of the user's visual field corresponds to the left side of the user, the first vibrator 131R is vibrated with a weaker intensity than the second vibrator 131L. A pattern is applied.
Moreover, the AR display object candidate that is the target of the tactile feedback process may be associated with a position outside the field of view.

[Operation example of AR system]
(AR display object candidate information acquisition process)
FIG. 10A is a flowchart showing a flow of processing for acquiring information on AR display object candidates in the control unit 20. In this operation example, the control unit 20 executes an AR display candidate information acquisition process when the current position of the user has changed by, for example, 50 m or more.
In the figure, the processing from ST101 to ST104 is executed by the portable information terminal 30. Further, the processes of ST201 to ST203 are executed by the AR server 40, and the process of ST301 is executed by the control unit 20.

First, the CPU 301 of the portable information terminal 30 determines whether or not an acquisition condition for information of an AR display object candidate is satisfied (ST101). Specifically, the CPU 301 may determine whether or not the user's current position has changed by 50 m or more based on the current position information acquired from the GPS communication unit 305. In addition to the above, or in place of the above, the CPU 301 may determine whether or not a notification regarding the state change of the AR display object candidate has been received from the AR server 40.
When it is determined that the information acquisition condition for the AR display object candidate is satisfied (Yes in ST101), the network communication unit 303 determines the current position acquired from the GPS communication unit 305 based on the control of the CPU 301 as the AR server 40. (ST102).

The network communication unit 403 of the AR server 40 receives the current position transmitted from the portable information terminal 30 based on the control of the CPU 401 (ST201). Subsequently, the CPU 401 acquires information on AR display object candidates corresponding to the acquired current position from the object database 404 (ST202). Then, based on the control of CPU 401, network communication unit 403 transmits information about the AR display object candidate acquired to portable information terminal 30 to portable information terminal 30 (ST203).

The network communication unit 303 of the portable information terminal 30 acquires information on the AR display object candidate transmitted from the AR server 40 based on the control of the CPU 301 (ST103). Subsequently, the short-range communication unit 304 transmits information on the acquired AR display object candidate to the control unit 20 based on the control of the CPU 301 (ST104).

Finally, the communication unit 203 of the control unit 20 receives the information of the AR display object candidate transmitted from the portable information terminal 30 based on the control of the CPU 201, and the CPU 201 stores the information in the memory 202 (ST301).
Thereby, the AR information management units 212 and 221 of the control unit 20 can acquire information on AR display object candidates.
On the other hand, the control unit 20 cooperates with the HMD 10 to execute a drawing process at a predetermined drawing timing. The outline of the drawing process will be described below.

(Drawing process)
FIG. 10B is a flowchart showing the flow of the drawing process.
In the figure, the processing of ST401 to ST404 is executed by the HMD 10. Further, the processing of ST501 to ST504 is executed by the control unit 20.

First, the HMD 10 determines whether or not the drawing timing of 1/30 sec has been reached (ST401).
When it is determined that the drawing timing has come (Yes in ST401), the HMD 10 outputs the detection result of the detection unit 14 to the control unit 20 (ST402).
Moreover, the visual field setting unit 211 of the control unit 20 acquires the output detection result of the detection unit 14 (ST501).

The AR information management units 212 and 221 of the control unit 20 that acquired the detection result acquires information on AR display object candidates from the memory 202 (ST502).
Then, the control unit 20 executes an AR display process (ST503) and a tactile feedback process (ST504) in cooperation with the CPU 201 and the memory 202. The processing results of the AR display process (ST503) and the tactile feedback process (ST504) are transmitted to the HMD 10, respectively.
The display unit 11 of the HMD 10 presents a field of view including a predetermined AR display object based on the processing result of the AR display process (ST403), and the tactile sense presentation unit 13 compares the object relative to the object based on the processing result of the tactile feedback process. The position information is presented by touch (ST404).

[Operation example of control unit]
FIG. 11 is a flowchart for explaining an operation example of the control unit. FIG. 11A is a flowchart of the AR display process (ST203), and FIG. 11B is a flowchart of the tactile feedback process (ST204).

(AR display processing)
Referring to FIG. 11A, first, the visual field setting unit 211 determines the range of the visual field V represented by Expression (3) and Expression (4) based on the attitude of the display unit 11 calculated from the detection result of the detection unit 14. Is set (see FIGS. 7 and 8) (ST601).
Subsequently, the display control unit 213 determines whether there is an object to be displayed in the field of view V based on the information on the AR display object candidate managed by the AR information management unit 212 (see FIG. 7) ( ST602). If determined to be present (Yes in ST602), the display control unit 213 executes a process of displaying an object overlapping the visual field V (ST603).
As described above, the control unit 20 ends the AR display process.

(Tactile feedback processing)
Subsequently, with reference to FIG. 11B, the haptic process determination unit 222 determines whether there is an AR display object candidate to be subjected to the haptic feedback process among the AR display object candidates acquired by the AR information management unit 221. (ST701).
Specifically, the tactile sensation processing determination unit 222 determines whether or not there is a newly acquired AR display object candidate among the AR display object candidates acquired by the AR information management unit 221. The newly acquired AR display object candidate is, for example, an AR display object candidate acquired by the AR information management unit 221 in the current process, which was not acquired by the AR information management unit 221 in the previous process.
When it is determined that there is a newly acquired AR display object candidate, the feedback generation unit 223 generates a tactile feedback pattern for each AR display object candidate determined to be a target as follows (ST702).

(Tactile feedback pattern generation example)
FIG. 12 is a schematic diagram illustrating a generation example of a haptic feedback pattern.
First, the feedback generation unit 223 calculates the rotation angle (coordinate in the circumferential direction) θ′b [°] of the object B included in the AR display object candidate determined as the target with reference to the front direction of the user U. (ST702-1).
The coordinate θv0 in the front direction of the user U is represented by the following formula (5) in view of the range in the circumferential direction of the visual field V shown by the formula (3).
θv0 [°] = (θv1 + θv2) / 2 (5)
In the cylindrical coordinates C0, the area on the right side of the user U is represented by coordinates not less than θv0 [°] and not more than (θv0 + 180) [°], and the area on the left side of the user U is not less than (θv0 + 180) [°]. (Θv0 + 360) It is expressed by coordinates below [°].
The relative rotation angle θ′b of the object B is expressed by the following equation (6) using the object coordinate θb in the cylindrical coordinate C0 and the coordinate θv0 in the front direction of the user U.
θ′b = θb−θv0 (6)

Subsequently, the feedback generation unit 223 refers to, for example, a lookup table stored in advance in the memory 202, and determines a region to which θ′b belongs (ST703-2).
As shown in FIG. 12, in this operation example, first to fifth regions R1 to R5 are defined so as to surround the user U. FIG. 12 shows an example in which θ′b belongs to the first region R1. The range of θ′b belonging to the first region R1 is expressed by the following formula (10).
θ′v2 <θ′b ≦ 90 ° (10)
In the equation, θ′v2 is a relative coordinate of the right end portion of the visual field V with the front direction of the user U as a reference (0 °), and satisfies θ′v2 = θv2−θv0.
Similarly, the range of θ′b belonging to the visual field V and the second to fifth regions R2 to R5 is shown in Table 1 below.
In the table, θ′v1 is a relative coordinate of the left end portion of the field of view V with the front direction of the user U as a reference (0 °), and satisfies θ′v1 = θv1−θv0 similarly to θ′v2.

Subsequently, feedback generation section 223 refers to a lookup table stored in advance in memory 202, and determines the vibration pattern of each transducer 131R, 131L according to the region to which θ′b belongs (ST702-3).
FIG. 12 and Table 1 show the relationship between each of the first to fifth regions R1 to R5 stored as a lookup table and the strength of each transducer 131R, 131L.
In FIG. 12, “R” indicates the first vibrator 131R disposed on the right side, and “L” indicates the second vibrator 131L disposed on the left side. Also, the numerical values corresponding to “R” and “L” in FIG. 12 indicate the vibration strengths of the vibrators 131R and 131L. Regarding the numerical values of the vibration intensity shown in the chart, 0 indicates no vibration, the vibration intensity increases as the numerical value increases, and 10 indicates the maximum intensity.
As described above, the control unit 20 ends the tactile feedback generation process.

(Operation of this operation example)
Referring to FIG. 12 and Table 1, for example, the first region R1 corresponds to the right front of the user U and substantially coincides with the movable range of the right arm of the user U that can be recognized as the right side of the user U. In this case, the first vibrator 131R arranged on the right side vibrates at the maximum intensity, and the second vibrator 131L arranged on the left side does not vibrate. Thereby, the user can perceive a large vibration in the right side of the head and intuitively recognize that a new object exists on the right side of the user.
Similarly, the second region R2 corresponds to the right rear of the user U. In this case, the first vibrator 131R vibrates with a strength of about 80% of the maximum strength, and the second vibrator 131L vibrates with a strength of about 20% of the maximum strength. As a result, the user can perceive a large vibration in the right side of the head, and can also perceive a slight vibration in the left side of the head, and slightly toward the left rather than the right side of the user (the first region R1). It can be recognized that there is a new object in the rotated direction, that is, right rear.
Similarly, the third region R3 corresponds to the back of the user U, that is, the back surface. In this case, both vibrators 131R and 131L vibrate with substantially the same intensity. As a result, the user U can recognize that there is a new object in an area that is not biased to the right or left of the user U, that is, behind.
Similarly, when a new object exists in the fourth region R4 and the fifth region R5 corresponding to the left side of the user U, the user can recognize the presence and direction of the new object by vibration.
Further, as shown in FIG. 12 and Table 1, none of the vibrators 131R and 131L vibrates for the AR display object candidate included in the visual field V. That is, the AR display object candidate to be subjected to the tactile feedback process is associated with a position outside the visual field V. Thereby, according to this operation example, the direction of the AR display object candidate not included in the visual field V can be recognized by the user by tactile feedback.

[Operational effects of this embodiment]
As described above, according to the present embodiment, the control unit 20 can generate a haptic feedback pattern capable of presenting the direction of the AR display object candidate based on the center direction of the user's visual field. Thereby, even when the user cannot visually recognize the object, the user can accurately grasp the relative position of the AR display object candidate based on the tactile feedback pattern.
In particular, according to the present embodiment, the tactile sense providing unit 13 can present the information about the newly acquired AR display object candidate by tactile sense. Thereby, new information can be presented to the user together with the position information, and the object can be used effectively.

Furthermore, according to this embodiment, the direction of the AR display object candidate is adjusted by adjusting the vibration intensity by linking the arrangement of the plurality of vibrators 131R and 131L provided on the left and right and the relative position of the AR display object candidate. It is possible to grasp intuitively.

Furthermore, according to the present embodiment, the tactile sense presenting unit 13 is provided not on the display unit 11 but on the mounting unit 12, and in particular, by being arranged in the modern units 123 </ b> R and 123 </ b> L spaced apart from the display unit 11, vibration is transmitted to the display unit 11. It can be configured to be difficult to transmit. Thereby, the displacement of each of the display surfaces 111R and 111L due to the tactile feedback can be suppressed, and an image that is clear and easily visible even during vibration can be presented.
Moreover, the said effect can be further heightened by raising the rigidity of modern part 123R, 123L.

[AR system application example]
An application example of the AR system 100 (control unit 20) of the present embodiment will be described.

(Application example 1)
The AR system 100 can be applied to an application program (hereinafter also referred to as an application) that presents a position of a predetermined person in real time. In this case, the object is a person such as a family or a friend registered in advance by the user, and the object can be information (name or the like) specifying the person.
For example, when the AR server 40 determines that a person (object) exists within a predetermined distance (for example, 1 km) from the user, the AR server 40 acquires information on the person and the object as information on the AR display object candidate. Then, the tactile sense determination unit 222 of the control unit 20 determines whether or not the AR display object candidate acquired via the portable information terminal 30 is newly acquired. When it is determined that the information is newly acquired, the feedback generation unit 223 generates a tactile feedback pattern based on the position information of the person or the object.
Thereby, even when the object is not displayed, the user can grasp the approach of the person and the position information of the object, and can move the field of view toward the object.
Further, when the distance between the user and the person changes and further approaches, and the AR server 40 determines that the person exists within another predetermined distance (for example, 200 m) from the user, the information on the person and the object is stored in the AR. It is newly acquired again as information on the display object candidate. Then, the tactile sense determination unit 222 of the control unit 20 determines whether or not the AR display object candidate acquired via the portable information terminal 30 is newly acquired. When it is determined that the information is newly acquired, the feedback generation unit 223 generates a tactile feedback pattern based on the position information of the person or the object.
Thereby, the tactile sense providing unit 13 can present information related to the relative position of the AR display object candidate viewed from the user with a tactile sensation for the AR display object candidate in which a predetermined change has occurred in the distance from the user. Therefore, even when the object is not displayed, the user can grasp the further approach of the person and the position information of the object, and can also meet the person by moving the field of view toward the object.

(Application example 2)
The AR system 100 can be applied to an application that notifies a dangerous location such as a fire location, an accident location, or an accident-prone spot. In this case, the object is a dangerous place, and the object can be information (position or information for calling attention) associated with the dangerous place.
For example, when the AR server 40 determines that the dangerous part (target object) exists within a predetermined distance (for example, 10 km) from the user, the AR server 40 acquires the information on the dangerous part and the object as information on the AR display object candidate. Then, the tactile sense determination unit 222 of the control unit 20 determines whether or not the AR display object candidate acquired via the portable information terminal 30 is newly acquired. When it is determined that the information is newly acquired, the feedback generation unit 223 generates a tactile feedback pattern based on the dangerous location and the position information of the object.
Thereby, even when the object is not displayed, the user can grasp the approach of the dangerous place and the position information of the object, can move the field of view toward the object, escape in the direction opposite to the dangerous place, etc. You can take avoidance actions.
The distance informing the dangerous location may be changed depending on the type of danger and the user's action (whether walking or riding a car). Further, the vibration intensity of the tactile feedback pattern may be changed according to the degree of risk. Further, even when an object is displayed, a haptic feedback pattern may be generated. This makes it possible to avoid danger without fail.

(Application example 3)
The AR system 100 can be applied to a navigation application that provides route guidance and a town guide application that provides local information. In this case, the object is a tourist spot, a station, a bus stop, a store, a gas station, a parking lot, an evacuation site, a traffic jam site, a construction site, etc., and the object is information attached to the corresponding site (spot name, location information, etc.) ).
For example, when the AR server 40 determines that the corresponding part (target object) exists within a predetermined distance (for example, 10 km) from the user, the AR server 40 acquires the information on the corresponding part and the object as information on the AR display object candidate. Then, the tactile sense determination unit 222 of the control unit 20 determines whether or not the AR display object candidate acquired via the portable information terminal 30 is newly acquired. When it is determined that the data is newly acquired, the feedback generation unit 223 generates a haptic feedback pattern based on the position information of the corresponding part and the object.
Alternatively, the haptic process determination unit 222 determines whether or not the display mode of the already acquired AR display object candidate has changed. If it is determined that the display mode has changed, the feedback generation unit 223 generates the haptic feedback pattern. Good. Thereby, the tactile sense providing unit 13 can present information about the AR display object candidate whose display mode has changed by tactile sense. The display mode can change when, for example, the congestion status of the corresponding part, which is the display content of the object, changes.
Thereby, even when the object is not displayed, the user can grasp the approach of the corresponding part, the change of the content of the object, the position information of the object, and can move the visual field toward the object.

(Application example 4)
The AR system 100 can be applied to a game application associated with the real world. In this case, for example, the AR display object does not have an object, and has only objects such as game characters and items.
For example, when the AR server 40 determines that the set position of the object exists within a predetermined distance (for example, several hundred meters to several km) from the user, the AR server 40 acquires the information of the object as the AR display object candidate information. . Then, the tactile processing determination unit 222 of the control unit 20 determines whether or not the information on the AR display object candidate acquired via the portable information terminal 30 is information that is newly acquired. When it is determined that the information is newly acquired, the feedback generation unit 223 generates a haptic feedback pattern based on the position information of the object.
Thereby, the user can grasp that there are enemy characters and items in the vicinity, and can approach them.
Alternatively, the haptic process determination unit 222 determines whether or not the display mode of the already acquired object has changed, and when it is determined that the display has changed, the feedback generation unit 223 may generate a haptic feedback pattern. The display mode can change, for example, when the content of an item that is an object changes or when the battle situation displayed as an object changes. Alternatively, the tactile feedback pattern can also be generated when an enemy character that is an object performs an attacking action or when another user connected via the Internet 50 is newly registered.
Thereby, even when the object is not displayed, the user can grasp the approach of the corresponding part, the change of the content of the object, the position information of the object, and can move the visual field toward the object.

[Modification 1-1]
The generation example of the tactile feedback pattern is not limited to the example shown in FIG. 12 and Table 1. Even if the region to which the rotation angle θ′b with respect to the front direction of the user U of the object belongs is set as follows, Good.
For example, as shown in FIG. 13, the region to which θ′b belongs may be only the first and second regions R1 and R2 in addition to the visual field V. In this case, the first region R1 is represented by the following equation (11), and the second region R2 is represented by the following equation (12).
θ′v2 <θ′b ≦ 180 ° (11)
180 ° <θ′b <θ′v1 (12)
When the object is in the first region R1, since the object exists on the right side of the user, the first vibrator 131R vibrates and the second vibrator 131L does not vibrate. On the other hand, when the object is in the second region R2, since the object exists on the left side of the user, the second vibrator 131L vibrates and the first vibrator 131R does not vibrate.

Further, the strengths of the vibrators 131R and 131L are not limited to the examples shown in FIG. 12 and Table 1, and can be appropriately adjusted.

Alternatively, the information related to the direction of the AR display object candidate based on the center direction of the user's visual field may include information on the height of the object based on the height of the center of the visual field. In this case, the area to which the relative position of the object belongs may be an area set for the value of h in the height direction.
For example, when there are AR display object candidate objects above and below the field of view V, the first and second vibrators 131R and 131L can be vibrated with a specific vibration pattern with an intensity of 5, respectively.

Further, tactile feedback may be generated for objects included in the field of view V.
Thereby, when there is an object or the like that is difficult for the user to watch in the visual field, the presence and position information of the object can be provided to the user by tactile feedback.

[Modification 1-2]
In the above embodiment, for example, the example in which the tactile feedback pattern is generated based on the information on the direction of the AR display object candidate based on the center direction of the user's visual field has been described, but the present invention is not limited thereto.
For example, the feedback generation unit 223 can generate a haptic feedback pattern based on information related to the distance between the AR display object candidate and the user.

FIG. 14 is a schematic diagram illustrating a generation example of a haptic feedback pattern according to the present modification.
The feedback generation unit 223 determines the region to which the target object belongs based on the distance between the target AR display target candidate target and the user U.
As shown in the figure, in the present modification, the first region R11 farthest from the user U, the second region R12 closer to the first region R11, and closer to the second region R12. A third region R13 is set. The distance from the user U in each region can be set as appropriate.
Subsequently, the feedback generation unit 223 refers to a lookup table stored in advance in the memory 202 and determines a vibration pattern of the vibrator 131 according to the determined area.

The graph in FIG. 14 is a graph showing an example of a vibration pattern when the position of the object corresponds to each of the first to third regions R11 to R13, with the horizontal axis representing time and the vertical axis representing vibration intensity.
As shown in these graphs, the vibrator 131 oscillates intermittently, for example, at a predetermined pitch. Referring to the graph of the first region R11, the length of one cycle (referred to as a vibration pitch) in the first region R11 is t1, and the continuous vibration time in each cycle is w1. Similarly, the vibration pitch in the second region R12 is t2, the continuous vibration time in each cycle is w2, the vibration pitch in the third region R13 is t3, and the continuous vibration time in each cycle is w3. The relationship between t1 to t3 and w1 to w3 is expressed by the following equations (13) and (14), respectively.
t1>t2> t3 (13)
w1 <w2 <w3 (14)
That is, according to the vibration pattern, the continuous vibration time is longer and the vibration pitch is shorter as the distance between the user U and the object is shorter. Thereby, even when the object is not displayed, the user can grasp the distance of the target object based on the vibration pitch and the continuous vibration time.

The vibration pattern of this modification is not limited to the above example as long as the vibration pattern is set according to the distance between the user U and the object.
For example, the vibrator 131 may vibrate in a pulse shape instead of continuously. In this case, in the vibration pattern, the number of pulses that are continuously generated may increase as the distance between the user U and the object decreases.
Alternatively, the strength of the vibrator 131 may be changed instead of changing the time during which the vibrator 131 continuously vibrates.
Further, the vibrator 131 may vibrate at different pitches instead of vibrating at equal intervals.
Furthermore, the first and second vibrators 131R and 131L may vibrate with the same vibration intensity, and are set with respect to θ′b [°] as described with reference to FIG. 12 and Table 1. You may vibrate with vibration intensity. As a result, the feedback generation unit 223 detects the tactile feedback pattern based on information related to both the direction of the AR display object candidate based on the center direction of the user's visual field and the distance between the AR display object candidate and the user. Can be generated.

[Modification 1-3]
FIG. 15 is a schematic side view of the HMD 10 of the present modification.
As shown in the figure, the tactile sense presentation unit 13 includes a plurality of vibrators 131a, 131b, and 131c arranged on the second mounting member 121L. The plurality of vibrators 131a, 131b, and 131c are all disposed in the modern portion 123L of the second mounting member 121L.
The plurality of vibrators 131a, 131b, and 131c constitute a vibrator group 132 that is arranged side by side in a predetermined direction. The predetermined direction is, for example, the extending direction of the modern portion 123L, and can be, for example, the Z-axis direction.
The plurality of vibrators 131a, 131b, and 131c included in the vibrator group 132 are sequentially arranged along a predetermined direction when the direction of the AR display object candidate with respect to the center direction of the user's visual field corresponds to the predetermined direction. Can vibrate. A specific example will be described below using the graph in FIG. In this example, it is assumed that the object exists behind the user.
As shown in the graphs, the vibrators 131a, 131b, and 131c have substantially the same vibration pitch and continuous vibration time, but have different vibration timings. Specifically, the vibrator 131b and the vibrator 131c vibrate sequentially from the vibrator 131a toward the rear in the Z-axis direction. Thereby, the user can perceive vibration from the front in the Z-axis direction to the rear, and can intuitively recognize that the object is present in the rear (in the Z-axis direction).

In this modification, the arrangement of the plurality of vibrators is not limited to the above. For example, the tactile sense providing unit 13 may include a transducer group in the first mounting member 121R on the right side in addition to the transducer group 132 disposed on the second mounting member 121L on the left side. The first and second vibrators 131R and 131L described in the above embodiment may constitute a vibrator group. In this case, the vibrator group is arranged in the left-right direction (X-axis direction). For example, when the object corresponds to the right side of the user, the vibrator 131L and the vibrator 131R may vibrate in this order. Good.

[Modification 1-4]
The configuration of the mounting portion 12 of the HMD 10 is not limited to the above example.
FIG. 16 is a top view schematically showing the HMD 10A of the present modification.
As shown in the figure, the mounting portion 12A does not include the first and second mounting members 121R and 121L but includes a band-shaped mounting member 121A. The end of the mounting member 121A is connected to the right end and the left end of the display unit 11, respectively. The mounting member 121A is mounted on the user's head from one temporal region to the occipital region and the other temporal region. The mounting portion 12A may have an earpiece (not shown).
The tactile sense providing unit 13A includes a plurality of vibrators 131Aa, 131Ab, 131Ac, 131Ad, 131Ae, and 131Af arranged on the mounting member 121A. The plurality of vibrators 131Aa to 131Af are arranged along the longitudinal direction of the mounting member 121A, for example, at a portion of the mounting member 121A that is mounted from the user's side to the back of the head.
These vibrators 131Aa to 131Af can provide position information of AR display object candidates to the user by presenting various vibration patterns.

The plurality of vibrators 131Aa to 131Af are, for example, the first vibrator group 132Aa arranged side by side in the X-axis direction and the Z-axis direction, and the second vibrator arranged side by side in the X-axis direction and the Z-axis direction. The vibrator group 132Ab. The first transducer group 132Aa includes transducers 131Aa, 131Ab, and 131Ac. The second transducer group 132Ab includes transducers 131Ad, 131Ae, and 131Af.
For example, when the object is arranged behind the user, the first transducer group 132Aa and / or the second transducer group 132Ab vibrate in order from the front to the rear in the Z-axis direction.
Alternatively, when the object is arranged on the right side of the user, the first transducer group 132Aa vibrates sequentially from the left in the X-axis direction toward the right. Similarly, when the object is arranged on the left side of the user, the second transducer group 132Ab vibrates sequentially from the right in the X-axis direction to the left.
As a result, as in the modified example 1-3, the user can intuitively perceive the position information of the AR display object.

[Modification 1-5]
The tactile sense providing unit 13 is not limited to the configuration having the vibrators 131R and 131L configured by a vibration motor.
The tactile sense presentation unit 13 is, for example, a linear actuator whose weight is reciprocally driven in a uniaxial direction, a “smart material” such as a piezoelectric material, an electroactive polymer or a shape memory alloy, a macro composite fiber actuator, an electrostatic actuator, an electric tactile actuator, You may have an actuator etc. Further, the haptic presentation unit 13 includes, for example, a device that uses electrostatic friction (“ESF”) and ultrasonic surface friction (“USF”), a device that generates an acoustic radiation pressure using an ultrasonic haptic transducer, a haptic substrate, and A device using a flexible or deformable surface, or a launch-type tactile presentation device such as air blowing using an air jet may be included.

FIG. 17 is a schematic top view showing the HMD 10B of this modification.
The tactile sense providing unit 13B of the HMD 10B includes a first linear actuator 133R and a second linear actuator 133L. The first and second linear actuators 133R and 133L are disposed in the first and second modern portions 123R and 123L, respectively, and are, for example, regions that contact the upper pinna of the first and second modern portions 123R and 123L. May be arranged.
The first and second linear actuators 133R and 133L include weights 134R and 134L that reciprocate in predetermined axial directions, respectively. The arrows in FIG. 17 indicate the driving directions of the weights 134R and 134L. In the example shown in FIG. 17, the AR display object candidate S corresponds to the right side of the user U, the weight 134R is driven forward in the Z-axis direction, and the weight 134L is driven backward in the Z-axis direction.

FIG. 18 is a graph for explaining an example of driving the weight 134R shown in FIG. 17, in which the horizontal axis represents time, and the vertical axis represents the front position in the Z-axis direction.
As shown in the figure, first, at the start of tactile feedback presentation, the weight 134R moves forward in the Z-axis direction at a relatively fast speed, reaches a predetermined position, and then moves backward in the Z-axis direction at a relatively slow speed. Return. The linear actuator 133L also has a similar drive profile, although the movement direction of the weight is opposite to that of the linear actuator 133R.
With such a driving profile, the user can obtain a feeling that the user is hit by someone from the front right. Therefore, the HMD 10B can prompt the user to move the field of view to the right, and can present the user with a tactile feedback pattern linked to the action that the user wants to guide.

Alternatively, the tactile sense providing unit 13B may have one linear actuator. Even in such a configuration, the tactile sense providing unit 13B can present to the user a tactile feedback pattern in which the driving direction of the weight and the direction of the AR display object candidate based on the user are linked.

<Second Embodiment>
The display unit 11 of the HMD 10 may be configured to present an auxiliary display that assists information regarding the relative position of the AR display object candidate viewed from the user presented by the tactile feedback pattern in the visual field.
In the present embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

19 and 20 are diagrams illustrating examples of auxiliary display.
FIG. 19 shows an example in which the AR display object candidate exists on the right side of the user. In this case, the auxiliary display D1 moves from left to right along the upper end of the visual field V when presenting tactile feedback.
FIG. 20 shows an example in which the AR display object candidate exists above the visual field. In this case, the auxiliary display D2 moves from below to above along the right end of the visual field V when presenting tactile feedback.
As shown in FIGS. 19 and 20, the auxiliary displays D <b> 1 and D <b> 2 move along the end of the visual field V, so that the influence on other displays in the visual field V can be reduced.
The shapes of the auxiliary displays D1 and D2 may be rectangular, circular or other geometric shapes, and may be other animations or illustrations.

FIG. 21 is a block diagram showing a functional configuration of the control unit 20C according to the present embodiment.
The control unit 20C includes an AR display processing unit 21 and a tactile feedback processing unit 22 as well as an auxiliary display processing unit 23, as in the first embodiment.
The auxiliary display processing unit 23 includes an auxiliary display control unit 231 that executes processing for assisting information related to the relative position of the AR display object candidate viewed from the user, which is presented by the tactile feedback pattern. The auxiliary display control unit 231 is executed by the CPU 201 of the control unit 20.
The auxiliary display control unit 231 is supplied with the determination result from the tactile processing determination unit 222 and is also supplied with information regarding the relative position of the AR display object candidate viewed from the user from the AR information management unit 221. The auxiliary display control unit 231 executes auxiliary display drawing processing and animation processing based on these pieces of information.

According to the present embodiment, by combining the haptic feedback with the auxiliary display, the user can more easily grasp the position information of the AR display object candidate.

[Modification 2-1]
The display mode of the auxiliary display is not limited to the examples shown in FIGS.
For example, as shown in FIG. 22, the auxiliary displays D3 and D4 may display the position of the AR display object candidate, the contents of the object, and the like.
Thereby, the positional information and contents of the AR display object candidate can be grasped in more detail.

Alternatively, as shown in FIG. 23, the auxiliary display D5 is arranged in a partial region of the visual field V (a region along the lower end in the example shown in the figure), and is a schematic of the cylindrical coordinates C0 surrounding the user. It may be a development view. In this case, the auxiliary display D5 has, for example, a region DV corresponding to the developed view of the visual field V at the center, and regions DR and DL on the right and left sides thereof. The center of the auxiliary display D5 (position in the area DV) corresponds to the front direction of the user. Regions DR and DL correspond to the right and left development views outside the field of view. The auxiliary display D5 has a sub-object SB corresponding to the AR display object candidate object. The sub-object SB is arranged at a position in the auxiliary display D5 corresponding to the circumferential direction with the AR display candidate user as a reference.
Thereby, the position information of the AR display object candidate can be accurately grasped by the user.

As mentioned above, although embodiment of this technique was described, this technique is not limited only to the above-mentioned embodiment, Of course, various changes can be added within the range which does not deviate from the summary of this technique.

In the above embodiments, the AR server, the portable information terminal, the control unit, and the HMD have been described as the AR system. However, the present invention is not limited to this.
For example, the AR system does not have a portable information terminal, and the AR server and the control unit may be configured to communicate directly.
Further, the control unit may be configured integrally with the HMD, or may be configured by a portable information terminal. Or a control unit may be comprised by several apparatuses, such as AR server and a portable information terminal, for example.

In addition, the wearable display may be configured to include a mounting unit mounted on the user's head or the like, a display unit not supported by the mounting unit, and a tactile sense providing unit provided in the mounting unit.
In this case, the display unit may be, for example, a contact lens type display device arranged in the user's eye. Alternatively, the display unit may be a portable information terminal such as a smartphone or a display device configured to be attached to spectacles worn by the user.
For example, the mounting portion may have a configuration that hangs on a user's ear or a headphone configuration. Thereby, the mounting part can perceive a tactile sense with relatively high accuracy without blocking the user's visual field.

Furthermore, in the above operation example, necessary AR display object candidate information is acquired from the AR server each time the user's current position changes or the display unit posture changes, but the present invention is not limited to this.
For example, when the control unit (HMD) or the portable information terminal is activated, information of AR display object candidates necessary for the AR display process and the tactile feedback process are collectively acquired from the AR server and stored in the memory. Also good.

For example, in the above embodiment, the example in which the present technology is applied to the HMD has been described, but the present technology can also be applied to, for example, a wearable display that can be attached to a wrist, an arm, a neck, or the like.

In the above embodiment, an example of application to a see-through type (transmission type) HMD has been described. However, the present technology can also be applied to a non-transmission type HMD. In this case, a predetermined object according to the present technology may be displayed in an external field of view captured by a camera attached to the display unit.

Further, in the above embodiments, the AR display object has been described as including a predetermined object existing in the real space and an object including information related thereto, but the present invention is not limited to this, and the AR display object represents the object. You don't have to. In this case, as shown in Application Example 4 above, the object is associated with a predetermined position in the real space, and the control unit uses this position information to determine the distance between the user and the AR display object. It may be configured to calculate.

In addition, this technique can also take the following structures.
(1) a display unit that presents an AR (Augmented Reality) display in the user's field of view;
A mounting portion that can be mounted on the user;
When there is a predetermined change in an AR display object candidate that can be presented as the AR display object, information related to the relative position of the AR display object candidate viewed from the user is presented by a tactile sense using a tactile feedback pattern. A wearable display comprising: a tactile sense providing unit provided in the mounting unit.
(2) The wearable display according to (1) above,
The tactile feedback pattern is generated based on information related to a relative position of the AR display object candidate as viewed from the user.
(3) The wearable display according to (1) or (2) above,
The information regarding the relative position of the AR display object candidate viewed from the user includes the direction of the AR display object candidate based on the center direction of the user's visual field, and the distance between the AR display object candidate and the user. A wearable display that contains information related to at least one of the following.
(4) The wearable display according to any one of (1) to (3) above,
The tactile sense presentation unit includes a plurality of vibrators arranged at different positions of the mounting unit,
Each of the plurality of vibrators vibrates according to a vibration pattern defined based on information on the relative position and an arrangement of the plurality of vibrators.
(5) The wearable display according to (4) above,
The mounting part is
A first mounting member mounted on the right side of the user;
A second mounting member mounted on the left side of the user,
The plurality of vibrators are
A first vibrator disposed on the first mounting member;
A second vibrator disposed on the second mounting member,
The first vibrator vibrates with a stronger intensity than the second vibrator when the direction of the AR display candidate from the user corresponds to the right side of the user, and the AR display candidate When the direction seen from the user corresponds to the left side of the user, a wearable display that vibrates with a lower intensity than the second vibrator.
(6) The wearable display according to (4) or (5) above,
The plurality of vibrators have a vibrator group arranged side by side in a predetermined direction,
The plurality of vibrators included in the vibrator group vibrate in order along the predetermined direction when a direction of the AR display candidate viewed from the user corresponds to the predetermined direction.
(7) The wearable display according to any one of (1) to (6) above,
The mounting part is
A modern part worn on the user's ear;
A support part connected to the modern part and supporting the display part;
The tactile sense presentation unit is a wearable display arranged in the modern unit.
(8) The wearable display according to (7) above,
The modern part has a higher rigidity than the support part.
(9) The wearable display according to any one of (1) to (8) above,
The tactile sense presenting unit presents information related to the relative position of the AR display object candidate as viewed from the user with respect to the newly acquired AR display object candidate by a tactile sense.
(10) The wearable display according to any one of (1) to (9) above,
The tactile sensation providing unit presents information related to the relative position of the AR display object viewed from the user with the tactile sensation for the AR display object candidate whose display mode has changed.
(11) The wearable display according to any one of (1) to (10) above,
The tactile sensation providing unit presents information related to the relative position of the AR display object viewed from the user with respect to the AR display object candidate in which a predetermined change has occurred in the distance between the user and the wearable display.
(12) The wearable display according to any one of (1) to (11) above,
The AR display object candidate is a wearable display associated with a position outside the field of view.
(13) The wearable display according to any one of (1) to (12) above,
The display unit presents an auxiliary display that assists information related to a relative position of the AR display object viewed from the user presented by the tactile feedback pattern in the visual field.
(14) The wearable display according to any one of (1) to (13) above,
The display unit is a wearable display supported by the mounting unit.
(15) A display unit that presents an AR display object in a user's field of view, a mounting unit that can be mounted on the user, and a mounting unit that is provided on the mounting unit, and that provides a tactile feedback pattern to the user from the user. A tactile sensation presentation unit that presents information regarding the relative position of the viewed AR display object candidate by tactile sensation,
A control unit that generates a haptic feedback pattern and outputs the haptic feedback pattern to the haptic presentation unit when there is a predetermined change in an AR display candidate that can be presented as the AR display object. Information processing system.
(16) A display unit that presents an AR (Augmented Reality) display in the user's field of view, a mounting unit that can be mounted on the user, and a tactile sense providing unit provided on the mounting unit. A control method for a wearable display,
When there is a predetermined change in an AR display object candidate that can be presented as the AR display object, information related to the relative position of the AR display object candidate viewed from the user is presented by a tactile sense using a tactile feedback pattern. Yes Wearable display control method.

100 ... Information processing system (AR system)
10, 10A, 10B ... Wearable display (HMD)
DESCRIPTION OF SYMBOLS 11 ... Display part 12, 12A ... Mounting part 121R ... 1st mounting member (mounting member)
121L ... Second mounting member (mounting member)
122 ... support part 123R, 123L ... modern part 13, 13A ... tactile sense presentation part 131 ... multiple vibrators 131R ... first vibrator (vibrator)
131L ... Second vibrator (vibrator)
132 ... vibrator group 20, 20C ... control unit

Claims (16)

1. A display unit that presents AR (Augmented Reality) display in the user's field of view;
   A mounting portion that can be mounted on the user;
   When there is a predetermined change in an AR display object candidate that can be presented as the AR display object, information related to the relative position of the AR display object candidate as viewed from the user is presented by a tactile sense using a tactile feedback pattern. A wearable display comprising: a tactile sense providing unit provided in the mounting unit.
2. The wearable display according to claim 1,
   The tactile feedback pattern is generated based on information related to a relative position of the AR display object candidate as viewed from the user.
3. The wearable display according to claim 1,
   Information on the relative position of the AR display object candidate viewed from the user includes the direction of the AR display object candidate based on the center direction of the user's visual field, and the distance between the AR display object candidate and the user. A wearable display that contains information related to at least one of them.
4. The wearable display according to claim 1,
   The tactile sense presentation unit includes a plurality of vibrators arranged at different positions of the mounting unit,
   Each of the plurality of vibrators vibrates according to a vibration pattern defined based on information on the relative position and an arrangement of the plurality of vibrators.
5. The wearable display according to claim 4,
   The mounting part is
   A first mounting member mounted on the right side of the user;
   A second mounting member mounted on the left side of the user,
   The plurality of vibrators are:
   A first vibrator disposed on the first mounting member;
   A second vibrator disposed on the second mounting member,
   The first vibrator vibrates at a stronger intensity than the second vibrator when the direction of the AR display candidate from the user corresponds to the right side of the user, and the AR display candidate A wearable display that vibrates at a lower intensity than the second vibrator when the direction viewed from the user corresponds to the left side of the user.
6. The wearable display according to claim 4,
   The plurality of vibrators have a vibrator group arranged side by side in a predetermined direction,
   The plurality of vibrators included in the vibrator group vibrate in order along the predetermined direction when a direction of the AR display candidate viewed from the user corresponds to the predetermined direction.
7. The wearable display according to claim 1,
   The mounting part is
   A modern part worn on the user's ear;
   A support part connected to the modern part and supporting the display part;
   The tactile sense presentation unit is a wearable display arranged in the modern unit.
8. The wearable display according to claim 7,
   The modern part has a higher rigidity than the support part.
9. The wearable display according to claim 1,
   The tactile sensation providing unit presents information related to the relative position of the AR display object candidate viewed from the user by tactile sensation for the newly acquired AR display object candidate.
10. The wearable display according to claim 1,
    The tactile sensation providing unit presents information related to the relative position of the AR display object candidate viewed from the user by tactile sensation with respect to the AR display object candidate whose display mode has changed.
11. The wearable display according to claim 1,
    The tactile sense presenting unit presents information related to a relative position of the AR display object candidate viewed from the user by a tactile sense with respect to the AR display object candidate in which a predetermined change has occurred in the distance between the user and the wearable display.
12. The wearable display according to claim 1,
    The AR display object candidate is associated with a position outside the field of view.
13. The wearable display according to claim 1,
    The said display part presents the auxiliary display which assists the information regarding the relative position of the said AR display object candidate seen from the said user shown by the said tactile feedback pattern in the said visual field.
14. The wearable display according to claim 1,
    The display unit is a wearable display supported by the mounting unit.
15. A display unit that presents an AR display object in the user's field of view, a mounting unit that can be mounted on the user, and the mounting unit. A tactile sense presentation unit that presents information related to the relative position of the AR display object candidate by tactile sense;
    A control unit that generates a haptic feedback pattern and outputs the haptic feedback pattern to the haptic presentation unit when there is a predetermined change in an AR display candidate that can be presented as the AR display. Information processing system.
16. A wearable display control method comprising: a display unit that presents an AR display object in a user's field of view; a mounting unit that can be mounted on the user; and a tactile display unit provided in the mounting unit. And
    When there is a predetermined change in an AR display object candidate that can be presented as the AR display object, information related to the relative position of the AR display object candidate as viewed from the user is presented by a tactile sense using a tactile feedback pattern. Yes Wearable display control method.

Images