WO2017073014A1

WO2017073014A1 - Wearable display, image display device, and image display system

Info

Publication number: WO2017073014A1
Application number: PCT/JP2016/004319
Authority: WO
Inventors: 博隆石川; 孝文朝原; 岩津　健; 健渋井; 鈴木　謙治; 智英田辺
Original assignee: ソニー株式会社
Priority date: 2015-10-30
Filing date: 2016-09-23
Publication date: 2017-05-04

Abstract

A wearable display according to an embodiment of the present technology includes a display unit and a control unit. The display unit is configured to be wearable by a user and has a display region that provides the user with a view of real space. The control unit extracts image information associated with specific objects existing in the direction the user moves from a plurality of image information on the basis of user information including the information associated with the position and action of the user, and the plurality of image information associated with a plurality of specific objects existing in a peripheral region within a first distance from the user, and displays the extracted image information in the display region. According to the present technology, the information expected to be useful to a user can be preferentially displayed.

Description

Wearable display, image display device, and image display system

The present technology relates to a wearable display, an image display device, and an image display system capable of displaying an image including specific information in a display visual field.

A technology called augmented reality (AR) that adds a corresponding image to a real space or a screen that displays the real space is known. For example, Patent Document 1 describes a see-through type head-mounted display (HMD: Head-Mounted-Display) that can superimpose and display an image (AR object) associated with an object existing in real space. Has been.

International Publication No. 2014/128810

In this type of head mounted display, since AR objects related to objects existing around the user are displayed, not only information useful to the user but also unnecessary information may be presented.

For example, during the execution of the navigation mode that provides the user with route information to the destination, not only information related to the road that is currently running, but also information related to other nearby roads may be displayed. . Such information related to other roads is often unnecessary for the user, and the user is forced to judge which road the displayed information is related to. There was a risk of errors.

In view of the circumstances as described above, an object of the present technology is to provide a wearable display, an image display device, and an image display system capable of preferentially presenting information presumed to be useful to a user.

A wearable display according to an embodiment of the present technology includes a display unit and a control unit.
The display unit is configured to be attachable to the user, and includes a display area that provides the user with a real field of view.
The control unit is based on user information including information related to the position and behavior of the user and a plurality of pieces of image information related to a plurality of specific objects existing in a peripheral region within a first distance from the user. The image information related to the specific object existing in the moving direction of the user is extracted from the plurality of image information, and the extracted image information is presented in the display area.

The wearable display is useful for the user because it is configured to extract image information related to a specific object existing in the moving direction of the user from a plurality of pieces of image information and present it in the visual field area. Can be preferentially presented.

The user information may include map information around the position of the user, and the control unit extracts image information related to a specific object existing along a road on which the user travels from the plurality of image information. It may be configured to.
Thereby, only the information relevant to the road where the user is currently traveling or passing can be presented to the user.

The control unit obtains the moving speed of the user as the user information, and when the moving speed is equal to or lower than a predetermined value, the specific target existing within a second distance shorter than the first distance from the user. Image information related to the object may be presented in the display area.
In this way, the display range of the image may be changed according to the moving speed of the user.

The control unit may be configured to be able to present an image including the map information and the image information in the display area.
In this case, the map information may be displayed like a bird's eye view or a bird's eye view including image information.

The control unit may be configured to acquire the user's movement acceleration as the user information and change a display mode of the image information presented in the display area in accordance with the magnitude of the movement acceleration. .
When a user is driving an automobile or a motorcycle, the user's attention needs to be concentrated on driving under a predetermined situation such as acceleration / deceleration. In this case, according to the configuration of the control unit, it is possible to intentionally reduce the visibility of the image information presented in the display area so that the user's attention is directed to the real field of view instead of the image. .

For example, the control unit presents the image information at the first luminance in the display area when the user's moving acceleration is equal to or less than a predetermined value, and when the user's moving acceleration exceeds the predetermined value. The image information is presented in the display area with a second luminance lower than the first luminance. In this case, the control unit may gradually decrease the second luminance as the user's movement acceleration increases.

Alternatively, the control unit may be configured to change at least one of a position, a size, and an information amount of the image information presented in the display area according to the magnitude of the acceleration.

The wearable display may further include a detection unit capable of detecting an acceleration acting on the display unit as the user information. The detection unit may be configured by a motion sensor such as an acceleration sensor or an angular velocity sensor, or may be configured by, for example, a camera installed in a display unit.

An image display device according to an embodiment of the present technology includes a display unit and a control unit.
The display unit has a display area.
The control unit is based on user information including information related to a user's position and behavior, and a plurality of image information related to a plurality of specific objects existing in a peripheral region within a first distance from the user, Image information related to a specific object existing in the moving direction of the user is extracted from the plurality of image information, and the extracted image information is presented in the display area.

An image display system according to an embodiment of the present technology includes a display unit, an acquisition unit, and a control unit.
The display unit has a display area.
The acquisition unit acquires user information including information related to a user's position and behavior, and a plurality of pieces of image information related to a plurality of specific objects existing in a peripheral region within a first distance from the user. .
The control unit extracts image information related to a specific object existing in the moving direction of the user from the plurality of image information, and presents the extracted image information in the display area.

As described above, according to the present technology, it is possible to preferentially present information that is estimated to be useful to the user.
Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

It is the schematic explaining the function of the wearable display (HMD) which concerns on one Embodiment of this technique. It is a figure explaining an example of the function of the above-mentioned HMD, and is a schematic diagram of a visual field presented on a display part. It is a figure explaining an example of the function of the above-mentioned HMD, and is a schematic diagram of a visual field presented on a display part. 1 is an overall view of a system including the HMD. It is a block diagram which shows the structure of the said system. It is a functional block diagram of the control unit in the HMD. It is explanatory drawing of the coordinate position in the cylindrical coordinate system in the said HMD. It is an expanded view of the said cylindrical coordinate which shows notionally the relationship between a visual field and an object. It is a figure explaining the conversion method from a cylindrical coordinate (world coordinate) to a visual field (local coordinate). It is a figure explaining the conversion method from a cylindrical coordinate (world coordinate) to a visual field (local coordinate). It is a flowchart explaining the outline | summary of operation | movement of the said system. It is a figure explaining one effect | action of said HMD. It is a schematic diagram of the visual field explaining one effect | action of said HMD. It is a schematic diagram of the visual field explaining one effect | action of said HMD. 12 is a flowchart for explaining the operation of a wearable display (HMD) according to a second embodiment of the present technology. It is a figure explaining one effect | action of said HMD. It is a block diagram which shows the example of a structure of a part of said HMD. It is a schematic diagram of the visual field explaining one effect | action of said HMD. It is a schematic diagram of the visual field explaining one effect | action of said HMD. It is a figure explaining the application example of the wearable display (HMD) which concerns on 3rd Embodiment of this technique. It is a figure explaining the other application example of the said HMD. It is a figure explaining the other application example of the said HMD.

Hereinafter, embodiments of the present technology will be described with reference to the drawings.

<First Embodiment>
[Schematic configuration of HMD system]
FIG. 1 is a schematic diagram illustrating functions of a head mounted display (hereinafter referred to as “HMD”) as a wearable display according to an embodiment of the present technology.
In FIG. 1, an X-axis direction and a Y-axis direction indicate horizontal directions orthogonal to each other, and a Z-axis direction indicates a vertical axis direction. These XYZ orthogonal coordinate systems represent a coordinate system (real three-dimensional coordinate system) of the real space to which the user belongs, the X-axis arrow indicates the north direction, and the Y-axis arrow indicates the east direction. The Z-axis arrow indicates the direction of gravity.

First, an outline of basic functions of the HMD 100 according to the present embodiment will be described with reference to FIG.

[Overview of HMD functions]
The HMD 100 according to the present embodiment is mounted on the head of the user U and can display a virtual image (AR object; hereinafter also referred to as an object) in the field of view V (display field of view) of the user U. Composed. The object displayed in the field of view V relates to a specific object (A1, A2, A3, A4,..., Hereinafter collectively referred to as a specific object A unless otherwise described). Information other than the specific object A is included.

More specifically, the specific object A corresponds to, for example, scenery, shops, products existing around the user U, and terminals of transportation such as bus stops, stations, and airports. As information related to the specific object, as schematically illustrated in FIG. 2, an object B10 that indicates that a specific coupon is available in a specific store A10 with a visual field V is displayed.

On the other hand, the information other than the specific object A includes, for example, information related to the route to the destination set by the user, and is related to the orientation of the display unit 10 as schematically shown in FIG. Then, an object B20 composed of “arrows” indicating the traveling direction of the road or passage is displayed. In addition, a menu screen for setting the functions of the HMD 100, a pattern authentication screen, and the like are applicable.

Referring to FIG. 1, HMD 100 includes objects (B 1, B 2, B 3, B 4,..., Which are individually described below, surrounding a virtual world coordinate system surrounding user U wearing HMD. Except for object B). The world coordinate system is a coordinate system equivalent to the real space to which the user belongs, and determines the position of the specific object A based on the position of the user U and a predetermined axial direction. In the present embodiment, cylindrical coordinates C0 with the vertical axis as the center axis are adopted as the world coordinates, but other three-dimensional coordinates such as celestial coordinates centered on the user U may be adopted.

The radius R and height H of the cylindrical coordinates C0 can be set arbitrarily. Here, the radius R is set shorter than the distance from the user U to the specific object A, but may be longer than the distance. The height H is set to a size equal to or greater than the height (length in the vertical direction) Hv of the user's U visual field V provided via the HMD 100.

As described above, the object B includes information related to the specific object A existing in the world coordinate system or information not related to the specific object A. The object B may be an image including characters and pictures or an animation image. The object B may be a two-dimensional image or a three-dimensional image. Furthermore, the shape of the object B may be any arbitrary or significant geometric shape such as a rectangle, a circle, or the like, and can be appropriately set depending on the type (attribute) of the object B and the display content.

The coordinate position of the object B on the cylindrical coordinates C0 is associated with, for example, the intersection position between the user's eye line L watching the specific object A and the cylindrical coordinates C0. In the example of FIG. 1, the center position of each of the objects B1 to B4 is made coincident with the intersection position. However, the present invention is not limited to this, and a part of the periphery of the objects B1 to B4 (for example, part of the four corners) You may match. Alternatively, the coordinate positions of the objects B1 to B4 may be associated with any position away from the intersection position.

The cylindrical coordinate C0 is a height in the height direction that represents a vertical coordinate axis (θ) that represents an angle around the vertical axis with the north direction being 0 °, and a horizontal line of sight Lh of the user U. Coordinate axis (h). The coordinate axis (θ) has a positive direction around the east, and the coordinate axis (h) has a depression angle as a positive direction and an elevation angle as a negative direction.

As will be described later, the HMD 100 has a detection unit for detecting the viewpoint direction of the user U. Based on the output of the detection unit, the HMD 100 corresponds to which region on the cylindrical coordinate C0 the user's visual field V corresponds to. Judge whether to do. When any object (for example, object B1) exists in the corresponding area of the xy coordinate system that forms the visual field V, the HMD 100 presents (draws) the object B1 in the visual field V in the corresponding area.

As described above, the HMD 100 of the present embodiment displays information related to the target object A1 to the user U by displaying the AR object (B1) in the field of view V superimposed on the specific target object A1 in the real space. To do. Further, the HMD 100 presents to the user U AR objects (B1 to B4) related to predetermined specific objects A1 to A4 according to the orientation or direction of the viewpoint of the user U.

Subsequently, details of the HMD 100 will be described. 4 is an overall view showing the HMD 100, and FIG. 5 is a block diagram showing the configuration thereof.

[Configuration of HMD]
The HMD 100 includes a display unit 10, a detection unit 20 that detects the attitude of the display unit 10, and a control unit 30 that controls driving of the display unit 10. In the present embodiment, the HMD 100 is configured as a see-through HMD that can provide a user with a visual field V in real space.

(Display section)
The display unit 10 is configured to be attachable to the user U's head. The display unit 10 includes first and second display surfaces 11R and 11L, first and second

image generation units

12R and 12L, and a support body 13.

The first and second display surfaces 11R and 11L are configured by an optical element having a light-transmissive display region 110 that can provide a real space visual field (external field visual field) to the right eye and the left eye of the user U, respectively. . The first and second

image generation units

12R and 12L are configured to be able to generate images to be presented to the user U via the first and second display surfaces 11R and 11L, respectively. The support 13 supports the display surfaces 11R and 11L and the

image generation units

12R and 12L, and the user's head so that the first and second display surfaces 11L and 11R face the right eye and the left eye of the user U, respectively. It has an appropriate shape that can be attached to the part.

The display unit 10 configured as described above is configured to provide the user U with a field of view V in which a predetermined image (or virtual image) is superimposed on the real space via the display surfaces 11R and 11L. Is done. In this case, the cylindrical coordinates C0 for the right eye and the cylindrical coordinates C0 for the left eye are set, and the objects drawn on the cylindrical coordinates are respectively projected onto the display areas 110 of the display surfaces 11R and 11L.

(Detection unit)
The detection unit 20 is configured to be able to detect a change in orientation or posture around at least one axis of the display unit 10. In the present embodiment, the detection unit 20 is configured to detect a change in orientation or posture of the display unit 10 around the X, Y, and Z axes.

Here, the orientation of the display unit 10 typically means the front direction of the display unit 10. In the present embodiment, the orientation of the display unit 10 is defined as the face direction of the user U.

In addition, the detection unit 20 detects a motion sensor such as an acceleration sensor or an angular velocity sensor that can detect the accelerations in the three axis directions acting on the display unit 10, and illuminance (brightness) around the display unit 10. An illuminance sensor or the like that can be used. In addition, the detection unit 20 may include an image sensor (camera), and may be configured to detect acceleration, movement speed, illuminance, and the like acting on the display unit 10.

The detection unit 20 may be configured by a sensor unit in which each of the angular velocity sensor and the acceleration sensor is arranged in three axial directions, or may use different sensors depending on the respective axes. For example, an integrated value of the output of the angular velocity sensor can be used as the posture change of the display unit 10, the direction of the change, the amount of the change, and the like.

Further, a geomagnetic sensor may be employed for detecting the orientation of the display unit 10 around the vertical axis (Z axis). Alternatively, the geomagnetic sensor and the motion sensor may be combined. As a result, it is possible to detect the orientation or orientation change of the display unit 10 with high accuracy.

The detection unit 20 is arranged at an appropriate position on the display unit 10. The position of the detection unit 20 is not particularly limited. For example, the detection unit 20 is disposed on one of the

image generation units

12R and 12L or a part of the support 13. The number of detection units 20 is not limited to one, and a plurality of detection units 20 may be arranged at appropriate positions on the display unit 10.

(Controller unit)
The control unit 30 generates a control signal for controlling the driving of the display unit 10 (

image generation units

12R and 12L) based on the output of the detection unit 20. In the present embodiment, the control unit 30 is electrically connected to the display unit 10 via a connection cable 30a. Of course, the present invention is not limited to this, and the control unit 30 may be connected to the display unit 10 through a wireless communication line.

As shown in FIG. 5, the control unit 30 includes a CPU 301, a memory 302 (storage unit), a transmission / reception unit 303, an internal power supply 304, and an input operation unit 305.

CPU 301 controls the operation of the entire HMD 100. The memory 302 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, a program for executing control of the HMD 100 by the CPU 301, various parameters, an image (object) to be displayed on the display unit 10, and the like. Store the necessary data. The transmission / reception unit 303 constitutes an interface for communication with the portable information terminal 200 described later. The internal power supply 304 supplies power necessary for driving the HMD 100.

The input operation unit 305 is for controlling an image displayed on the display unit 10 by a user operation. The input operation unit 305 may be configured with a mechanical switch or a touch sensor. The input operation unit 305 may be provided in the display unit 10.

The HMD 100 may further include a sound output unit such as a speaker, a camera, and the like. In this case, the audio output unit and the camera are typically provided in the display unit 10. Furthermore, the control unit 30 may be provided with a display device that displays an input operation screen or the like of the display unit 10. In this case, the input operation unit 305 may be configured by a touch panel provided in the display device.

(Personal digital assistant)
The portable information terminal 200 is configured to be able to communicate with the control unit 30 via a wireless communication line. The portable information terminal 200 has a function (acquisition unit) for acquiring an image (object) to be displayed on the display unit 10 and a function for transmitting the acquired image (object) to the control unit 30. The portable information terminal 200 is organically combined with the HMD 100 to construct an HMD system (image display system).

The portable information terminal 200 is carried by a user U who wears the display unit 10 and includes an information processing device such as a personal computer (PC: Personal Computer), a smartphone, a mobile phone, a tablet PC, or a PDA (Personal Digital Assistant). However, a terminal device dedicated to the HMD 100 may be used.

As shown in FIG. 5, the portable information terminal 200 includes a CPU 201, a memory 202, a transmission / reception unit 203, an internal power supply 204, a display unit 205, a camera 206, and a position information acquisition unit 207.

The CPU 201 controls the operation of the mobile information terminal 200 as a whole. The memory 202 includes a ROM, a RAM, and the like, and stores programs and various parameters for executing control of the portable information terminal 200 by the CPU 201, images (objects) transmitted to the control unit 30, and other necessary data. The internal power supply 204 supplies power necessary for driving the portable information terminal 200.

The transmission / reception unit 203 is connected to the server N, the control unit 30, other nearby portable information terminals, etc. using a wireless LAN (IEEE802.11, etc.) such as WiFi (Wireless Fidelity) or a 3G or 4G network for mobile communication. connect. The portable information terminal 200 downloads an image (object) to be transmitted to the control unit 30 from the server N via the transmission / reception unit 203, an application for displaying the image, and the like, and stores them in the memory 202.

The display unit 205 is composed of, for example, an LCD or an OLED, and displays various menus, application GUIs, and the like. Typically, the display unit 205 is integrated with a touch panel and can accept a user's touch operation. The portable information terminal 200 is configured to be able to input a predetermined operation signal to the control unit 30 by a touch operation on the display unit 205.

The location information acquisition unit 207 typically includes a GPS (Global Positioning System) receiver. The portable information terminal 200 measures the current position (longitude, latitude, altitude) of the user U (display unit 10) using the position information acquisition unit 207, calculates the moving speed of the user U (display unit 10), and Map information such as necessary images (objects) and road maps can be acquired from N. That is, the server N acquires information (user information) regarding the current position and behavior of the user, and transmits image data, application software, map information, and the like corresponding to the information to the portable information terminal 200.

The server N is typically configured by a computer including a CPU, a memory, and the like, and transmits predetermined information to the portable information terminal 200 in response to a request from the user U or automatically regardless of the intention of the user U. . The server N stores a plurality of types of image data that can be displayed by the HMD 100. The server N is configured to be able to transmit a plurality of image data selected according to the position, operation, etc. of the user U to the portable information terminal 200 collectively or sequentially as a part of the predetermined information. Is done.

More specifically, in the present embodiment, the mobile information terminal 200 displays image information related to an AR object related to a plurality of specific objects existing in a peripheral area within a predetermined distance from the current position of the user U (display unit 10). Send to. The predetermined distance may be fixed or variable, and typically, the mobile information terminal 200 voluntarily or based on an instruction from the control unit 30, the mobile information terminal 200 sends a server The predetermined distance is instructed to N.

(Details of control unit)
Next, details of the control unit 30 will be described.

FIG. 6 is a functional block diagram of the CPU 301. The CPU 301 includes a coordinate setting unit 311, an image management unit 312, a coordinate determination unit 313, an image extraction unit 314, and a display control unit 315. The CPU 301 executes processing in the coordinate setting unit 311, the image management unit 312, the coordinate determination unit 313, the image extraction unit 314, and the display control unit 315 according to the program stored in the memory 302.

The coordinate setting unit 311 is configured to execute processing for setting three-dimensional coordinates surrounding the user U (display unit 10). In this example, cylindrical coordinates C0 (see FIG. 1) centered on the vertical axis Az are used as the three-dimensional coordinates. The coordinate setting unit 311 sets the radius R and the height H of the cylindrical coordinates C0. The coordinate setting unit 311 typically sets the radius R and the height H of the cylindrical coordinates C0 according to the number and type of objects to be presented to the user U.

The radius R of the cylindrical coordinates C0 may be a fixed value, but may be a variable value that can be arbitrarily set according to the size (pixel size) of the image to be displayed. The height H of the cylindrical coordinates C0 is set to, for example, 1 to 3 times the height Hv (see FIG. 1) of the vertical direction (vertical direction) of the visual field V provided to the user U by the display unit 10. Is done. The upper limit of the height H is not limited to 3 times Hv, but may be a size exceeding 3 times Hv.

As described above, the cylindrical coordinate C0 represents the angle in the vertical direction with reference to the circumferential coordinate axis (θ) representing the angle around the vertical axis with the north direction being 0 ° and the horizontal line of sight Lh of the user U. And a coordinate axis (h) in the height direction. The coordinate axis (θ) has a positive direction around the east, and the coordinate axis (h) has a depression angle as a positive direction and an elevation angle as a negative direction. The height h represents the size when the height Hv of the visual field V is 100%, and the origin OP1 of the cylindrical coordinates C0 is the north direction (0 °) and the horizontal line of sight of the user U It is set at the intersection with Lh (h = 0%).

The image management unit 312 has a function of managing the images stored in the memory 302. For example, the image management unit 312 stores one or a plurality of images displayed via the display unit 10 in the memory 302 and stores them in the memory 302. It is configured to execute a process of selectively deleting the processed image. An image stored in the memory 302 is transmitted from the portable information terminal 200. The image management unit 312 also requests the portable information terminal 200 to transmit an image via the transmission / reception unit 303.

The memory 302 is configured to store one or a plurality of images (objects) to be displayed in the visual field V in association with the cylindrical coordinates C0. That is, the memory 302 stores the individual objects B1 to B4 on the cylindrical coordinates C0 shown in FIG. 1 together with the coordinate positions on the cylindrical coordinates C0.

As shown in FIG. 7, the cylindrical coordinate system (θ, h) and the orthogonal coordinate system (X, Y, Z) have a relationship of X = r cos θ, Y = rsin θ, and Z = h. As shown in FIG. 1, each of the objects B1 to B4 to be displayed corresponding to the azimuth or orientation of the visual field V occupies a coordinate area on a specific cylindrical coordinate C0, and specific coordinates in that area It is stored in the memory 302 together with the position P (θ, h).

The coordinates (θ, h) of the objects B1 to B4 on the cylindrical coordinates C0 are a straight line connecting the positions of the objects A1 to A4 defined by the orthogonal coordinate system (X, Y, Z) and the position of the user, Corresponding to the coordinates of the cylindrical coordinate system at the intersection of the cylindrical coordinates C0 with the cylindrical surface. That is, the coordinates of the objects B1 to B4 correspond to the coordinates of the objects A1 to A4 converted from the actual three-dimensional coordinates to the cylindrical coordinates C0, respectively. For example, the coordinate conversion of such an object is executed in the image management unit 312 and each object is stored in the memory 302 together with the coordinate position. By adopting the cylindrical coordinates C0 in the world coordinate system, the objects B1 to B4 can be drawn in a plane.

The coordinate positions of the objects B1 to B4 may be set to any position within the display area of each object B1 to B4, or may be set to one specific point (for example, the center position) for each object. Two or more points (for example, two diagonal points or four corner points) may be set.

Also, as shown in FIG. 1, when the coordinate positions of the objects B1 to B4 are associated with the intersection position of the user's line of sight L that looks at the objects A1 to A4 and the cylindrical coordinates C0, the user U has the object B1. To B4 are visually recognized at positions overlapping the objects A1 to A4. Instead, the coordinate positions of the objects B1 to B4 may be associated with an arbitrary position away from the intersection position. Thus, the objects B1 to B4 can be displayed or drawn at desired positions with respect to the objects A1 to A4.

The coordinate determination unit 313 is configured to execute a process of determining which region on the cylindrical coordinates C0 corresponds to the field of view V of the user U based on the output of the detection unit 20. That is, the visual field V moves on the cylindrical coordinates C0 due to a change in the posture of the user U (display unit 10), and the movement direction and movement amount are calculated based on the output of the detection unit 20. The coordinate determination unit 313 calculates the movement direction and movement amount of the display unit 10 based on the output of the detection unit 20, and determines to which region on the cylindrical coordinates C0 the visual field V belongs.

FIG. 8 is a development view of the cylindrical coordinates C0 conceptually showing the relationship between the visual field V on the cylindrical coordinates C0 and the objects B1 to B4. The field of view V is substantially rectangular and has xy coordinates (local coordinates) with the upper left corner as the origin OP2. The x axis is an axis extending in the horizontal direction from the origin OP2, and the y axis is an axis extending in the vertical direction from the origin OP2. The coordinate determination unit 313 is configured to execute processing for determining whether any of the objects B1 to B4 exists in the corresponding region of the visual field V.

The image extraction unit 314 extracts image information related to the specific object existing in the moving direction of the user U (display unit 10) from the plurality of pieces of image information (AR object) stored in the memory 302. The extraction method is set as appropriate according to the function (type of application) executed by the HMD 100, and typically includes an AR object or the like of a specific object that exists along a road on which the user U runs or passes.

Also, the image extraction unit 314 is configured to extract image information according to the type of user information acquired from the detection unit 20 or the portable information terminal 200. For example, when the user U is stationary, the image extraction unit 314 extracts all-around object data corresponding to the orientation of the display unit 10 (the line of sight of the user U), and the user U is moving. In this case, the object data along the moving direction is extracted.

Further, the image extraction unit 314 may be configured to extract only the objects related to the specific object existing around the user U when the moving speed of the user U (display unit 10) is equal to or lower than a predetermined value. Good.

The image information extraction method in the image extraction unit 314 is preset according to the type of application executed by the HMD 100 as described above, but may be selected or set by an input operation by the user. Note that a function that can invalidate the function of the image extraction unit 314 described above may be set.

The display control unit 315 performs a process of presenting (drawing) the image information (object) extracted by the image extraction unit 314 in the visual field V based on the output of the detection unit 20 (that is, the determination result of the coordinate determination unit 313). Configured to do. For example, as shown in FIG. 8, when the current orientation of the visual field V overlaps the display areas of the objects B1 and B2 on the cylindrical coordinates C0, images corresponding to the overlapping areas B10 and B20 are displayed in the visual field V. (Local rendering: Local） Rendering).

9A and 9B are diagrams for explaining a conversion method from the cylindrical coordinates C0 (world coordinates) to the visual field V (local coordinates).

As shown in FIG. 9A, the coordinates of the reference point of the visual field V on the cylindrical coordinates C0 are (θv, hv), and the coordinates of the reference point of the object B located in the region of the visual field V are (θ0, h0). The reference point of the field of view V and the object B may be set to any point, and in this example, the reference point is set at the upper left corner of the field of view V and the object B which are rectangular. αv [°] is the width angle of the visual field V on the world coordinates, and its value is determined by the design or specification of the display unit 10.

The display control unit 315 determines the display position of the object B in the visual field V by converting the cylindrical coordinate system (θ, h) to the local coordinate system (x, y). As shown in FIG. 9B, if the height and width of the field of view V in the local coordinate system are Hv and Wv, respectively, and the coordinates of the reference point of the object B in the local coordinate system (x, y) are (x0, y0), conversion is performed. The formula is as follows.
x0 = (θ0−θv) · Wv / αv (1)
y0 = (h0−hv) · Hv / 100 (2)

The display control unit 315 moves the object B in the field of view V in the direction opposite to the moving direction of the display unit 10 in accordance with the change in orientation of the display unit 10. That is, the display control unit 315 changes the display position of the object B within the field of view V following the change in the orientation or orientation of the display unit 10. This control is continued as long as at least part of the object B exists in the visual field V.

[Operation of HMD]
Next, typical operations of the HMD 100 will be described.

FIG. 10 is a flowchart for explaining an outline of the operation of the HMD system according to the present embodiment.

First, the current position of the user U (display unit 10) is measured using the position information acquisition unit 207 of the portable information terminal 200 (step 101). The position information of the display unit 10 is transmitted to the server N. Then, the portable information terminal 200 acquires from the server N object data, map information, and the like related to a specific object existing in a peripheral area (real space) within a predetermined distance (first distance) from the user U ( Step 102).

The predetermined distance is not particularly limited, and is set, for example, within a radius of several hundred meters to several kilometers centering on the user U. The predetermined distance may be set by default or may be set by a user input operation.

Next, the portable information terminal 200 notifies the control unit 30 that it is ready to send object data. The control unit 30 (in this example, the coordinate setting unit 311) sets the height (H) and radius (R) of the cylindrical coordinates C0 as the world coordinate system according to the type of object data and the like (step 103).

Subsequently, the control unit 30 detects the orientation of the visual field V based on the output of the detection unit 20 (step 104), acquires the object data from the portable information terminal 200, and stores it in the memory 302 (step 105). The orientation of the visual field V is converted into the world coordinate system (θ, h), and it is monitored which position on the cylindrical coordinates C0 corresponds to.

Next, the control unit 30 extracts object data to be displayed on the display unit 10 (display area 110, field of view V) from the object data stored in the memory 302 (step 106).

At this time, when the control unit 30 (image extraction unit 314) determines that the user U is stationary from the user information acquired from the detection unit 20 or the portable information terminal 200, the orientation of the display unit 10 (the user U's) All surrounding object data corresponding to the line of sight is extracted. Further, when the control unit 30 determines that the user U (display unit 10) is moving, the control unit 30 extracts object data existing in the moving direction.

At this time, if the moving speed of the user U is greater than or equal to a predetermined value, the control unit 30 extracts the object related to the specific object existing relatively far in the moving direction, and moves the user U When the speed is less than or equal to a predetermined value, only object data related to a specific object existing around the user U may be extracted.

When any object data is present in the corresponding region of the visual field V at the cylindrical coordinates C0, the control unit 30 displays (draws) the object at the corresponding position of the visual field V via the display unit 10 ( Step 107).

Thereafter, the above operation is repeatedly executed. That is, the current position of the user U (display unit 10) is measured again (step 101), and when the current position of the user has changed by, for example, 50 meters or more since the object data was previously acquired from the server N, the current position Is sent from the server N to the portable information terminal 200 (step 102). In the control unit 30, object data related to the moving direction of the user U is extracted from the new object data transmitted from the portable information terminal 200, and is drawn on the display unit 10. Such control is repeatedly executed.

Note that object data that is more than a predetermined distance (first distance) away from the user may be deleted from the memory 202 of the portable information terminal 200 and the memory 302 of the control unit 30 as the user moves. Thereby, the capacity of the

memories

202 and 302 can be reduced.

Next, a specific application example will be described. In this example, an application example of image information presented to a user U driving a car will be described.

FIG. 11 is a conceptual diagram showing a positional relationship between the current position of the user U who is riding a moving body such as an automobile or a motorcycle and traveling north on the road R101 at a predetermined speed, and a specific object around it. When the server N acquires information about the current position from the portable information terminal 200, the server N transmits object data and map information related to the specific objects A12 to A22 within a radius of, for example, 10 km to the portable information terminal 200. The portable information terminal 200 transmits the received object data and map information related to the specific objects A12 to A22 to the control unit 30.

The control unit 30 extracts object data related to the specific objects A12, A13, A14, A16, A19, and A22 existing along the road R101 from the received object data, and displays the image (object) on the display unit 10. indicate.

As described above, in the present embodiment, only image information related to a specific object existing in the moving direction of the user is presented to the user from among the plurality of image information, and thus information presumed to be useful to the user is preferential. Will be presented. Accordingly, images related to the roads other than the road R101 on which the user is currently traveling are not displayed, so that necessary information can be obtained without causing confusion or error to the user. Can be accurately presented to the user. In addition, since a large amount of information is not presented to the user U, it is possible to minimize the effort of selecting information by the user.

At this time, when the moving speed of the user U is equal to or lower than the predetermined speed, as shown in FIG. 12A, a comparison within a second distance (for example, a radius of 1 km) shorter than the first distance (radius of 10 km) from the user. Only information on the target neighborhood (objects B12, B13, B14, B16) may be presented on the display unit 10. When the moving speed of the user U exceeds the predetermined speed, as shown in FIG. 12B, the display unit 10 includes relatively distant information (objects B19, B22) within a radius of 1 km to 10 km, for example. May be presented. In this way, by changing the range of information to be displayed according to the moving speed of the user, it is necessary to see only nearby information when traveling at low speed, or to quickly check distant information when traveling at high speed. Can also be supported.

Also, when the navigation mode is executed as an application program, the user's travel route can be estimated from the set route. Therefore, in addition to objects related to the road along which the vehicle is currently traveling, objects related to the route along the set route can also be extracted and presented on the display unit 10.

Furthermore, by detecting a predetermined operation or a predetermined posture by the user U, an image (screen) including map information and image information shown as a bird's-eye view or a bird's-eye view as shown in FIG. It may be configured to present. Thereby, it is possible to present a list of objects related to the surrounding specific structure to the user.

<Second Embodiment>
Subsequently, a second embodiment of the present technology will be described. Hereinafter, the configuration different from the first embodiment will be mainly described, and the description of the same configuration as the above-described embodiment will be omitted or simplified.

When a user wearing an HMD drives a vehicle (moving body) such as an automobile or a motorcycle, the user may not be able to concentrate on driving depending on the brightness of the display image. In particular, when accelerating, decelerating, or turning, it is necessary to focus the user on the surrounding real space environment.
Therefore, in the present embodiment, a configuration of the HMD that can provide an environment in which the user can concentrate on driving will be described.

The control unit 30 (display control unit 315) in the HMD of this embodiment acquires the user's movement acceleration as user information, and changes the display mode of the image information presented in the display area according to the magnitude of the movement acceleration. Configured to let

FIG. 13 is a flowchart showing an example of the operation of the control unit 30.

As in the first embodiment, the control unit 30 obtains object data, map information, and moving speed information of the peripheral area based on the current position of the user from the portable information terminal 200, and the user presents the current data from these information. Object data along the traveling road is extracted (steps 201 to 205). And the control unit 30 determines the display mode of the extracted object based on the user's movement acceleration information acquired from the detection part 20, and displays this on the display part 10 (steps 206 and 207). The acceleration of the display unit 10 detected by an acceleration sensor built in the detection unit 20 is referred to as the user's moving acceleration.

When the user is driving a car or a motorcycle, it is necessary to concentrate the user's attention on the driving under certain conditions such as acceleration / deceleration. Therefore, the HMD of the present embodiment is configured to intentionally reduce the visibility of the displayed object so that the user's attention is directed to the field of view of the real space instead of the object.

Typically, the control unit 30 (display control unit 315) has at least one of brightness (brightness), color, contrast, position, size, information amount, etc. of the object according to the magnitude of the user's moving acceleration. By changing one, the user's attention is directed to the real space environment. Specifically, the greater the acceleration, the lower the object's visibility by reducing the display brightness of the object, changing to a lighter color, or reducing the contrast. Alternatively, as the acceleration increases, the front position of the real space is secured by moving the position of the object from the center to the peripheral edge of the field of view, reducing the size, or reducing the amount of information.

(Change object brightness)
The control unit 30 (display control unit 315) presents the object to the display area 110 with the first luminance when the user's moving acceleration is below a predetermined value, and when the user's moving acceleration exceeds the predetermined value. The object is displayed on the display area 110 with a second luminance lower than the first luminance.

The user's moving acceleration is typically the magnitude of a combined vector of accelerations along the three-axis directions of the X-axis, Y-axis, and Z-axis (see FIG. 1). Since gravitational acceleration is always applied in the Z-axis direction, the moving acceleration is a value exceeding 1 G except when the user is stationary or moving at a constant speed. Therefore, by setting the predetermined value to 1G, it is possible to reliably detect the movement of the user.

Also, the luminance (brightness) of the object is not limited to two, ie, the first luminance and the second luminance, and the second luminance may be gradually decreased as the acceleration increases. The aspect in which the second luminance is decreased is not particularly limited. For example, as illustrated in FIG. 14, the second luminance may be decreased linearly as the acceleration increases, or may be decreased stepwise.

In the example of FIG. 14, the luminance becomes zero at an acceleration of 2G. However, the present invention is not limited to this, and the luminance may be zero at an acceleration exceeding 2G. Alternatively, the display brightness of the object may be lowered according to the time change of the acceleration. For example, when the acceleration change rate is equal to or higher than a predetermined value, a technique such as temporarily reducing the display luminance can be applied.

Note that when the acceleration decreases, the display brightness of the object may be recovered so as to follow this. However, after the acceleration decreases, the display brightness may be recovered after a certain time has elapsed. Thereby, when the acceleration / deceleration is switched relatively quickly, the change in luminance of the object can be slowed down. In addition, a low-pass filter may be interposed in the output of the acceleration sensor in order to eliminate the influence of vibration during driving of the vehicle, road gradient, and the like.

The above dimming control may be executed as part of the function of the display control unit 315, or a dimming unit 306 provided separately in the control unit 30 may be used in combination as shown in FIG. As shown in the figure, the dimming unit 306 includes a manual dimming unit 41, an automatic dimming unit 42, a changeover switch 43, and a dimming correction unit 44.

The manual dimming unit 41 is a part where the user can arbitrarily set the luminance of the object, and is operated via the input operation unit 305 (see FIG. 5), for example. Based on the output of the illuminance sensor 21 built in the detection unit 20, the automatic light control unit 42 outputs a luminance value set in advance according to the output. The change-over switch 43 is for switching between manual adjustment and automatic adjustment. In the case of manual adjustment, the output of the manual dimming unit 41 is input to the dimming correction unit 44, and in the case of automatic adjustment, automatic dimming is performed. The output of the unit 42 is input to the dimming correction unit 44. Based on the output of the acceleration sensor 22 built in the detection unit 20, the dimming correction unit 44 performs correction to reduce the luminance value in the manner shown in FIG. 14, and outputs the correction to the CPU 301 (display control unit 315).

(Change of object position, size, etc.)
FIG. 16A is a schematic diagram of a visual field V showing an example of a change in the position and size of an object. FIG. 16B is a schematic diagram of the field of view V illustrating an example of a change in the position and information amount of the object B30. As shown in these figures, when the acceleration exceeds 1G, the object B30 is moved to one corner or the lower part of the field of view V, or only a part of the object is displayed, thereby ensuring a wide front field of view. be able to.

In addition to the control for changing the position, size, and information amount of the object B30 in accordance with the increase in acceleration, the display luminance of the object may be decreased at the same time as described above.

<Third Embodiment>
Subsequently, a second embodiment of the present technology will be described. Hereinafter, the configuration different from the first embodiment will be mainly described, and the description of the same configuration as the above-described embodiment will be omitted or simplified.

In the present embodiment, an example will be described in which the driving state and driving time of a vehicle are acquired as user information, and predetermined information is presented on the display unit 10 based on such information.

In the present embodiment, the control unit 30 is configured to be able to estimate the behavior of the vehicle from the acceleration information (user information) acquired by the detection unit 20. For example, as shown in FIG. 17, when the user is driving a motorcycle, the control unit 30 performs centrifugal force acting on the display unit 10 from acceleration information (for example, lateral acceleration) acquired by the detection unit 20. And the bank angle of the vehicle body at the time of turning or the like is estimated from the value. The control unit 30 is configured to present warning information for notifying the user of the risk of falling on the display unit 10 when the bank angle is equal to or greater than a predetermined value. The warning information may be character information, graphic information, or a combination thereof.

Further, the remaining amount of fuel in the vehicle may be detected, and information indicating that may be presented to the user when it becomes below a predetermined level. As this type of information, characters and figures may be used as described above, or information on the position of a nearby gas station may be displayed as an AR. This example can be useful information not only for general vehicle users but also for motorcycle users who are not equipped with fuel gauges.

As another example, as shown in FIG. 18, the control unit 30 determines the degree of fatigue of the user who drives the vehicle from travel time information (user information) acquired from the portable information terminal 200 (position information acquisition unit 207). It is configured to estimate and present information related to a break point such as a service area or a convenience store on the display unit 10. According to this example, as described in the first embodiment, for example, information related to a break point existing along a traveling road is presented. Thereby, information useful for the user can be provided at a timely timing.

As yet another example, as shown in FIG. 19, the control unit 30 estimates the user's moving destination from the moving direction acquired from the portable information terminal 200 (position information acquisition unit 207), and relates to the destination. Information (for example, station departure time and train operation information) is configured to be displayed on the display unit 10. Information regarding the weather etc. at the destination may be provided. Even in such a configuration, it is possible to provide useful information to the user at a timely timing.

In this example, the user's past action history may be referred to as the user information. The information is stored in the memory 302 of the control unit 30 or the memory 202 of the portable information terminal 200. Thereby, estimation accuracy of a user's action or its purpose can be improved. For example, for a user who is in the habit of jogging, the behavior can be estimated from the movement start time, the movement speed, and the like, and the destination can be estimated from the traveling route. As a result, information corresponding to the action or the destination can be provided to the user.

As mentioned above, although embodiment of this technique was described, this technique is not limited only to the above-mentioned embodiment, Of course, various changes can be added within the range which does not deviate from the summary of this technique.

For example, in the above embodiment, the example in which the present technology is applied to the HMD has been described. However, as an image display device other than the HMD, for example, a head-up display (HUD) mounted in a driver's seat of a vehicle or a cockpit of an aircraft or the like The present technology can also be applied to other image display apparatuses.

In the above embodiment, an example of application to a see-through type (transmission type) HMD has been described. However, the present technology can also be applied to a non-transmission type HMD. In this case, a predetermined object according to the present technology may be displayed in an external field of view captured by a camera attached to the display unit.

Further, in the above-described embodiment, the wearable display has been described by taking the HMD attached to the user's head as an example. However, the present invention is not limited to this, and for example, a display device that is used attached to the user's arm, wrist, or the like. The present technology can also be applied to a display device that is directly attached to the eyeball, such as a contact lens.

In addition, this technique can also take the following structures.
(1) A display unit configured to be wearable by a user and having a display area that provides a real-time visual field to the user;
The plurality of images based on user information including information related to the user's position and behavior, and a plurality of pieces of image information related to a plurality of specific objects existing in a peripheral region within a first distance from the user. A wearable display comprising: a control unit configured to extract image information related to a specific object existing in the moving direction of the user from the information, and present the extracted image information in the display area.
(2) The wearable display according to (1) above,
The user information includes map information around the user's location,
The said control unit is a wearable display which extracts the image information relevant to the specific target object which exists along the road where the said user travels from the said several image information.
(3) The wearable display according to (2) above,
The control unit acquires the moving speed of the user as the user information, and when the moving speed is equal to or less than a predetermined value, the specific target existing within a second distance shorter than the first distance from the user A wearable display for presenting image information related to an object in the display area.
(4) The wearable display according to (2) or (3) above,
The control unit is configured to be capable of presenting an image including the map information and the image information in the display area.
(5) The wearable display according to any one of (1) to (4) above,
The said control unit acquires the said user's movement acceleration as the said user information, The wearable display which changes the display mode of the said image information shown to the said display area according to the magnitude | size of the said movement acceleration.
(6) The wearable display according to (5) above,
The control unit presents the image information at the first luminance on the display area when the user's moving acceleration is less than or equal to a predetermined value, and when the user's moving acceleration exceeds the predetermined value, A wearable display that presents image information in the display area with a second luminance lower than the first luminance.
(7) The wearable display according to (6) above,
The control unit is a wearable display that gradually decreases the second luminance as the moving acceleration of the user increases.
(8) The wearable display according to any one of (5) to (7) above,
The control unit changes at least one of a position, a size, and an information amount of the image information presented in the display area according to the magnitude of the acceleration.
(9) The wearable display according to any one of (1) to (8) above,
A wearable display further comprising a detection unit capable of detecting an acceleration acting on the display unit as the user information.
(10) a display unit having a display area;
The plurality of pieces of image information based on user information including information relating to a user's position and behavior and a plurality of pieces of image information relating to a plurality of specific objects existing in a peripheral area within a first distance from the user. An image display device comprising: a control unit configured to extract image information related to a specific object existing in a moving direction of the user from the image and present the extracted image information in the display area.
(11) a display unit having a display area;
An acquisition unit that acquires user information including information related to a user's position and behavior, and a plurality of pieces of image information related to a plurality of specific objects existing in a peripheral region within a first distance from the user;
An image display system comprising: a control unit that extracts image information related to a specific object existing in the moving direction of the user from the plurality of image information, and presents the extracted image information in the display area.

DESCRIPTION OF SYMBOLS 10 ... Display part 20 ... Detection part 30 ... Control unit 100 ... Head mounted display (HMD)
DESCRIPTION OF SYMBOLS 110 ... Display area 200 ... Portable information terminal 314 ... Display control part A1-A4, A12-A22 ... Specific object B1-B4, B12-B14, B16, B19, B22 ... Object V ... Visual field

Claims

A display unit configured to be worn by the user and having a display area that provides the user with a field of view in real space;
The plurality of images based on user information including information related to the user's position and behavior, and a plurality of pieces of image information related to a plurality of specific objects existing in a peripheral region within a first distance from the user. A wearable display comprising: a control unit configured to extract image information related to a specific object existing in the moving direction of the user from the information, and present the extracted image information in the display area.
The wearable display according to claim 1,
The user information includes map information around the user's location,
The said control unit is a wearable display which extracts the image information relevant to the specific target object which exists along the road where the said user travels from the said several image information.
The wearable display according to claim 2,
The control unit acquires the moving speed of the user as the user information, and when the moving speed is equal to or less than a predetermined value, the specific target existing within a second distance shorter than the first distance from the user A wearable display for presenting image information related to an object in the display area.
The wearable display according to claim 2,
The control unit is configured to be capable of presenting an image including the map information and the image information in the display area.
The wearable display according to claim 1,
The said control unit acquires the said user's movement acceleration as the said user information, The wearable display which changes the display mode of the said image information shown to the said display area according to the magnitude | size of the said movement acceleration.
The wearable display according to claim 5,
The control unit presents the image information at the first luminance on the display area when the user's moving acceleration is less than or equal to a predetermined value, and when the user's moving acceleration exceeds the predetermined value, A wearable display that presents image information in the display area with a second luminance lower than the first luminance.
The wearable display according to claim 6,
The control unit is a wearable display that gradually decreases the second luminance as the moving acceleration of the user increases.
The wearable display according to claim 5,
The control unit changes at least one of a position, a size, and an information amount of the image information presented in the display area according to the magnitude of the acceleration.
The wearable display according to claim 1,
A wearable display further comprising a detection unit capable of detecting an acceleration acting on the display unit as the user information.
A display unit having a display area;
The plurality of pieces of image information based on user information including information relating to a user's position and behavior and a plurality of pieces of image information relating to a plurality of specific objects existing in a peripheral area within a first distance from the user. An image display device comprising: a control unit configured to extract image information related to a specific object existing in a moving direction of the user from the image and present the extracted image information in the display area.
A display unit having a display area;
An acquisition unit that acquires user information including information related to a user's position and behavior, and a plurality of pieces of image information related to a plurality of specific objects existing in a peripheral region within a first distance from the user;
An image display system comprising: a control unit that extracts image information related to a specific object existing in the moving direction of the user from the plurality of image information, and presents the extracted image information in the display area.