JP2019066564A - Display, display control method, and program - Google Patents

Abstract

To provide: a display capable of reducing troubles that a content in a superimposed image is visually recognized to shift when a user visually recognizing a superimposed image moves the line of sight from a distant place to a nearby place; a display control method; and a program.SOLUTION: The display comprises: display means for displaying a superimposed image visually recognized to overlap scenery by a user; display control means for controlling the display of the superimposed image; and gaze distance determining means for estimating a user's gaze distance to determine whether or not the estimated gaze distance is closer than a predetermined distance. The display control means hardly visually recognizes or eliminates contents C5 and C6 displayed in portions except a predetermined region of the superimposed image when the gaze distance determining means determines that the estimated gaze distance is short.SELECTED DRAWING: Figure 10

Description

  The present invention relates to a display device, a display control method, and a program.

  As a display device for displaying an image visually perceived to be superimposed on a landscape by the user, for example, there is one disclosed in Patent Document 1. This type of display device is also called a head mounted display (HMD: Head Mounted Display), and is a virtual image showing various information superimposed on a scene in front of a translucent member provided corresponding to the user's eye Hereinafter, AR (Augmented Reality) is realized by displaying a superimposed image).

JP, 2016-186561, A

  A display device such as an HMD controls a superimposed image of a set size to be displayed at a certain imaging position, but for the user, the user views it at his own gaze position (the distance to the object being gazed). There is a characteristic that the size of the superimposed image is felt to change accordingly. This characteristic results from the recognition of the size of the superimposed image as a comparison with the landscape that the user views with the superimposed image. Due to this characteristic, the user feels the superimposed image smaller as the gaze position is closer, and feels larger as the distance is farther. According to such a characteristic, when the user viewing the superimposed image shifts the gaze from a distance to a near position, the content (a predetermined image in the superimposed image) displayed overlapping with the specific object in the landscape at the time of the distance There is a problem that it is viewed off the specific target.

  The present invention has been made in view of the above-mentioned circumstances, and it is troublesome for the content in the superimposed image to be viewed in a shifted manner when the user viewing the superimposed image shifts the line of sight from the far side to the near side. It is an object of the present invention to provide a display device, display control method, and program capable of reducing

In order to achieve the above object, a display device according to a first aspect of the present invention is:
Display means provided corresponding to the user's eyes and displaying a superimposed image that is viewed by the user as being superimposed on the landscape;
Display control means for controlling display of the superimposed image;
Fixation distance determination means for estimating the fixation distance of the user and determining whether the estimated fixation distance is a short distance closer than a predetermined distance;
The display control means, when judging that the gaze distance judging means is at the short distance, makes the content displayed in the portion other than the predetermined area in the superimposed image visually difficult or erased.

In order to achieve the above object, a display control method according to a second aspect of the present invention is:
A display control method for displaying a superimposed image that is viewed by the user in an overlapping manner with a landscape on display means provided corresponding to the user's eyes,
Estimating the gaze distance of the user;
If the estimated gaze distance is a short distance closer than a predetermined distance, it is difficult to visually recognize or delete the content displayed in a portion other than the predetermined region in the superimposed image.

In order to achieve the above object, a program according to a third aspect of the present invention is
A program for displaying on a display means provided corresponding to a user's eyes a superimposed image that is viewed by the user in an overlapping manner with a landscape,
On the computer
A process of estimating the gaze distance of the user and determining whether the estimated gaze distance is a short distance closer than a predetermined distance;
Processing for making it difficult to visually recognize or delete the content displayed in the portion other than the predetermined area in the superimposed image in accordance with the determination that the short distance is determined;
Run

  According to the present invention, it is possible to reduce the inconvenience of the content in the superimposed image being misaligned and viewed when the user who views the superimposed image changes the line of sight from a distance to a near position.

It is a figure which shows the display apparatus based on 1st Embodiment of this invention, and is a figure containing the schematic perspective view of an image light production | generation apparatus. It is a block diagram of a display. It is a figure for demonstrating the path | route of a 1st display part and image light. It is a figure for demonstrating the size change characteristic on recognition of the superimposition image according to the gaze distance. It is a figure for demonstrating the problem which arises by the size change characteristic on recognition. (A) is a figure which shows the relationship between a gaze distance and a convergence angle etc., (b) is a figure for demonstrating the elevation angle calculated based on motion data. It is a figure which shows the superimposed image which concerns on 1st Embodiment, (a) shows the form for near, and (b) and (c) is a figure which shows the form for distance. (A) is a flowchart which shows an example of a display mode switching process, (b) is a flowchart which shows an example of a gaze distance estimation process. (A) And (b) is a figure which shows the form for distant use of the superimposed image which concerns on 2nd Embodiment. (A)-(c) is a figure for demonstrating the production | generation procedure of the near form of the superimposition image which concerns on 2nd Embodiment. (A)-(d) is a figure for demonstrating the gaze position induction | guidance | derivation image displayed in the superimposed image which concerns on 3rd Embodiment.

  A display device according to an embodiment of the present invention will be described with reference to the drawings.

First Embodiment
As shown in FIG. 1, the display device 100 according to the first embodiment includes an image light generation device 1 and a control device 2 that controls the operation of the image light generation device 1. The display device 100 is configured as a head mounted display (HMD: Head Mounted Display) that displays a superimposed image that is viewed by the user as being superimposed on a landscape as a virtual image V (see FIG. 3). The display device 100 of this embodiment is used in the vehicle interior 3 shown in FIG. 5 when the user drives the vehicle.

  As shown in FIGS. 1 to 3, the image light generation device 1 includes a first display unit 10, a second display unit 20, a frame 30, and an inner camera 31 and a motion sensor 32 disposed on the frame 30. And an outside camera 33.

  The image light generating device 1 is formed in the shape of glasses as shown in FIG. 1, and is mounted on the head of the user via the frame 30 and used. The frame 30 is roughly divided into a rim 30a for holding the first display unit 10 and the second display unit 20, and a temple 30b applied to the side of the user and the upper side of the ear.

  The first display unit 10 and the second display unit 20 are provided corresponding to the eye E of the user, as shown in FIG. Specifically, the first display unit 10 is provided corresponding to the right eye E1 of the user, and the second display unit 20 is provided corresponding to the left eye E2 of the user. The first display unit 10 displays a virtual image V1 visually recognized by the right eye E1 by generating and emitting the image light L1. Further, the second display unit 20 generates a virtual image V2 to be viewed by the left eye E2 by generating and emitting the image light L2. In addition, since the virtual image V1 and the virtual image V2 basically indicate the same image, hereinafter, unless otherwise specified, both are also referred to as a virtual image V (or a superimposed image) without distinction. Similarly, the image light L1 and the image light L2 are also referred to as the image light L without distinction.

  The image light generation device 1 allows the user to see the scenery (real scenery) through the first display unit 10 and the second display unit 20 and to visually recognize the scenery (real scenery), and to observe the user from the first display unit 10 and the second display unit 20. The virtual image V is displayed in front of.

  The first display unit 10 includes a display element 11 and a light guide device 12. In addition, the second display unit 20 includes a display element 21 and a light guide device 22. The image light generating device 1 is formed to be substantially symmetrical in the left-right direction, and the first display unit 10 and the second display unit 20 are configured in the same manner. Therefore, in the following, the first display unit 10 will be mainly described, with a specific description of the configuration of the second display unit 20 omitted.

  The display element 11 includes, for example, a full color TFT (Thin Film Transistor) type liquid crystal display panel or a backlight, and displays an image by a combination of a plurality of lighting pixels. That is, the display element 11 emits the image light L (L1) representing the image. The backlight illuminates the liquid crystal display panel from behind, and is configured of, for example, a light emitting diode (LED) or a light guide. The display element 11 is driven by the control of the control device 2 via the drive unit 11a shown in FIG. The driving unit 11a includes, for example, a liquid crystal driver that switches the transmission / non-transmission state of each of the plurality of pixels of the liquid crystal display panel according to an image signal, a light source driver that drives an LED, and the like.

  The display device 11 may be any device capable of emitting the image light L, and a reflective type such as an organic EL (Electro-Luminescence) panel, a DMD (Digital Micromirror Device), or an LCOS (Liquid Crystal On Silicon). The projector may be configured of a display device.

  As illustrated in FIG. 3, the light guide device 12 includes a barrel unit 120, a light guide unit 121, and a transmission unit 122. The barrel unit 120 accommodates an optical element (for example, composed of a plurality of lenses) (not shown) that transmits the image light L emitted by the display element 11. The optical element located in the lens barrel 120 cooperates with the light guide 121 to form an intermediate image of the superimposed image (virtual image V) inside the light guide 121. The barrel unit 120 is provided, for example, in the vicinity of a connection portion between the rim 30 a and the temple 30 b in the frame 30.

  The light guiding portion 121 and the transmitting portion 122 are formed of a translucent resin, and as shown in FIG. 2, when the user wears the image light generating device 1, they are positioned in front of the right eye E1 of the user and It has a shape corresponding to a lens for the right eye.

  The light guiding portion 121 receives the image light L transmitted through the optical element in the lens barrel portion 120 and guides the light L inside. The light guiding portion 121 is formed with a reflecting surface 121 a that reflects the image light L along the optical axis of the optical element toward the front of the eye E of the user. The image light L reflected by the reflective surface 121 a is guided while being reflected a plurality of times in the light guide portion 121. The light guide portion 121 has such a light guide function and a see-through function for allowing the user to see through the scenery. The transmitting unit 122 is joined to the light guiding unit 121 and assists the fluoroscopic function of the light guiding unit 121. A half mirror layer 123 is formed between the light guide portion 121 and the transmission portion 122. By adjusting the transmittance of the half mirror layer 123, the display appearance of the virtual image V with respect to the scene to be seen through is adjusted.

Here, the optical path of the image light L will be briefly described with reference to FIG. The image light L represented by a dotted line in FIG. 2 is schematically shown.
In the first display unit 10, the image light L (L1) emitted from the display element 11 toward the light guiding device 12 is converged by passing through the optical element in the lens barrel 120, and is transmitted to the light guiding unit 121. It will be incident. The image light L incident on the light guiding portion 121 is reflected by the reflecting surface 121 a, and is further reflected a plurality of times before entering the half mirror layer 123. In addition, an intermediate image is formed in the light guide portion 121 in the process of a plurality of reflections. A part of the image light L incident on the half mirror layer 123 is reflected by the half mirror layer 123 and then emitted from the light guide 121 to reach the right eye E1 of the user. Thereby, the user visually recognizes a virtual image V (V1) based on the image light L in front of the light guide 121 and the transmission unit 122. That is, the user visually recognizes the image displayed by the display element 11 as the virtual image V. Similarly, in the second display unit 20, the image light L (L2) emitted by the display element 21 toward the light guiding device 22 reaches the user's left eye E2 and is visually recognized as a virtual image V (V2). Be done.

  Returning to FIG. 2, the inner camera 31 is configured to acquire information on the eyeballs of the user, and images the face (mainly the eye E) of the user under the control of the control device 2. The inner camera 31 is provided, for example, on the inner side of the rim 30a (a position facing the user's face). The inner camera 31 is provided corresponding to each of the right eye E1 and the left eye E2. In addition, in order to image one eye from a plurality of directions, it is preferable that a plurality of inner cameras 31 be provided for the one eye. For example, the inner camera 31 can be configured by four cameras in total: two cameras that capture the right eye E1 from different directions and two cameras that capture the left eye E2 from different directions. The inner camera 31 supplies data of a captured image including the captured right eye E1 and the left eye E2 (hereinafter, referred to as inner captured data D1) to the control device 2.

  The motion sensor 32 detects the movement and orientation of the image light generation device 1, that is, the movement and orientation of the face of the user wearing the image light generation device 1. The motion sensor 32 includes, for example, one or more of an acceleration sensor, a gyro sensor, and a geomagnetic sensor, and supplies data (hereinafter referred to as motion data D2) indicating a detection result to the control device 2.

  The outside camera 33 is a configuration for acquiring information related to a landscape visually recognized by the user, and controls the control device 2 to capture an image of the scenery seen through the light guiding devices 12 and 22 by the user. The outer camera 33 is configured, for example, as a stereo camera, and as shown in FIG. 1, is provided at each of left and right end portions on the outer side of the rim 30a. The outer camera 33 supplies data of the captured image (hereinafter, referred to as outer captured data) to the control device 2.

  The control device 2 is configured, for example, as a controller that allows the user to perform various operations of the image light generation device 1, and communicates with the image light generation device 1 wirelessly or by wire via an interface (I / F) not shown. . As wireless communication, a method based on standards such as Bluetooth (registered trademark), UWB (Ultra Wide Band), Z-Wave, and ZigBee can be used. Further, as wired communication, a known method such as USB (Universal Serial Bus) can be used. The control device 2 may be incorporated in the image light generating device 1.

  The control device 2 includes a control unit 40 and a communication unit 50. In addition, the control device 2 includes a power supply that supplies operating power, an operation unit that receives an operation of a user, an interface for communicating with the image light generation device 1, and the like as a configuration not shown.

  The communication unit 50 communicates with an external device of the display device 100 in a wired or wireless manner, and includes, for example, a wired interface, a wireless communication module having an antenna, a high frequency circuit, and the like. The available wired or wireless communication schemes are the same as described above.

  The communication unit 50 communicates with the in-vehicle device 200 mounted on a vehicle on which the user boarding, and receives various information (hereinafter referred to as vehicle information) regarding the vehicle. Vehicle information includes not only information on the vehicle itself but also information on the outside of the vehicle. The virtual image V in this embodiment is displayed mainly to notify the user of the vehicle information. The in-vehicle device 200 is configured of, for example, an electronic control unit (ECU) that controls each part of the vehicle, a car navigation device, a complex instrument, and the like. The communication unit 50 receives, as vehicle information, vehicle speed data D3 and navigation data regarding the current vehicle position and the determined guide route from the in-vehicle apparatus 200. Further, the communication unit 50 may receive at least a part of the image data to be displayed as the virtual image V from the in-vehicle apparatus 200. The communication unit 50 may communicate with the mobile terminal 300 and receive content data (image data, audio data, etc.) by the mobile terminal 300. For example, the content data may include navigation data from a portable terminal 300 having a navigation function.

  The control unit 40 includes a central processing unit (CPU) 41, a storage unit 42, and a display control unit (graphic controller) not shown. Note that a part of the control unit 40 may be configured by a dedicated circuit such as an application specific integrated circuit (ASIC).

  The storage unit 42 includes a ROM (Read Only Memory) and a RAM (Random Access Memory) as a control memory, and a non-volatile memory for storing image data as a video memory. The ROM stores in advance an operation program (a program PG or the like for executing a display mode switching process described later) and fixed data. The RAM temporarily stores various calculation results and the like. For example, the RAM stores data indicating the determination result or the determination result in the display mode switching process described later.

  The CPU 41 executes processing for controlling the overall operation of the display device 100 using the operation program and fixed data read from the ROM of the storage unit 42. At this time, a fixed data read operation in which the CPU 41 reads fixed data from the ROM, a data write operation in which the CPU 41 writes various data in the RAM of the storage unit 42 and temporarily stores them, and the various data temporarily stored in the RAM by the CPU 41 A data read operation for reading out, a transmission / reception operation for the CPU 41 to communicate with the image light generating device 1 and an external device such as the on-vehicle device 200 are performed.

  In addition, the CPU 41 controls display (image light) of the image light generation device 1 (first display unit 10, second display unit 20) based on image (video) data stored in the video memory via the display control unit. Control the generation of L). That is, the CPU 41 performs display control of the superimposed image visually recognized as the virtual image V through the display control unit. Further, based on the outside imaging data acquired from the outside camera 33, the CPU 41 calculates depth information including the distance to the object OB (see FIG. 6) existing in the landscape. The CPU 41 controls the position and the size of the content image (image for performing various types of notification) to be superimposed and displayed on the object OB according to the calculated depth information. The content image is an image for performing various notifications, and includes a vehicle speed image C1, a route guidance image C2, and a time image C3 described later. The content image is an image included in the superimposed image (virtual image V) which is a displayable area. Hereinafter, the content image superimposed and displayed on the predetermined object OB will be referred to as “AR (Augmented Reality) display”, and the predetermined object OB will also be referred to as “AR object”.

  Here, as described in the above task, the display device 100 configured as the HMD has a characteristic that the user feels that the size of the superimposed image has changed according to the gaze position (gaze distance G) of the user. There is. With this characteristic, the user recognizes that the size of the superimposed image is approximately doubled, for example, as shown in FIG. Thus, in recognition of the user, the size of the superimposed image changes in proportion to the gaze distance G.

  Due to the above-mentioned characteristics, as shown in FIG. 5, when the user viewing the superimposed image (virtual image V) while driving the vehicle moves the sight line from the near position P1 to the far position P2, It seems that the predetermined content image C (image showing “40 km / h” in the example of FIG. 5) included in the image suddenly becomes large, obstructing the visual field or making the user feel complicated. In some cases, it may not be possible to make an appropriate display accordingly. In FIG. 5, in order to facilitate understanding of the superimposed image superimposed on the landscape, the traveling lane 5 is depicted as a landscape visually recognized through the windshield 4 of the vehicle. The object OB to be gazed by the user may be in the vicinity of the dashboard 6 of the vehicle, the driving lane 5, or the leading vehicle 7 in the driving lane 5, depending on the gaze position.

  In this embodiment, the display form of the superimposed image is switched according to whether the gaze position of the user is far or near, in order to appropriately display according to the situation. First, a method of determining whether the user's gaze distance G is far or near will be described. As an example, the gaze distance G is determined as in the following (i) to (iii).

(I) The CPU 41 estimates a gaze distance G, which is the distance from the user's eye E to the object OB being gazed, based on the inside imaging data D1 acquired from the inner camera 31. For example, the CPU 41 executes known image processing on the images of the right eye E1 and the left eye E2 included in the inside imaging data D1, and shown in FIG. 6A, the right eye E1 and the left of the user. The interpupillary distance W of the eye E2 and the convergence angle α are calculated. Then, using the trigonometric function, the gaze distance G is calculated (estimated) from the calculated inter-pupil distance W and the convergence angle α. In addition, the method of calculating the gaze distance G is arbitrary, and a well-known method can be used suitably. For example, an eye potential sensor may be provided in the frame 30, and the gaze distance G may be calculated using at least one of the measured eye potential and the inner imaging data D1. Further, the estimated gaze distance G may be acquired by referring to table data in which the estimated gaze distance G and the convergence angle α are associated with each other without using the equation. Formulas and table data used to estimate the gaze distance G may be stored in the ROM of the storage unit 42 in advance. The CPU 41 determines that the user is looking at a distance when the gaze distance G calculated in this manner is equal to or greater than the set distance stored in advance in the ROM, and the user is closer when the gaze distance G is less than the set distance. Determine that you are looking at

(Ii) Further, based on the motion data D2 acquired from the motion sensor 32, the CPU 41 calculates information on the face orientation of the user by a known method. In this embodiment, as an example, based on the motion data D2, as shown in FIG. 6B, the elevation angle θ is calculated based on the case where the user looks in the horizontal direction. In FIG. 6B, the axis along the horizontal direction is taken as the X axis, and the axis along the vertical direction is taken as the Y axis. The CPU 41 determines that the user is looking at a distance when the elevation angle θ calculated in this manner is equal to or greater than the set angle β stored in advance in the ROM, and the user is closer if the angle is less than the set angle β. Determine that you are looking at

(Iii) In addition, if the user is looking at a distance when the vehicle speed indicated by the vehicle speed data D3 acquired from the in-vehicle apparatus 200 via the communication unit 50 is equal to or higher than the set speed stored in advance in the ROM. If it is determined that the speed is less than the set speed, it is determined that the user is looking close. The determination method utilizes the property that the driver looks farther as the vehicle speed is higher.

  In the first embodiment, as described above, it is determined whether the gaze distance G is far or near, and when it is determined that the gaze distance G is far, the display form of the superimposed image is switched from near use to far use.

  In the near mode, when the object OB to be viewed with the virtual image V (superimposed image) is relatively close (for example, when the object OB is the dashboard 6 or the traveling lane 5 seen from the vicinity of the dashboard 6) It is a suitable display form. For example, in the near mode, as shown in FIG. 7A, the vehicle speed image C1 is at the lower center in the display area of the virtual image V, the route guidance image C2 is at the lower right, and the time image C3 is at the upper right. Display various content images.

  The distant form is a display form suitable for the case where the object OB to be viewed together with the virtual image V is relatively far (for example, when the object OB is the traveling lane 5 or the leading vehicle 7). For example, in the far-end mode, as shown in FIG. 7B, the vehicle speed image C1 in the near-end mode is displayed in a reduced size and moved upward to the left in the display area of the virtual image V. Further, the route guidance image C2 in the near mode is moved to the lower central portion in the display area of the virtual image V. Further, the time image C3 displayed in the near mode is deleted. In the far-end mode, the display image may be lowered or the saturation may be lowered compared to the near-end mode without completely erasing the time image C3 to make visibility difficult. With such a far-field configuration, it is possible to realize a display suitable for the visual field during driving.

  In the far-end mode, far-end contents not displayed in the near-end mode may be displayed, or contents displayed in the near-end mode may be displayed in a design different from that for the near-end mode. For example, as shown in FIG. 7C, in the far-field type, the destination image C4 indicating the destination set by the car navigation device is displayed, or the vehicle speed image C1 is displayed in a design different from that for near. You may

  The configuration of the display device 100 and the superimposed image have been described above. Subsequently, an example of the display mode switching process executed by the control unit 40 (CPU 41) of the control device 2 will be described with reference to FIGS. 8 (a) and 8 (b). The display mode switching process is performed in a predetermined control cycle while the display device 100 is in operation. At the start of the display mode switching process, the distance flag described later is in the OFF state, and the superimposed image (virtual image V) in the near mode is displayed.

  When the display mode switching process shown in FIG. 8A is started, first, the control unit 40 acquires the inside imaging data D1 from the inner camera 31, acquires the movement data D2 from the motion sensor 32, and the vehicle speed from the in-vehicle device 200. Data D3 is acquired (step S1).

  Subsequently, the control unit 40 executes gaze distance estimation processing (step S2), and shifts to the processing of step S21 shown in FIG. 8 (b). In step S21, the gaze distance G is calculated based on the acquired inside imaging data D1, and it is determined whether the calculated gaze distance G is equal to or greater than the set distance stored in advance in the ROM. If the calculated gaze distance G is equal to or greater than the set distance (step S21; Yes), the control unit 40 sets a far flag indicating that the estimated gaze distance G is far (step S24). The flag for distance is stored in a flag storage area in the RAM.

  If the gaze distance G calculated in step S21 is less than the set distance (step S21; No), the control unit 40 calculates and calculates the elevation angle θ indicating the face orientation angle of the user based on the acquired motion data D2 It is determined whether or not the elevation angle θ is equal to or greater than the set angle β stored in advance in the ROM (step S22). If the calculated elevation angle θ is equal to or greater than the set angle β (step S22; Yes), the control unit 40 sets the distance flag to the on state (step S24).

If the elevation angle θ calculated in step S22 is less than the set angle β (step S22; No), the control unit 40 determines whether the vehicle speed indicated by the acquired vehicle speed data D3 is equal to or greater than the set speed stored in advance in the ROM. It discriminates (step S23). If the vehicle speed is equal to or higher than the set speed (step S23; Yes), the control unit 40 sets the distance flag to the on state (step S24). That is, the flag for distance is also in the on state when any of the steps S21, S22, and S23 is Yes, which indicates that the estimated gaze distance G is far from the predetermined distance.
On the other hand, when the vehicle speed is less than the set speed (step S23; No), the control unit 40 clears the distance flag to the off state (step S25).

  When the gaze distance estimation process described above is executed (step S2), the control unit 40 determines whether the gaze distance G to be estimated is far (step S3). The control unit 40 determines that the estimated gaze distance G is far when the distance flag is in the on state (step S3; Yes), and the estimated gaze distance when the distance flag is in the off state. It is determined that G is close (step S3; No).

  If the gaze distance G is far (step S3; Yes), the control unit 40 controls the display of the image light generation device 1 and displays the superimposed image in the far form (step S4). For example, when displayed in the near form shown in FIG. 7 (a), the control unit 40 displays the superimposed image from the near use form as shown in FIG. 7 (b) or FIG. 7 (c). Perform display control to switch to the application form.

  On the other hand, when the gaze distance G is near (step S3; No), the control unit 40 controls the display of the image light generation device 1 and displays the superimposed image in the near form (step S5). For example, when it is displayed in the near mode shown in FIG. 7A, the display in the near mode is continued. When the mode is switched to the distant mode shown in FIG. 7 (b) or FIG. 7 (c), the control unit 40 changes the superimposed image from the distant mode to the near mode shown in FIG. 7 (a). Perform display control to switch back to the form (switch). For example, the control unit 40 continuously executes the display mode switching process described above until receiving a stop operation by the user. The description of the first embodiment is as described above.

  From here, other embodiments in which the near and far forms of the superimposed image are different from the first embodiment will be described. Note that the following embodiments are the same as the first embodiment and the configuration of the display apparatus 100, and mainly differ in image processing. Therefore, in the following, each configuration similar to that of the first embodiment will be described using the same reference numeral as that of the first embodiment, and points different from the first embodiment will be mainly described.

Second Embodiment
In the second embodiment, a superimposed image VF (hereinafter, also referred to as a distant image VF) as shown in FIG. 9A is displayed at the start of the display mode switching process as the distant mode. The distant image VF shown in FIG. 9A displays a route guidance image C5 and a warning image C6 in the display area. In the figure, the arrow display as the route guidance image C5 is superimposed on the lane on the left side of the branch road, and an example in which a left turn is urged is shown. Further, the warning image C6 is, for example, for emphasizing the presence of the leading vehicle 7, and is displayed at a position superimposed on the leading vehicle 7 viewed as a real view.

  When the user changes the gaze position from a distance to a near position suitable for the distance image VF, it is recognized as an image of a size smaller than that at the time of the distance due to the characteristics described with reference to FIG. Then, as shown in FIG. 9B, the route guidance image C5 and the warning image C6 may be deviated from an object to be actually superimposed, which hinders driving. That is, when the user changes the gaze position from a distance to a near position, a shift of the AR display with respect to the AR target (hereinafter, referred to as “AR shift”) may occur.

  In order to solve such a problem, in the second embodiment, when it is determined that the image is near, the display mode of the superimposed image is generated from the image VF for distance and the image VF for distance is generated. The image is switched to the superimposed image VN for the user (hereinafter, also referred to as a near image VN). The procedure for creating the near direction image VN will be described with reference to FIGS. 10 (a) to 10 (c).

  When creating the near direction image VN, the control unit 40 first calculates the recognized image size Sn in the near view. For example, the control unit 40 calculates the gaze distance Gn at the time of the near vision of the user based on the inside imaging data D1 acquired from the inner camera 31 as described above. As described later, the gaze distance Gn at the time of near vision can be acquired as the gaze distance when it is determined No in step S21. The control unit 40 previously stores data indicating the set image size S0 of the image VF for distance and the reference gaze distance G0 in the ROM, and based on the data, for example, Sn = (Gn / G0) · S0 The recognition image size Sn at the near time is calculated by the following equation. In FIG. 10A, the size of the image indicated by the dotted line corresponds to Sn. The control unit 40 does not have to calculate Sn at this time as needed. For example, Sn may be stored in advance in the ROM as an assumed image size, and the control unit 40 may obtain Sn from the ROM.

  Subsequently, based on the calculated or acquired Sn, as shown in FIG. 10 (b), the control unit 40 displays the distant image VF currently displayed and the image of the recognition image size Sn at the time of near vision. The difference area VD is extracted.

  Subsequently, as shown in FIG. 10C, the control unit 40 deletes the difference area VD from the image for distance VF, generates the image for near direction VN, and causes the image light generation device 1 to display the image for near. In FIG. 10 (c), the deleted part is represented by a dotted line. In this way, when the gaze position shifts from a distance to a near position, the image (route guidance image C5 and the warning image C6) in which the AR shift has occurred becomes unviewable, and thus the display becomes an obstacle to driving. Can be suppressed. When the user's gaze position returns to the distant place by the display mode switching processing, the display switches to the distant image VF, so that the AR display such as the route guidance image C5 or the warning image C6 can be appropriately viewed again.

  Further, as shown in FIG. 10C, in the near image VN, the control unit 40 causes the user to move in the direction of the content (the route guidance image C5 and the warning image C6) displayed in the difference area VD. A guidance image C7 may be displayed in a manner of guiding the gaze of the subject. This makes it possible to notify the user viewing the near image VN that the AR display is present if looking further from the distance, which is user friendly.

  Subsequently, an example of the display mode switching process according to the second embodiment will be described with reference to FIGS. 8 (a) and 8 (b). In the second embodiment, when the display mode switching process is started, the distance flag is in the on state, and the superimposed image (image for distance VF) in the mode for distance is displayed.

  When the display mode switching process shown in FIG. 8A is started, the control unit 40 acquires various data (step S1) as in the first embodiment, and then the gaze distance estimation process (step S2) is performed. It executes according to the processing procedure shown in FIG.

  Subsequently, the control unit 40 refers to the flag for distance and determines whether the gaze distance G estimated is distant (step S3). If the gaze distance G to be estimated is not far (step S3; No), that is, if it is near, the superimposed image is displayed in the near form (step S5). Here, for example, immediately before clearing the distance flag to the off state (step S25), the control unit 40 sets the gaze distance calculated in step S21 as the gaze distance Gn at the time of near vision, as described above. The size of the recognized image Sn is calculated. Then, the control unit 40 extracts the difference area VD shown in FIG. 10B based on the calculated recognition image size Sn, deletes the difference area VD from the image for distance VF, and displays the difference area VD as shown in FIG. The near image VN is generated and displayed on the image light generating device 1. Further, in the display control of the near image VN, the control unit 40 guides the user's gaze in the direction of the content (the route guidance image C5 and the warning image C6) displayed in the difference area VD. C7 is displayed in the near image VN. Thereby, for example, when the image for distance VF shown in FIG. 10A is displayed as the superimposed image, the image is switched to the image for near VN shown in FIG.

  On the other hand, when the gaze distance G to be estimated is far (step S3; Yes), the superimposed image is displayed in the form for distance (step S4). For example, when the image for distance VF shown in FIG. 10A is displayed, the display is continued as it is. When the near image VN shown in FIG. 10C is displayed, the control unit 40 returns the near image VN to the far image VF shown in FIG. Perform display control. The description of the second embodiment is as described above.

Third Embodiment
In the third embodiment, the AR shift that occurs when the user changes the gaze position from near to far is resolved by using a near image that is different from the second embodiment. In the third embodiment, a far-field image VF shown in FIG. 9A similar to that of the second embodiment is displayed at the start of the display mode switching process as the far-field mode. Then, when it is determined that the predetermined proximity is determined, the display mode of the superimposed image is switched from the image VF for distance to the image VM for near direction VM in which the gaze position induced image IN is additionally displayed on the image VF for distance.

  The gaze position induced image IN is displayed in such a manner as to guide to a set gaze position set according to the distance image VF. Specifically, as shown in FIGS. 11A and 11B, the gaze position induction image IN includes the image M1 for the right eye displayed in the virtual image V1 for the right eye and the virtual image V2 for the left eye. The parallax image is configured of the left-eye image M2 displayed inside, and the gaze position of the user is guided to the set gaze position by the parallax between the right-eye image M1 and the left-eye image M2. The set gaze position is, for example, the reference gaze distance G0 of the distance image VF stored in advance in the ROM as described above, and is determined as a position for performing AR display more appropriately by the distance image VF.

  The control unit 40 generates a parallax between the right-eye image M1 and the left-eye image M2 so that the gaze-position-guided image IN is displayed at the set gaze position at a distance when viewed from the user. Display control of the image M1 for left eye and the image M2 for left eye is performed. For example, the control unit 40 calculates the inter-pupil distance W of the user's right eye E1 and left eye E2 based on the inside imaging data D1, and uses a trigonometric function from the calculated inter-pupil distance W and the reference gaze distance G0. Then, the convergence angle α that defines the set gaze position is calculated. Then, based on the calculated convergence angle α, the control unit 40 generates parallax between the right-eye image M1 and the left-eye image M2 so that the gaze position induced image IN is displayed at the set gaze position. In addition, the display aspect and the production | generation method of a parallax image are arbitrary, and a well-known image processing method can be used suitably.

  Also, the gaze position induction image IN is not limited to the parallax images as shown in FIGS. 11A and 11B, and the gaze position of the user may be guided to the set gaze position by the moving image mode. For example, the gaze position induction image IN may be an icon image that blinks and displays as shown in FIG. 11C, or an icon image that moves and displays as shown in FIG. In the case of blinking display, for example, it is possible to change the blinking interval according to the gaze distance G of the user with respect to the set gaze position (reference gaze distance G0). The gaze distance G of the user can be calculated based on the inside imaging data D1 as described above. In addition, in the case of moving display, the icon may be moved so that the user raises the line of sight. In this case, the movement amount of the icon can also be changed according to the gaze distance G and the elevation angle θ that can be calculated as described above.

  As described above, by displaying the gaze position guidance image IN in the near image VM, it is possible to guide the user's gaze position to an appropriate position and restore it. Subsequently, an example of the display mode switching process according to the third embodiment will be described with reference to FIGS. 8 (a) and 8 (b). In the third embodiment, when the display mode switching process is started, the distance flag is in the on state, and the distance image VF is displayed.

  When the display mode switching process shown in FIG. 8A is started, the control unit 40 executes the processes from step S1 to step S3 as in the first and second embodiments. If the gaze distance G to be estimated is not far (step S3; No), that is, if it is near, the superimposed image is displayed in the near form (step S5). Thereby, for example, when the image for distance VF shown in FIG. 10A is displayed as a superimposed image, a gaze position induced image IN is displayed as shown in FIGS. 11A and 11B. The image is switched to the near side image VM. The near image VM may be in the form shown in FIG. 11 (c) or 11 (d).

  On the other hand, when the gaze distance G to be estimated is far (step S3; Yes), the superimposed image is displayed in the form for distance (step S4). For example, when the image for distance VF shown in FIG. 10A is displayed, the display is continued as it is. When the near image VM shown in FIGS. 11A and 11B is displayed, the control unit 40 displays the near image VM as shown in FIG. 10A. Perform display control to switch back to VF. The description of the third embodiment is as described above.

  The present invention is not limited by the above embodiment and the drawings. Changes (including deletion of components) can be made as appropriate without changing the scope of the present invention.

(Modification)
Although the display apparatus 100 is comprised by HMD above and the example which mounts the image light generating apparatus 1 on a user's head was shown above, it is not restricted to this. The image light generating device 1 may not be worn by the user as long as it can follow the movement of the head of the user. For example, the image light generating device 1 may be suspended from the ceiling in the car, driving It may be fixed to the seat. In addition, the display device 100 may be used when boarding a vehicle other than a vehicle (aircraft, ship, etc.), or may be used when the user walks, not when boarding the vehicle. Good.

  In the above, an example in which the image light generation device 1 includes the first display unit 10 and the second display unit 20 corresponding to each of the right eye E1 and the left eye E2 has been described, but the parallax image described in the third embodiment The configuration may be such that only one of the first display unit 10 and the second display unit 20 is provided except in the case where the gaze position induction image IN (see FIGS. 11A and 11B) is displayed.

  In the fixation distance estimation process shown in FIG. 8B, the distance flag is turned on if the elevation angle θ indicating the face direction angle or the vehicle speed is equal to or greater than the threshold value, that is, the fixation distance G is indirectly Although an example of estimation is shown, table data in which the elevation angle θ and the vehicle speed are associated with the estimated gaze distance G is stored in advance in the ROM, and the control unit 40 refers to the table data to determine the elevation angle θ. An estimated gaze distance G may be obtained in accordance with the vehicle speed. Further, information on the face orientation of the user (such as elevation angle θ) may be obtained based on the inside imaging data D1. The said information can be calculated | required by performing well-known image processing, such as pattern matching, with respect to the captured image which inner side imaging data D1 shows, for example. Moreover, the camera and sensor used for estimation and calculation of the gaze distance G are not limited to those provided in the image light generating device 1. For example, the control unit 40 may acquire information from a camera or an infrared sensor provided in a car on which the user boarded, and estimate the gaze distance G based on the information. Further, the configuration for detecting the front object is not limited to the outside camera 33 provided in the image light generating device 1. For example, the control unit 40 may detect information from a camera provided in a vehicle on which the user boarded, information from front object detection means such as a sonar, ultrasonic sensor, millimeter wave radar, etc., and detect the front object. .

  In the above, in the fixation distance estimation process, when the determination in any of steps S21, S22, and S23 is Yes, it is estimated that the fixation distance G is far and all of steps S21, S22, and S23 become No. Although the example which presumes that gaze distance G is near was shown to, the estimation method of gaze distance G is not limited to this, but it is arbitrary. For example, the gaze distance G may be estimated to be far if a plurality of steps S21, S22, and S23 result in Yes. Further, for example, a plurality of gaze distances G estimated from the inner imaging data D1, the motion data D2, the vehicle speed data D3 and the like are calculated, and weighting is taken into consideration to the calculated plurality of gaze distances. By performing weighted averaging according to the estimation accuracy of {circle around (1)}, the gaze distance G to be determined as distant or near may be calculated.

In the first embodiment, the superimposed image of the near mode is displayed as the default in the display mode switching process, but the superimposed image of the far mode may be displayed as the default.
In the second embodiment, the difference area VD is deleted from the distance image VF to generate the near image VN. However, the present invention is not limited to this. For example, a content image (a route guidance image C5 or a warning image C6) existing in the extracted difference area VD may be specified, and a mode in which the specified content image is not displayed may be set as the near image VN. In addition, an aspect in which it is difficult to visually recognize the specified content image (for example, the display luminance is reduced or the saturation is reduced as compared to the case of the distant mode) as compared to the distant mode. As the near image VN.
In the third embodiment, the gaze position induced image IN is displayed when the gaze distance G is estimated to be near, but the present invention is not limited to this. For example, the gaze position induced image IN may be constantly displayed in the superimposed image.

  Further, the image control of the first to third embodiments may be combined and executed. For example, the gaze position guidance image IN described in the third embodiment may be displayed in the superimposed image in the first or second embodiment. In addition, setting of a threshold serving as a determination reference of whether the fixation distance G is far or near is arbitrary. The threshold may be different in each of the first to third embodiments. In addition, when switching the superimposed image from one to the other of the near mode and the far mode, the image immediately before the switching is faded out and the image after the switching is faded in order to reduce the discomfort of switching suddenly. You may give the effect of. Also, by setting a plurality of threshold values to be the discrimination reference of the gaze distance G in addition to the two steps of "near" and "far" the gaze distance G to be estimated, "near", far from near "standard time It may be divided into a plurality of three or more stages such as "far" from standard time. In such a case, the switching of the superimposed image may be performed in the same process as in the first to third embodiments in a predetermined stage and a stage farther than the predetermined stage among the plurality of stages. For example, when the gaze distance G to be estimated is divided into three stages of “near”, “standard time”, and “far”, the image shown in FIG. 7A according to the first embodiment in the case of “near”. Is displayed, and the image shown in FIG. 7 (b) or (c) according to the first embodiment is displayed when "far", and FIG. 7 (b) according to the first embodiment or during "standard time" A combination of displaying the image obtained by deleting the difference area VD described in the second embodiment from the image shown in (c) is also possible.

  Further, the type and form of the content image as AR display are not limited to the route guidance images C2 and C5, the destination image C4 and the warning image C6 exemplified above, and are arbitrary. Further, the type of the object OB as an AR target is also arbitrary. In addition, image data for displaying a superimposed image may be prepared in advance in storage unit 42, or may be obtainable by communication from an external device such as in-vehicle device 200 or portable terminal 300. .

  Although the operation program (program PG) for executing the display mode switching process is stored in advance in the ROM of the storage unit 72, it may be distributed and provided by a removable recording medium. The program PG may be downloaded from another device connected to the display device 100. In addition, the display device 100 may execute each processing according to the program PG by exchanging various data with other devices via a telecommunication network or the like.

(1) The display device 100 according to the first embodiment described above is provided corresponding to the eye E of the user, and is an image that displays a superimposed image (virtual image V) that is viewed by the user in an overlapping manner with the landscape. A light generation device 1 (an example of a display unit) and a control unit 40 are provided. The control unit 40 estimates the gaze distance G of the user by the display control means for performing display control of the superimposed image, and the gaze distance for determining whether the gaze distance G estimated is a far distance greater than a predetermined distance. A determination means is provided as a function. If the display control means determines that the gaze distance determination means is a long distance, the display control means switches the superimposed image to the far form, and in the far form (for example, the form shown in FIG. 7 (b) or (c)) The predetermined content (for example, the vehicle speed image C1 and the route guidance image C2) displayed in the superimposed image of (for example, the near form shown in FIG. At least one of display control is performed to display at different positions.
In addition, the display control method by the display device 100 according to the first embodiment includes a step of estimating the gaze distance G of the user, and a superimposed image when the estimated gaze distance G is longer than a predetermined distance. Switching to the far-end mode, and in the far-end mode, at least one of display control of whether the predetermined content displayed in the superimposed image before switching is reduced and displayed or displayed at a different position from before switching And performing.
In addition, the program PG according to the first embodiment causes the computer (the control unit 40) to estimate the gaze distance G of the user and determine whether the estimated gaze distance G is longer than a predetermined distance. Whether the superimposed image is switched to the distant form according to the process to be performed and that it is determined to be distant, and in the distant form, the predetermined content displayed in the superimposed image before switching is reduced and displayed And a process of performing display control at least one of displaying at a position different from that before switching.

  Since it did in this way, as mentioned above, when the user who visually recognizes a superimposition image changes eyes | visual_axis from nearness to distantly, it can suppress suppressing recognition of a landscape.

Further, the display control means may make the specific content (for example, the time image C3) displayed in the superimposed image before switching difficult to view or delete in the mode for distance use.
Further, the display control means may display, in the distant form, the distant content (for example, the destination image C4) which has not been displayed in the superimposed image before switching.
By doing this, it is possible to perform appropriate display depending on the situation, and it is user-friendly.

(2) The display device 100 according to the second embodiment described above includes the image light generation device 1 (an example of the display unit) and the control unit 40. The control unit 40 estimates the gaze distance G of the user by the display control means that performs display control of the superimposed image, and the gaze distance for determining whether the gaze distance G estimated is a short distance closer than a predetermined distance. A determination means is provided as a function. When the display control means determines that the gaze distance determination means is at a short distance, the display control means displays the superimposed image in a portion other than the predetermined region (for example, a portion corresponding to the difference region VD shown in FIG. The content (for example, the route guidance image C5 or the warning image C6) is made difficult to visually recognize or deleted.
In the display control method by the display device 100 according to the second embodiment, in the step of estimating the gaze distance G of the user, and in the case where the estimated gaze distance G is a short distance closer than a predetermined distance, Making it difficult to visually recognize or erase the content displayed in a portion other than the predetermined region;
In addition, the program PG according to the second embodiment causes the computer (control unit 40) to estimate the gaze distance G of the user, and determine whether the estimated gaze distance G is closer than a predetermined distance. In accordance with the determination of the short distance and the determination of the short distance, a process of making it difficult to visually recognize or delete the content displayed in the portion other than the predetermined area in the superimposed image is executed.

  Since it did in this way, as above-mentioned, when the user who visually recognizes a superimposition image changes eyes | visual_axis from a distance from a distance, the troublesomeness of the content in a superimposition image being misaligned and being visually recognized (AR shift | offset | difference) Can be reduced.

In addition, the display control means may delete a portion (difference area VD) other than the predetermined area in the superimposed image when it is determined that the gaze distance determination means is at a short distance.
Further, the display control means displays the guidance image C7 in a predetermined area (in the near image VN) of the superimposed image when the gaze distance judgment means judges that the distance is short distance, and the guidance image C7 has a predetermined area It may be displayed in such a manner that the line of sight of the user is guided in the direction of the content (for example, the route guidance image C5 or the warning image C6) displayed in other parts. In this way, the user can be informed that the AR display is present if looking further away, which is user-friendly.

(3) The display device 100 according to the third embodiment described above includes the image light generation device 1 (an example of the display unit) and the control unit 40. The control unit 40 has a display control unit that controls display of a superimposed image as a function. The display control means displays a gaze position induction image IN (see FIGS. 11A to 11D) for guiding the user's gaze position to the set gaze position set according to the superimposed image in the superimposed image. Do.
Further, in the display control method by the display device 100 according to the third embodiment, a gaze position induction image for guiding the gaze position of the user to the set gaze position set according to the superimposed image is displayed in the superimposed image .
In addition, the program PG according to the third embodiment causes the computer (control unit 40) to superimpose a gaze position induction image for guiding the gaze position of the user to the set gaze position set according to the superimposed image in the superimposed image. Execute the process to be displayed on.

  Since it did in this way, even when the gaze position of the user who visually recognizes a superimposition image changes, it can guide to the suitable gaze position according to a superimposition image

The superimposed image is displayed corresponding to each of the left eye E2 and the right eye E1 of the user, and the gaze position guidance image IN includes the image M2 for the left eye and the image M1 for the right eye, and is an image for the left eye The gaze position of the user may be guided to the set gaze position by the parallax between M2 and the right-eye image M1.
In addition, the gaze position guidance image IN may guide the user's gaze position to the set gaze position by the moving image mode.

  In addition, the control unit 40 further includes a gaze distance determination unit that estimates the gaze distance G of the user and determines whether the estimated gaze distance G is a short distance closer than a predetermined distance, as a function of display control The means may display the gaze position induced image IN when it is determined that the gaze distance determination means is at a short distance. In this way, it is possible to quickly eliminate the AR shift that is likely to occur when the user viewing the superimposed image shifts the line of sight from a distance to a near position, by guiding it to the appropriate gaze position according to the superimposed image.

  Further, in any of the first to third embodiments, the display device 100 is used when the vehicle is on board, and the gaze distance determination unit is information (for example, the inside imaging data D1) indicating a captured image obtained by capturing the user's face. If the gaze distance G of the user is estimated on the basis of at least one of information indicating the face orientation of the user (for example, motion data D2) and vehicle information (for example, vehicle speed data D3) acquired from the vehicle Good.

  In the above description, in order to facilitate understanding of the present invention, the description of known technical matters is appropriately omitted.

DESCRIPTION OF SYMBOLS 100 ... Display apparatus 1 ... Image light generation apparatus 10 ... 1st display part 11 ... Display element, 11a ... Drive part 12 ... Light guide apparatus, 120 ... Lens-barrel part, 121 ... Light guide part, 121a ... Reflection surface 122 ... Transmission Part 123 123 Half mirror layer 20 Second display part 21 Display element 22 Light guiding device 30 Frame 30a Rim 30b Temple 31 Camera inside D1, inside image data 32 Motion sensor D2: Motion data 33: Outside camera 2 ... Control device 40: Control unit, 41: CPU, 42: Storage unit, PG: Program 50: Communication unit 200: On-vehicle device, D3: Vehicle speed data 300: Mobile terminal 3. In car, Reference Signs List 4 windshield 5 driving lane 6 dashboard 7 leading vehicle E eye E 1 right eye E 2 left eye L left L 1 L 2 image light V, V 1, V 2 virtual image OB target C1 ... vehicle speed image, C2 ... route guidance image, C3 ... time image, C4 ... destination image C5 ... route guidance image, C6 ... warning image, C7 ... guidance image VF ... distance image, VN ... near image VM ... Near image, M1 ... right eye image, M2 ... left eye image, IN ... gaze position guidance image

Claims (6)

Display means provided corresponding to the user's eyes and displaying a superimposed image that is viewed by the user as being superimposed on the landscape;
Display control means for controlling display of the superimposed image;
Fixation distance determination means for estimating the fixation distance of the user and determining whether the estimated fixation distance is a short distance closer than a predetermined distance;
The display control means makes the content displayed in a portion other than the predetermined area in the superimposed image visually difficult or erased when the gaze distance determination means determines that the short distance is the short distance.
A display device characterized by The display control means erases a portion other than the predetermined area in the superimposed image when the gaze distance determining means determines that the short distance is the short distance.
The display device according to claim 1, The display control means displays the guidance image in the predetermined area of the superimposed image when the gaze distance determination means determines that the short distance is the short distance,
The guidance image is displayed in such a manner as to guide the user's line of sight in the direction of the content displayed in the part other than the predetermined area.
The display device according to claim 1 or 2, characterized in that: Used when boarding a vehicle,
The gaze distance determination means is based on at least one of information indicating a captured image obtained by capturing the face of the user, information indicating a face orientation of the user, and vehicle information acquired from the vehicle. Estimate the gaze distance of the user,
The display device according to any one of claims 1 to 3, characterized in that: A display control method for displaying a superimposed image that is viewed by the user in an overlapping manner with a landscape on display means provided corresponding to the user's eyes,
Estimating the gaze distance of the user;
Displaying the content displayed in a portion other than the predetermined area in the superimposed image as being difficult to view or deleting the superimposed image when the estimated gaze distance is a short distance closer than the predetermined distance . A program for displaying on a display means provided corresponding to a user's eyes a superimposed image that is viewed by the user in an overlapping manner with a landscape,
On the computer
A process of estimating the gaze distance of the user and determining whether the estimated gaze distance is a short distance closer than a predetermined distance;
Processing for making it difficult to visually recognize or delete the content displayed in the portion other than the predetermined area in the superimposed image in accordance with the determination that the short distance is determined;
A program that runs

Images