JP6733174B2 - Head-mounted display device, head-mounted display device control method, and computer program - Google Patents

Description

The present invention relates to a technique of a head-mounted display device.

2. Description of the Related Art A head-mounted display device (head mounted display, HMD) that is a display device mounted on the head is known. A head-mounted display device, for example, uses a liquid crystal display and a light source to generate image light representing an image, and guides the generated image light to a user's eye using a projection optical system or a light guide plate. This allows the user to visually recognize the virtual image. There are two types of head-mounted display devices, a transmissive type in which a user can visually recognize an outside scene in addition to a virtual image and a non-transmissive type in which a user cannot visually recognize an outside scene. The transmissive head-mounted display device includes an optical transmissive type and a video transmissive type.

Patent Document 1 describes a head-mounted display device that divides a visual field of a user into two or more regions and displays a virtual image corresponding to each of the divided regions. Further, Patent Document 2 describes a multimodal interface that displays an image showing guide information for assisting input in the vicinity of the detected line-of-sight position of the user. Further, in Patent Document 3, a head-mounted display device that displays an image visually recognized by the user at the same position as the distance to the gazing point of the user, which changes according to a moving state of the user such as walking. Is listed.

Special table 2014-526157 gazette JP, 11-54778, A JP, 2014-225727, A

However, in the technique described in Patent Document 1, when a large number of virtual images are displayed in the divided areas, the processing load for displaying the virtual images is heavy and the display of the virtual images may be delayed. was there. Further, even in the technologies described in Patent Documents 2 and 3, when the capacity of the displayed image is large, the processing load for displaying the image may be heavy. In addition, in the conventional head-mounted display device technology, there have been demands for downsizing, cost reduction, easy manufacturing, and improvement of usability.

The present invention has been made to solve at least a part of the problems described above, and can be realized as the following modes.
A head-mounted display device according to an aspect of the present invention is capable of displaying an image in a display area, and an image display unit capable of transmitting an outside scene, and a line-of-sight direction specifying unit that specifies a line-of-sight direction of a user, An image changing unit that changes a display image displayed in the display area according to the identified line-of-sight direction, an outside scene image capturing unit that captures the outside scene, and an original of the display image associated with a specific object. And an image data acquisition unit that identifies and acquires the image data for each type of the image data, and the image changing unit is imaged when the outside scene imaging unit images the specific object. A specific object is detected from the image of the outside scene, the position of the display image is set in association with the position of the detected specific object, and the type of the image data acquired by the image data acquisition unit, and A mode of the display image associated with the specific object is determined based on the imaged position of the specific object with respect to the display area.
According to the head-mounted display device of this aspect, by changing the display mode of the image according to the position visually recognized by the user, the visibility of the image displayed by the user is not reduced, and the image is displayed. It is possible to reduce the processing load for displaying. In addition, the correspondence between the specific object and the displayed image can be more visually recognized by the user, and the usability for the user is improved. In addition, the display mode of the image visually recognized by the user varies depending on the type of image data, and the image can be visually recognized by the user as needed.
A head-mounted display device according to another aspect of the present invention is an image display unit capable of displaying an image in a display area and capable of transmitting an outside scene, and a line-of-sight direction specifying unit for specifying a line-of-sight direction of a user. And an image changing unit that changes a display image displayed in the display area according to the specified line-of-sight direction, an outside scene imaging unit that captures the outside scene, and a moving object that is moving in the outside scene. And a moving object specifying unit that does, the image changing unit detects a specific target object from the captured image of the outside scene, and associates the position of the display image with the position of the detected specific target object. The display mode of the display image associated with the specific object is set based on the speed at which the moving object approaches the image display unit.
According to the head-mounted display device of this aspect, by changing the display mode of the image according to the position visually recognized by the user, the visibility of the image displayed by the user is not reduced, and the image is displayed. It is possible to reduce the processing load for displaying. In addition, the correspondence between the specific object and the displayed image can be more visually recognized by the user, and the usability for the user is improved. Further, it is possible to inform the user of a danger of a car approaching the user.
A head-mounted display device according to still another aspect of the present invention is capable of displaying an image in a display area, and an image display unit capable of transmitting an outside scene, and a line-of-sight direction specification for specifying a line-of-sight direction of a user. Unit, an image changing unit that changes a display image displayed in the display area according to the identified line-of-sight direction, and an outside-view image capturing unit that captures the outside scene. Furthermore, the distance to the visual target visually recognized by the user is specified, the image changing unit detects the specific object from the captured image of the outside scene, and corresponds to the position of the detected specific object. The position of the display image is set, and the specific target is based on the visual target and a motion characteristic based on a moving background that is in a range other than the visual target and that is moving with respect to the visual target. A display mode of the display image associated with an object is set.
According to the head-mounted display device of this aspect, by changing the display mode of the image according to the position visually recognized by the user, the visibility of the image displayed by the user is not reduced, and the image is displayed. It is possible to reduce the processing load for displaying. In addition, the correspondence between the specific object and the displayed image can be more visually recognized by the user, and the usability for the user is improved. Further, since the display mode of the image is set by comparing the moving speed with the visual target visually recognized by the user, the sense of presence given to the user can be adjusted.
In addition, the present invention can be implemented as the following modes.

(1) According to one aspect of the invention, a head-mounted display device is provided. This head-mounted display device includes an image display unit capable of displaying an image in a display region; a line-of-sight direction specifying unit that specifies a user's line-of-sight direction; and a display line in the display region according to the specified line-of-sight direction. An image changing unit that changes the displayed image to be displayed. According to the head-mounted display device of this aspect, by changing the display mode of the image according to the position visually recognized by the user, the visibility of the image displayed by the user is not reduced, and the image is displayed. It is possible to reduce the processing load for displaying.

(2) In the head-mounted display device according to the above aspect, the image changing unit may change the display image in correspondence with the visual characteristic of the user that changes according to the specified line-of-sight direction. According to the head-mounted display device of this aspect, the speed of the process of displaying the image can be improved without impairing the visual characteristics such as the recognition of the user who visually recognizes the image.

(3) In the head-mounted display device according to the above aspect, the visual characteristic is a characteristic associated with a saccade; the image changing unit sets a range including a visual angle of the saccade as a visual field range of the user. You may set it. According to the head-mounted display device of this aspect, since the display mode of the display image is set in consideration of the saccade, the image in which the range of the eye movement not intended by the user is also considered is displayed in the display area in advance. it can.

(4) In the head-mounted display device according to the above aspect, when the image changing unit displays the display image in a part of the display area, depending on the position of the display image and the line-of-sight direction, The display image may be changed. According to the head-mounted display device of this aspect, the speed of the process of displaying the image can be improved without reducing the easiness of recognition by the user who visually recognizes the image.

(5) In the head-mounted display device according to the above aspect, the image display unit can transmit an outside scene; the head-mounted display device further includes an outside scene imaging unit that images the outside scene; The image changing unit may detect a specific object from the captured image of the outside scene and set the position of the display image in association with the detected position of the specific object. According to the head-mounted display device of this aspect, it is possible to make the user visually recognize the correspondence between the specific object and the displayed image, and the usability for the user is improved.

(6) In the head-mounted display device according to the above aspect, the line-of-sight direction identification unit includes a display unit direction detection unit that detects the orientation of the image display unit, and based on the detected orientation of the image display unit. Alternatively, the line-of-sight direction may be specified. According to the head-mounted display device of this aspect, for example, it is possible to easily and inexpensively specify the line-of-sight direction of the user without using a device that images the eyes of the user.

(7) In the head-mounted display device according to the above aspect, the line-of-sight direction identifying unit includes an eye image capturing unit that captures an image of the user's eye, and the line-of-sight based on the captured image of the user's eye. The direction may be specified. According to the head-mounted display device of this aspect, it is possible to accurately specify the line-of-sight direction of the user.

(8) In the head-mounted display device according to the above aspect, the line-of-sight direction specifying unit further specifies a distance to a visual recognition target visually recognized by a user; and the image changing unit includes the visual recognition target and the visual recognition target. Based on the movement characteristics based on the moving background that is moving with respect to the visual target as well as in the range other than the visual target, even if the display mode of the display image associated with the specific object is set. Good. According to the head-mounted display device of this aspect, since the display mode of the image is set by comparing the moving speed with the visual target visually recognized by the user, it is possible to adjust the sense of presence given to the user.

(9) In the head-mounted display device according to the aspect described above, the image changing unit may be configured such that, when the specific object is the moving background, the specific object does not move with respect to the visual target. Compared with, the display mode of the display image associated with the specific object may be set to a simple mode. According to the head-mounted display device of this aspect, the visibility of the image of the user is not reduced by reducing the processing for displaying the image associated with the target having the moving speed different from the visual target. Moreover, the processing load can be reduced.

(10) In the head-mounted display device according to the above aspect, the image display unit can transmit an outside scene; the head-mounted display device further includes: an outside scene imaging unit that captures the outside scene; An image data acquisition unit that identifies and acquires, for each type of image data, image data that is the source of the display image that is associated with an object; Corresponding to the specific object based on the type of the image data acquired by the image data acquisition unit and the position of the specific object captured with respect to the display area when the specific object is imaged The mode of the attached display image may be determined. According to the head-mounted display device of this aspect, the display mode of the image visually recognized by the user varies depending on the type of image data, and the image can be visually recognized by the user as necessary.

(11) In the head-mounted display device of the above aspect, further comprising: a moving object specifying unit that specifies a moving object that is moving in an outside scene; the image changing unit causes the moving object to approach the image display unit. You may set the display mode of the said display image matched with the said specific target object based on speed. According to the head-mounted display device of this aspect, it is possible to notify the user of a danger of a car or the like approaching the user.

(12) In the head-mounted display device according to the above aspect, the image changing unit may approach the image display unit at a speed equal to or higher than a preset speed of the moving object other than a predetermined area centered on the line-of-sight direction. In this case, even if the moving object is not set as the specific target object in advance, the moving object may be determined as the specific target object, and the display image set in advance for the determined specific target object may be set. According to the head-mounted display device of this aspect, the user can view the required image only when necessary, and the usability of the head-mounted display device for the user is improved.

(13) In the head-mounted display device according to the above aspect, when the display image includes a character image, the image changing unit displays the character image in the first visual field range including the specified line-of-sight direction. The character image is displayed as the display image in the second view range that is displayed in the display area as the display image and is larger than the first view range and does not include the first view range. Need not be displayed in. According to the head-mounted display device of this aspect, an image that does not include a character image is displayed in the display area in the visual field range where it is difficult for the user to recognize the character. The speed of the process of displaying an image can be improved without further reducing the size.

(14) In the head-mounted display device according to the above aspect, the image changing unit may include a third direction including the specified line-of-sight direction when the display image is an image visually recognized by a user as a three-dimensional image. In the visual field range, the display image is displayed in the display area as a three-dimensional image, and the display image is displayed in the fourth visual field range that is larger than the third visual field range and does not include the third visual field range. The display image may be displayed as a two-dimensional image in the display area so that it can be visually recognized by the user. According to the head-mounted display device of this aspect, since the two-dimensional image is displayed in the display area in the visual field range in which it is difficult for the user to recognize the distance to the display image, the user can visually recognize the corresponding image. The speed of the processing for displaying the image can be further improved without reducing the ease.

(15) In the head-mounted display device according to the above aspect, the image changing unit may include a fifth visual field including the specified line-of-sight direction when the display image is an image including information of a plurality of colors. In the range, the display image is displayed in the display area as an image including information of a plurality of colors, and in the sixth visual field range which is larger than the fifth visual field range and does not include the fifth visual field range. The display image may be displayed in the display area by using monochromatic shades. According to the head-mounted display device of this aspect, it is possible to further improve the speed of the process of displaying the image without reducing the easiness of recognition by the user who visually recognizes the image.

(16) In the head-mounted display device according to the above aspect, the image changing unit may display only the contour of the display image in the display area in the sixth visual field range. According to the head-mounted display device of this aspect, only the outer frame of the image is displayed in the range in which it is difficult for the user to recognize the color information, which reduces the ease of recognition by the user who visually recognizes the corresponding image. Without doing so, the speed of the process of displaying the image can be further improved.

(17) In the head-mounted display device according to the above aspect, the image changing unit may be a range that is larger than a seventh visual field range including the specified line-of-sight direction and does not include the seventh visual field range. In the view field range of 8, the display image may be displayed in the display area as a simplified image that is simpler than the image displayed in the seventh view range. According to the head-mounted display device of this aspect, only a simplified image indicating that there is an image is displayed in the display area in the visual field range where the user cannot recognize the detailed contents of the display image. The speed of the process of displaying the corresponding image can be further improved without reducing the easiness of recognition by the user who visually recognizes the corresponding image.

All of the plurality of constituent elements of each mode of the present invention described above are not indispensable, and for solving part or all of the above problems, or part or all of the effects described in the present specification. In order to achieve the above, it is possible to appropriately change or delete some of the constituent elements of the plurality of constituent elements, replace them with new constituent elements, and partially delete the limited contents. Further, in order to solve some or all of the above problems, or to achieve some or all of the effects described in the present specification, the technical features included in one embodiment of the present invention described above. Part or all of the technical features included in other forms of the present invention described above may be combined with each other to form an independent form of the present invention.

For example, one embodiment of the present invention can be realized as an apparatus including some or all of the three elements of the image display unit, the line-of-sight direction specifying unit, and the image changing unit. That is, this device may or may not have the image display unit. The device may or may not include the line-of-sight direction specifying unit. The device may or may not have the image changing unit. The image display unit may be capable of displaying an image in the display area, for example. The line-of-sight direction specifying unit may specify the line-of-sight direction of the user, for example. The image changing unit may change the display image displayed in the display area, for example, according to the specified line-of-sight direction. Such a device can be realized as, for example, a head-mounted display device, but can also be realized as a device other than the head-mounted display device. According to such a form, at least one of various problems such as improvement and simplification of operability of the device, integration of the device, and improvement of convenience for a user who uses the device can be solved. it can. Any or all of the technical features of the respective forms of the head-mounted display device described above can be applied to this device.

The present invention can be realized in various forms other than the head-mounted display device. For example, a display device, a head-mounted display device and a control method for the display device, a control system, a head-mounted display system, a display device, a computer program for realizing the functions of the control system and the display device, and the computer program It can be realized in the form of a recorded recording medium, a data signal including the computer program, and embodied in a carrier wave.

It is explanatory drawing which shows the external appearance structure of the head mounted display device (HMD) in this embodiment. It is a block diagram which shows functionally the structure of HMD in this embodiment. It is explanatory drawing of a gaze point and a user's gaze direction. It is a schematic diagram of a horizontal visual field and a vertical visual field centering on a gazing point. It is explanatory drawing which shows the degree of human recognition for every range of the visual field in a horizontal visual field. It is explanatory drawing which shows the degree of human recognition for every range of the visual field in a vertical visual field. It is explanatory drawing which shows a mode that image light is emitted by an image light generation part. It is a flow chart which shows a flow of image display processing. It is explanatory drawing which shows the relationship between the image display position of a corresponding image, and the distinguished visual field range. It is explanatory drawing which shows an example of a visual field of a user when a corresponding image is displayed. It is explanatory drawing which shows an example of a visual field of a user, when the corresponding image after the imaging range of a camera changes is displayed. It is explanatory drawing which shows an example of the visual field which a user visually recognizes, when the corresponding image after a user's gaze direction changes is displayed. It is explanatory drawing for comparing the visual field range of 1st Embodiment, and the visual field range of 2nd Embodiment. It is explanatory drawing of the visual field visually recognized by the user who is driving the motor vehicle in 3rd Embodiment. It is explanatory drawing of the visual field visually recognized by the user who is driving the motor vehicle in 3rd Embodiment. It is explanatory drawing of the visual field visually recognized by the user who is a pedestrian in 4th Embodiment. It is explanatory drawing which shows an example of the visual field which the user visually recognizes in a modification. It is explanatory drawing which shows an example of the visual field which the user visually recognizes in a modification. It is explanatory drawing which shows an example of the visual field which a user visually recognizes, when a corresponding image belongs to a visual field range. It is explanatory drawing of the display position of the corresponding image in a user's visual field in a modification. It is explanatory drawing which shows the external structure of HMD in a modification.

A. First embodiment:
A-1. Head-mounted display device configuration:
FIG. 1 is an explanatory diagram showing an external configuration of a head-mounted display device 100 (HMD100) according to this embodiment. The HMD 100 is a display device worn on the head and is also called a head mounted display (HMD). The HMD 100 of the present embodiment is an optical transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly visually recognize an outside scene. In the present specification, the virtual image visually recognized by the user by the HMD 100 is also referred to as “display image” for convenience.

The HMD 100 includes an image display unit 20 that allows the user to visually recognize a virtual image when mounted on the head of the user, and a control unit 10 (controller 10) that controls the image display unit 20.

The image display unit 20 is a mounting body mounted on the head of the user, and has a spectacle shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a first display unit. The camera 61, the second camera 62, the right-eye imaging camera 37, and the left-eye imaging camera 38 are included. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrows of the user when the user wears the image display unit 20.

The right holding portion 21 is provided by extending from the end portion ER which is the other end of the right optical image display portion 26 to a position corresponding to the user's temporal region when the user wears the image display portion 20. It is a member. Similarly, the left holding unit 23 extends from the end EL, which is the other end of the left optical image display unit 28, to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple (vine) of glasses.

The right display drive unit 22 and the left display drive unit 24 are arranged on the side facing the head of the user when the user wears the image display unit 20. Note that, hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are also collectively referred to simply as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as "optical image display unit".

The display drive units 22 and 24 include liquid crystal displays 241, 242 (Liquid Crystal Display, hereinafter also referred to as “LCDs 241 and 242”), projection optical systems 251, 252, and the like (see FIG. 2). The details of the configuration of the display drive units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 2) and a light control plate. The light guide plates 261 and 262 are formed of a light transmissive resin material or the like, and guide the image light output from the display drive units 22 and 24 to the eyes of the user. The light control plate is a thin plate-shaped optical element, and is arranged so as to cover the front side of the image display unit 20, which is the side opposite to the eye side of the user. The light control plate protects the light guide plates 261 and 262 and suppresses damage to the light guide plates 261 and 262 and adhesion of dirt. In addition, by adjusting the light transmittance of the light control plate, it is possible to adjust the amount of external light entering the eyes of the user and adjust the visibility of the virtual image. The light control plate can be omitted.

The first camera 61 and the second camera 62 capture an outside scene. The first camera 61 and the second camera 62 are arranged at different positions on the image display unit 20. The first camera 61 is arranged at the end ER, which is the other end of the right optical image display unit 26, in the head direction of the user when the image display unit 20 is mounted on the user. Similarly, the second camera 62 is arranged at the end EL, which is the other end of the left optical image display unit 28, in the head direction of the user when the image display unit 20 is attached to the user. ing. In the present embodiment, each of the first camera 61 and the second camera 62 has a position where one end of the right optical image display unit 26, which is the center of the image display unit 20, and one end of the left optical image display unit 28 are connected. They are arranged in line symmetry with respect to the passing center line. Note that, hereinafter, the first camera 61 and the second camera 62 are collectively referred to as cameras 61 and 62. In the present embodiment, since the place where the first camera 61 is arranged in the image display unit 20 and the place where the second camera 62 is arranged in the image display unit 20 are different, the imaging range of the first camera 61 The distance to the object imaged in both the image capturing range of the second camera 62 and the image capturing range of the second camera 62 can be calculated using trigonometry. The cameras 61 and 62 correspond to the outside scene image pickup section in the claims.

The right-eye imaging camera 37 is a camera that images the right eye of the user wearing the image display unit 20. Similarly, the left-eye imaging camera 38 is a camera that images the left eye of the user who wears the image display unit 20. With respect to the image of the right eye of the user captured by the right-eye imaging camera 37 and the image of the left eye of the user captured by the left-eye imaging camera 38, the direction identifying unit 166 described later identifies the line-of-sight direction of the user. Used for processing. The right-eye imaging camera 37 and the left-eye imaging camera 38 are monocular cameras, but may be stereo cameras. Hereinafter, the right-eye imaging camera 37 and the left-eye imaging camera 38 are collectively referred to as eye-imaging cameras 37 and 38. The eye imaging cameras 37 and 38 and the direction specifying unit 166 described later correspond to the line-of-sight direction specifying unit in the claims. The eye image pickup cameras 37 and 38 correspond to the eye image pickup section in the claims.

The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control unit 10. The connecting portion 40 includes a main body cord 48 connected to the control portion 10, a right cord 42, a left cord 44, and a connecting member 46. The right cord 42 and the left cord 44 are cords in which the main body cord 48 is branched into two. The right cord 42 is inserted into the housing of the right holding unit 21 from the front end portion AP of the right holding unit 21 in the extending direction, and is connected to the right display drive unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the front end portion AP of the left holding unit 23 in the extending direction, and is connected to the left display driving unit 24. The connecting member 46 is provided at a branch point between the main body cord 48 and the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30.

The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. A connector (not shown) that is fitted to each other is provided at each of the end of the body cord 48 on the opposite side of the connecting member 46 and the control unit 10. By mating/unmating the connector of the main body cord 48 and the connector of the control unit 10, the control unit 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

The control unit 10 is a device for controlling the HMD 100. The control unit 10 includes an enter key 11, a lighting unit 12, a display switching key 13, a track pad 14, a brightness switching key 15, a direction key 16, a menu key 17, and a power switch 18. There is. The enter key 11 detects a pressing operation and outputs a signal for determining the content operated by the control unit 10. The lighting unit 12 notifies the operating state of the HMD 100 by its light emitting state. The operating state of the HMD 100 includes, for example, turning on/off a power source. As the lighting unit 12, for example, an LED is used. The display switching key 13 detects a pressing operation and outputs, for example, a signal for switching the display mode of the content moving image between 3D and 2D. The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal according to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The brightness switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the brightness of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up, down, left, and right directions, and outputs a signal according to the detected content. The power switch 18 switches the power-on state of the HMD 100 by detecting the sliding operation of the switch.

FIG. 2 is a block diagram functionally showing the configuration of the HMD 100 in this embodiment. As illustrated in FIG. 2, the control unit 10 includes a storage unit 120, a power supply 130, a wireless communication unit 132, an operation unit 135, a CPU 140, an interface 180, a transmission unit 51 (Tx51), and a transmission unit 52 ( Tx52). The operation unit 135 receives an operation by the user, and includes an enter key 11, a display switching key 13, a track pad 14, a brightness switching key 15, a direction key 16, a menu key 17, and a power switch 18. The power supply 130 supplies electric power to each unit of the HMD 100. As the power supply 130, for example, a secondary battery can be used. The wireless communication unit 132 performs wireless communication with other devices such as a content server, a television, and a personal computer according to a predetermined wireless communication standard such as a wireless LAN and Bluetooth.

The storage unit 120 has a ROM that stores computer programs and a RAM that is used when the CPU 140 executes writing and reading of various computer programs.

The CPU 140 reads a computer program stored in the ROM of the storage unit 120, and writes and reads the computer program in the RAM of the storage unit 120, thereby operating system 150 (OS 150), display control unit 190, voice processing unit 170, It functions as the image processing unit 160, the image setting unit 165, the distance measuring unit 168, and the direction specifying unit 166.

The display control unit 190 generates a control signal for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 causes the right LCD control unit 211 to drive the right LCD 241 ON/OFF, the right backlight control unit 201 to drive the right backlight 221 ON/OFF, and the left LCD control unit according to control signals. The driving ON/OFF of the left LCD 242 by 212, the driving ON/OFF of the left backlight 222 by the left backlight control unit 202, and the like are individually controlled. Accordingly, the display control unit 190 controls the generation and emission of image light by the right display driving unit 22 and the left display driving unit 24, respectively. For example, the display control unit 190 causes both the right display drive unit 22 and the left display drive unit 24 to generate image light, only one of them to generate image light, and both of them to not generate image light. Note that generating image light is also referred to as “displaying an image”.

The display control unit 190 transmits control signals to the right LCD control unit 211 and the left LCD control unit 212, respectively, via the transmission units 51 and 52. The display control unit 190 also transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

The image processing unit 160 acquires the image signal included in the content. The image processing unit 160 separates the synchronizing signals such as the vertical synchronizing signal VSync and the horizontal synchronizing signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the cycle of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the sync signal has been separated into a digital image signal using an A/D conversion circuit or the like (not shown). After that, the image processing unit 160 stores the converted digital image signal in the DRAM in the storage unit 120 for each frame as image data (RGB data) of the target image. Note that the image processing unit 160 may perform image processing such as resolution conversion processing, various color tone correction processing such as adjustment of brightness and saturation, and keystone correction processing on the image data as necessary. ..

The image processing unit 160 transmits the generated clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52. .. The image data transmitted via the transmitter 51 is also referred to as “right-eye image data”, and the image data transmitted via the transmitter 52 is also referred to as “left-eye image data”. The transmission units 51 and 52 function as transceivers for serial transmission between the control unit 10 and the image display unit 20.

The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker (not shown) in the left earphone 34 connected to the connecting member 46. (Not shown), an audio signal is supplied. Note that, for example, when the Dolby (registered trademark) system is adopted, the audio signal is processed, and the right earphone 32 and the left earphone 34 respectively output different sounds with different frequencies, for example.

The direction specifying unit 166 specifies the line-of-sight direction of the user by analyzing the image of the right eye RE and the image of the left eye LE of the user captured by the eye imaging cameras 37 and 38. Various known techniques can be used as a method of identifying the user's gaze direction based on the imaged image of the right eye RE and left eye LE of the user. For example, the direction identifying unit 166 performs pattern matching or the like on the captured eye image based on a database of images in which the captured image is stored in the storage unit 120 in advance and is associated with the user's line-of-sight direction. The line-of-sight direction of the user can be specified by using the statistical identification method.

The distance measuring unit 168 uses the trigonometric method for the target object included in the captured image of the first camera 61 and the target object included in the captured image of the second camera 62, so that the image capturing of the cameras 61 and 62 is performed. The distance to the image of the object included in the image is measured. In the present embodiment, the distance to the object is measured by the first camera 61 and the second camera 62, which are stereo cameras in different locations, but in other embodiments, the TOF (Time Of The distance to the object may be measured by different measuring means such as a flight sensor.

The image setting unit 165 makes various settings for an image (display image) to be displayed on the image display unit 20. For example, the image setting unit 165 sets the display position of the display image, the size of the display image, the brightness of the display image, or the like, or allows the user to stereoscopically (3D) visually recognize the display image as a three-dimensional image. The right-eye image data and the left-eye image data are set so as to form binocular parallax (simply referred to as “parallax”). Further, the image setting unit 165 detects a preset image to be detected from the captured image by using pattern matching or the like for the captured image.

In the present embodiment, the image setting unit 165 sets different images depending on the relationship between the identified user's line-of-sight direction and the position of the image area in which the image is displayed, even if the images are based on the same image data. The image is displayed on the image display unit 20. For example, the image setting unit 165 has RGB color information in a predetermined visual field range centered on the gazing point of the user, based on three-dimensional image data having RGB (Red Green Blue) color information, It is displayed on the image display unit 20 as a three-dimensional image using the measured distance information. On the other hand, the image setting unit 165 causes the image display unit 20 to display a two-dimensional image having color information of only white and black outside the range of the predetermined visual field based on the three-dimensional image data. The image setting unit 165 reads, via the ROM, a look-up table that determines the correspondence between the line-of-sight direction and the position of the image region, how to display different images even with the same image data. Then, the image to be displayed on the image display unit 20 is determined.

FIG. 3 is an explanatory diagram of the relationship between the gazing point POR and the user's line-of-sight direction ED. FIG. 3 shows the line-of-sight direction ED when the right eye RE and the left eye LE of the user are looking at the gazing point POR. In this embodiment, the direction from the right and left eyeball center MP, which is the midpoint between the right eye RE and the left eye LE, to the gazing point POR is defined as the line-of-sight direction ED. The definition of the line-of-sight direction ED of the user can be variously modified, and for example, the direction from the dominant eye to the gazing point POR may be defined as the line-of-sight direction ED.

A human can visually recognize an object existing in a visual field that is a fixed range centered on the gazing point POR. In general, a human being is present at the same distance as the distance to the point of interest POR, and a subject located closer to the point of interest POR can be seen more clearly. There are various forms of visual characteristics in which the human visibility changes depending on the range of the visual field of the human. For example, there is a visual characteristic that the contour of the object can be clearly visually recognized up to a certain visual field range, but that the target can be recognized in the other visual field range, but the contour of the target cannot be grasped. Note that the same can be said for the display image that is a substitute for the target. In other words, a person's ability to recognize an object included in the visual field decreases as the distance from the gazing point POR increases. Therefore, in the present embodiment, the image setting unit 165, for images displayed at a position away from the user's line-of-sight direction ED, even if the images are based on the same image data, for example, reduces the resolution of the image. The means causes the image display unit 20 to display the image while reducing the processing for displaying the image on the image display unit 20. The image setting unit 165 corresponds to the image changing unit in the claims.

A region within about 5 degrees centering on the gazing point POR is called a discriminative field of view having excellent visual functions such as visual acuity. Further, an area within about 30 degrees in the horizontal direction about the gazing point POR and within about 20 degrees in the vertical direction about the viewpoint is called an effective visual field in which an object can be instantly recognized only by eye movement. There is. In addition, an area within about 60 to 90 degrees in the horizontal direction with respect to the gazing point POR and within about 45 to 70 degrees in the vertical direction with respect to the viewpoint focuses on the eye movement or the head movement in which the user moves the eye. It is called a stable gaze field where you can recognize the object without difficulty.

FIG. 4 is a schematic diagram of a horizontal visual field VRw and a vertical visual field VRh centering on the gazing point POR. FIG. 4 shows the relationship between the discrimination visual field, the effective visual field, and the stable fixation visual field in the horizontal direction and the vertical direction. In FIG. 4, when the user is looking at the gazing point POR, a visual field VR that can be visually recognized around the gazing point POR, a horizontal visual field VRw for explaining the visual field VR by dividing into horizontal components, and a visual field VR are shown. And a vertical visual field VRh for describing the vertical direction component by dividing it into vertical components. In FIG. 4, the discrimination visual field, the effective visual field, and the stable fixation visual field are expressed as the following relationship (1). The angle in parentheses in the relationship (1) is an angle on one side when the point of interest POR is the center.
Discrimination field of view: W0, H0 (up to about 5 degrees)
Effective field of view: W1 (up to approximately 30 degrees), H1 (up to approximately 20 degrees) (1)
Stable fixation field: W2 (up to approximately 60 (90) degrees), H2 (up to approximately 45 (70) degrees)
As shown in FIG. 4, in the vertical direction of the visual field VR, the stable gaze field on the lower side is wider than the upper side. This indicates that it is easier for humans to visually recognize the lower side than the upper side along the vertical direction.

FIG. 5 is an explanatory diagram showing the degree of human recognition for each range of the visual field in the horizontal visual field VRw. FIG. 5 shows the visual field range VR1 to the visual field range VR6 which are divided into six areas centering on the line-of-sight direction ED of the user US, and the following relationship (2) is established. The angle in parentheses in the relationship (2) is an angle on one side with the point of interest POR as the center.
VR1: Range that can be regarded as gazing point POR (up to about 1 degree)
VR2: Character recognition range (up to about 10 degrees)
VR3: Range in which eye movement can be performed easily (up to about 15 degrees) (2)
VR4: Range in which the shape can be identified (up to about 30 degrees)
VR5: Range in which color information can be identified (up to about 60 degrees)
VR6: Limit of field of view (up to about 94 degrees)
As described above, as shown in FIG. 5, the information that the user US can recognize varies depending on the visual field.

FIG. 6 is an explanatory diagram showing the degree of human recognition for each visual field range in the vertical visual field VRh. FIG. 6 shows the distinguished user US field of view for a vertical field of view VRh as an alternative to the horizontal field of view VRw of FIG. Therefore, the description of what kind of visual field range the distinguished visual field range is omitted.

The interface 180 illustrated in FIG. 2 is an interface for connecting the control unit 10 to various external devices OA that are content supply sources. The external device OA is, for example, a personal computer (PC), a mobile phone terminal, a game terminal, or the like. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.

The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a first The camera 61 and the second camera 62 are provided.

The right display drive unit 22 includes a reception unit 53 (Rx53), a right backlight control unit 201 (right BL control unit 201) and a right backlight 221 (right BL 221) that function as light sources, and a right LCD functioning as a display element. The control unit 211, the right LCD 241, and the right projection optical system 251 are included. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as a display element. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221, and the right LCD 241 are collectively referred to as an “image light generation unit”.

The reception unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is, for example, a light emitting body such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK input via the reception unit 53, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data for the right eye. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 into a parallel light flux. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The right projection optical system 251 and the right light guide plate 261 are collectively referred to as a “light guide section”.

The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display drive unit 24 includes a reception unit 54 (Rx 54), a left backlight control unit 202 (left BL control unit 202) and a left backlight 222 (left BL 222) that function as light sources, and a left LCD that functions as a display element. The controller 212 and the left LCD 242 and the left projection optical system 252 are included. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as a display element. The left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are collectively referred to as an “image light generation unit”. In addition, the left projection optical system 252 is configured by a collimator lens that converts the image light emitted from the left LCD 242 into a parallel light flux. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the image light along a predetermined optical path. The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “light guide unit”.

FIG. 7 is an explanatory diagram showing how image light is emitted by the image light generator. The right LCD 241 changes the transmittance of light that passes through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, so that the illumination light IL emitted from the right backlight 221 is displayed as an image. Is modulated to an effective image light PL. The same applies to the left side. Note that, as shown in FIG. 2, the backlight system is adopted in the present embodiment, but the image light may be emitted by using the front light system or the reflection system.

A-2. Image display processing:
FIG. 8 is a flowchart showing the flow of image display processing. In the image display process, when the CPU 140 causes the image display unit 20 to display the corresponding image associated with the detection target detected from the images captured by the cameras 61 and 62, the CPU 140 focuses on the user's line-of-sight direction ED. This is a process of changing the display image based on the distinguished visual field range and the display position of the image and displaying the changed image on the image display unit 20.

In the image display processing, first, the image setting unit 165 captures an outside scene using the cameras 61 and 62 (step S11). The image setting unit 165 detects a detection target associated with the corresponding image displayed on the image display unit 20 from the captured images of the cameras 61 and 62 by pattern matching or the like (step S13). When the detection target is not detected from the captured image (step S13: NO), the image setting unit 165 continuously waits for detection of the detection target from the captured image (step S13).

In the process of step S13, when the detection target is detected from the captured image (step S13: YES), the image setting unit 165 causes the image display unit 20 to display the corresponding image previously associated with the detection target. The image display position is determined (step S15). The image display position of the corresponding image can be variously determined. For example, a fixed position in the image display maximum area PN where the image display unit 20 can display the image regardless of the position of the detection target in the captured image. , Or in the vicinity of the position of the feature point set as the detection target. The image display maximum area PN corresponds to the display area in the claims.

When the image display position of the corresponding image is determined (step S15), the CPU 140 determines the gazing point POR and the line-of-sight direction ED of the user who wears the image display unit 20 on the head of the eye imaging cameras 37 and 38. It is specified using the captured image (step S17). The CPU 140 measures the distance to the detection target measured by the distance measuring unit 168 using the images captured by the cameras 61 and 62, and the line of sight of the user identified by the direction identifying unit 166 using the image captured by the eye image capturing cameras 37 and 38. The direction ED is specified.

The image setting unit 165 specifies to which of the plurality of visual field ranges the image display position of the corresponding image in the maximum image display area PN belongs based on the specified visual line direction ED ( Step S19). For example, as shown in FIG. 5, the image setting unit 165 distinguishes a plurality of visual field ranges centered on the specified visual line direction ED, and specifies which visual field range the image display position belongs to.

FIG. 9 is an explanatory diagram showing the relationship between the image display position of the corresponding image IM1 and the distinguished visual field range. FIG. 9 shows an example of the visual field VR visually recognized by the user. As shown in FIG. 9, the user can visually recognize the outside scene SC transmitted through the optical image display units 26 and 28. The outside scene SC includes a "sky tree" as one of landmarks as the detection target TW. FIG. 9 shows the gazing point POR of the user specified by the CPU 140 and the visual field ranges VR2, VR3, VR4, and VR5 set in a predetermined range centering on the gazing point POR. The visual field range VR3 is a range included in the visual field range VR3 and not included in the visual field range VR2. Similarly, the visual field range VR4 refers to a range included in the visual field range VR4 and not included in the visual field ranges VR2 and VR3, and the visual field range VR5 is included in the visual field range VR5 and the visual field range VR2. , VR3, VR4. In FIG. 9, the broken line image display maximum area PN, the star mark indicating the gazing point POR, and the alternate long and short dash line indicating each visual field range are shown for convenience of explanation, but these are not visually recognized by the user. .. In the present embodiment, since the captured images corresponding to the visual field visually recognized by the user are created by the captured images of the cameras 61 and 62, the created captured image and the visual field VR of the user match. In other embodiments, the captured image and the visual field VR of the user may be different.

When the image setting unit 165 performs the process of step S19 in FIG. 8, it sets the display mode of the corresponding image to be displayed in the image display maximum area PN based on the image data of the corresponding image associated with the detection target TW ( Step S23). In the present embodiment, the image setting unit 165 changes the setting of the mode of the display image, even for display images based on the same image data, in correspondence with the position where the corresponding image is displayed in the image display maximum area PN. .. When the display mode of the corresponding image IM1 is set, the image setting unit 165 displays the corresponding image IM1 for which the display mode is set in the image display maximum area PN (step S23).

FIG. 10 is an explanatory diagram showing an example of the visual field VR of the user when the corresponding image IM1 is displayed. As shown in FIG. 10, in the visual field VR of the user, an outside scene SC that can be visually recognized by the user through the optical image display units 26 and 28, and a corresponding image IM1 displayed in the image display maximum area PN, It is included. The corresponding image IM1 is an image displayed in the image display maximum area PN in association with the detection target TW detected from the captured image. The corresponding image IM1 is preset so as to be displayed near the detection target TW so as not to overlap with the detection target TW. In particular, when the specified user's gazing point POR overlaps the detection target TW, the corresponding image IM1 is displayed according to the position of the specified gazing point POR. The detection target TW corresponds to the specific target in the claims.

The image setting unit 165 specifies the positional relationship between the image display position for displaying the corresponding image IM1 associated with the position of the specified gazing point POR and each visual field range. When the image setting unit 165 identifies that at least a part of the corresponding image IM1 is included in the visual field range VR2 closest to the gazing point POR, the corresponding image is displayed based on the image data of the corresponding image IM1. The image data of IM1 is set as the display mode showing the most detail. The image setting unit 165 displays an image satisfying all of the following contents (A) to (C) as the corresponding image IM1 of the set display mode in the image display maximum area PN.
(A) Character image (B) Three-dimensional image (including distance information)
(C) Image including RGB color information The corresponding image IM1 in the set display mode includes a character image and is a three-dimensional image that is visually recognized at the same distance as the distance to the detection target TW. It is displayed in the area PN. In addition, “here” can be selected by moving a cursor or the like separately displayed in the image display maximum area PN to “here” which is underlined in the character image included in the corresponding image IM1. In order to indicate that, it is displayed in a color different from other character images (for example, blue (BLUE)). When “here” included in the corresponding image IM1 is selected by a predetermined operation received by the operation unit 135, the CPU 140 is connected to the home page (HP) of the detection target TW via the wireless communication unit 132. .. In the present embodiment, when the image display position of the corresponding image IM1 is included in the plurality of visual field ranges, the image setting unit 165 displays the entire corresponding image IM1 in the innermost visual field range. In this manner, the corresponding image IM1 is displayed in the image display maximum area PN. It should be noted that in other embodiments, it is possible to variously change the specification of which visual field range the display position of the corresponding image IM1 is included in. For example, the visual field range having the largest display area included in the image is displayed. It may be specified as the visual field range for setting the aspect. The visual field range VR2 corresponds to the first visual field range in the claims, and the visual field range other than the visual field range VR2 corresponds to the second visual field range in the claims.

In step S23 of FIG. 8, when the corresponding image IM1 is displayed in the image display maximum area PN, the CPU 140 monitors the change in the imaging range of the cameras 61 and 62 (step S25). CPU140 repeats the process after step S13, when the change of the imaging range of the cameras 61 and 62 is detected (step S25: YES). Note that the CPU 140 may treat a change in the imaging range of the cameras 61 and 62 by using a predetermined threshold value, and if the change is less than or equal to the predetermined threshold value, the captured image may be treated as no change.

FIG. 11 is an explanatory diagram showing an example of the visual field VR of the user when the corresponding image IM2 after the imaging ranges of the cameras 61 and 62 are changed is displayed. FIG. 11 shows a visual field VR in the case where the user is present at a position farther from the detection target TW than in FIG. 10 and the gazing point POR of the user is the month MN included in the outside scene SC. ing. That is, in FIG. 11, the gazing point POR of the user also changes as compared with FIG. 10. In the example shown in FIG. 11, the detection target TW detected from the images captured by the cameras 61 and 62 is included in a range further outside the visual field range VR5. In this case, the image setting unit 165 uses the distance information in which only the position of the detection target TW is simplified as the display mode of the image displayed in the image display maximum area PN based on the image data of the corresponding image IM2 that is the same as the corresponding image IM1. Is set as a display mode without. As illustrated in FIG. 11, the image setting unit 165 displays the corresponding image IM2, which is a circular image overlapping the position of the detection target TW, as the simplified corresponding image IM2 based on the image data of the corresponding image IM2. The area PN is displayed at a position at infinity. In the present embodiment, the image setting unit 165 displays, as the corresponding image of the detection target TW included in the visual field range VR5, an image only representing the position of the detection target TW in the image display maximum area PN. The range outside the visual range VR5 corresponds to the eighth visual range in the claims, and the visual range inside the visual range VR5 including the visual range VR5 corresponds to the seventh visual range in the claims. ..

When the corresponding image IM2 is displayed in step S23 of FIG. 8, the CPU 140 monitors the change in the imaging range of the cameras 61 and 62 (step S25). When the CPU 140 determines that there is no change in the captured images of the cameras 61 and 62 (step S25: NO), the CPU 140 monitors the change in the line-of-sight direction ED of the user (step S27). When the CPU 140 determines that the line-of-sight direction ED of the user has changed (step S27: YES), it repeats the processing from step S17.

FIG. 12 is an explanatory diagram showing an example of the visual field VR visually recognized by the user when the corresponding image IM3 after the user's line-of-sight direction ED is changed is displayed. FIG. 12 shows the visual field VR in the case where the gazing point POR of the user changes, as compared with FIG. 11. In the example illustrated in FIG. 12, the gazing point POR of the user is the building BD1 included in the outside scene SC, and the position of the detection target TW in the images captured by the cameras 61 and 62 is included in the visual field range VR4 and the visual field range VR5. ing. Therefore, the image setting unit 165 determines that the detection target TW belongs to the visual field range VR4 and displays an image based on the image data of the corresponding image IM3 that is the same image data as the corresponding image IM1 associated with the detection target TW. The display mode of the image displayed in the maximum area PN is set. The image setting unit 165 is linked to the home page (HP) based on the set display mode, and displays the corresponding image IM3 as a selectable button in the same color as “here” of the corresponding image IM1 in FIG. It is displayed in the image display maximum area PN as an image visually recognized at a distance. In other words, the image setting unit 165 does not include (A) the content of the character image and (B) the content of the three-dimensional image including the distance information with respect to the corresponding image IM1 shown in FIG. ) A corresponding image IM3 is displayed in the image display maximum area PN as an image including RGB color information. The visual field range VR4 corresponds to the fourth visual field range in the claims, and the visual field range inside the visual field range VR4 corresponds to the third visual field range in the claims.

In the process of step S23 of FIG. 8, when the corresponding image IM3 is displayed in the image display maximum area PN, the CPU 140 monitors the change of the imaging range of the cameras 61 and 62 (step S25). When the CPU 140 determines that there is no change in the imaging range of the cameras 61 and 62 (step S25: NO), the CPU 140 monitors the change in the line-of-sight direction ED of the user (step S27). When the CPU 140 determines that there is no change in the line-of-sight direction ED of the user US (step S27: NO), the CPU 140 monitors reception of a predetermined operation for terminating the image display processing on the operation unit 135 (step S29). When the predetermined operation for ending the image display processing is not accepted (step S29: NO), the CPU 140 repeats the processing of step S25 and thereafter. In the process of step S29, when a predetermined operation for ending the image display process is accepted (step S29: YES), the CPU 140 ends the image display process.

Further, in the HMD 100 according to the present embodiment, the image setting unit 165, based on the relationship between the identified user's line-of-sight direction ED and the display position of the corresponding image, even if the image data of the same corresponding image is different, Is displayed in the maximum image display area PN of the image display unit 20. Therefore, in the HMD 100 of the present embodiment, the processing speed for displaying the corresponding image can be improved without reducing the easiness of recognition by the user who visually recognizes the corresponding image.

Further, in the HMD 100 of the present embodiment, the image setting unit 165 sets the image display position of the corresponding image in association with the position of the detection target TW detected from the images captured by the cameras 61 and 62. Therefore, in the HMD 100 of the present embodiment, the user can visually recognize the correspondence between the detection target TW and the corresponding image more easily, and the usability for the user is improved.

Further, in the HMD 100 of the present embodiment, the user's line-of-sight direction ED is specified by the images captured by the eye image pickup cameras 37 and 38, so that the user's line-of-sight direction ED and the gazing point POR can be specified accurately.

Further, in the HMD 100 of the present embodiment, the image setting unit 165 displays the corresponding image IM1 including the character image in the visual field range VR2 in the image display maximum area PN of the image display unit 20 based on the image data of the corresponding image IM1. .. On the other hand, the image setting unit 165 displays the corresponding image (for example, the corresponding image IM2) that does not include the character image in the image display unit 20 in the visual field range outside the visual field range VR2 based on the image data of the corresponding image IM1. The image is displayed in the maximum image display area PN. Therefore, in the HMD 100 of the present embodiment, the corresponding image that does not include the character image is displayed on the image display unit 20 in the visual field range where it is difficult for the user to recognize the character, so that the user who visually recognizes the corresponding image can easily recognize the corresponding image. The speed of the process of displaying the corresponding image can be improved without further reducing the size.

Further, in the HMD 100 of the present embodiment, the image setting unit 165, based on the image data of the corresponding image IM1, the corresponding image IM2 of the simplified circular image in the visual field range VR5, the image display maximum area of the image display unit 20. Display on PN. Therefore, in the HMD 100 of the present embodiment, only a simplified image indicating that there is an image in the visual field range in which the user cannot recognize the detailed contents of the displayed image is displayed on the image display unit 20, so that the corresponding image is displayed. It is possible to further improve the speed of the processing for displaying the corresponding image without reducing the easiness of recognition by the user who visually recognizes.

Further, in the HMD 100 of the present embodiment, the image setting unit 165, based on the image data of the corresponding image IM1, in the visual field range VR4, regardless of the distance to the gazing point POR of the user, two-dimensional information that does not include distance information. The corresponding image IM3 as an image is displayed in the image display maximum area PN of the image display unit 20. Therefore, in the HMD 100 of the present embodiment, since the two-dimensional image is displayed on the image display unit 20 in the visual field range in which it is difficult for the user to recognize the distance to the display image, it is easy for the user to visually recognize the corresponding image. It is possible to further improve the processing speed for displaying the corresponding image without reducing

B. Second embodiment:
The HMD 100a of the second embodiment is different from the HMD 100 of the first embodiment in the visual field range VR2 used by the image setting unit 165a to set the display mode of the corresponding image, and other configurations are the same as the first embodiment. The same as the HMD100.

In the second embodiment, the visual field range VR2 is determined in consideration of human eye movement. It is known that human eye movements can be divided into three types: fixational eye movements, following movements, and saccades. Even when a person is gazing at a specific object, the actual line of sight of the person is moving, and such an involuntary movement is called fixational eye movement. The following movement is a movement for tracking the movement of a moving object when a human is looking at the moving object. The speed of the follow-up movement is about 30 degrees/s (dps) at maximum. Saccade is a movement in which the eyeball moves in small steps at high speed. For example, when reading a book, a saccade is a movement in which the line of sight is moved to the left end, which is the beginning of a line-break sentence, after reading to the right end of a horizontally written sentence. The speed of the saccade eyeball is about 300 to 600 dps, and the initial movement of the saccade requires about 160 to 170 milliseconds (ms). When the saccade distance is represented by a visual angle, it is about 2 to 6 degrees (degree), although there are differences depending on individuals. In the second embodiment, the image setting unit 165a sets the visual field range VR2S in which the visual angle due to saccade is added to the visual field range VR2 of the first embodiment to be the same as the display aspect in VR2 of the first embodiment.

FIG. 13 is an explanatory diagram for comparing the visual field range VR2 of the first embodiment and the visual field range VR2S of the second embodiment. 13 does not show the visual field ranges VR3, VR4 and VR5 shown in FIG. 10 of the first embodiment, but newly shows the visual field range VR2S. Further, in FIG. 13, for convenience of comparison with the first embodiment, the visual field range VR2 of the first embodiment is shown by a broken line.

The image setting unit 165a of the second embodiment, even if it is outside the visual field range VR2 of the first embodiment, if it is inside the visual field range VR2S, sets the corresponding image IM1 as the image associated with the detection target TW. The image is displayed in the maximum image display area PN. In other words, in the second embodiment, the visual field area displayed as the corresponding image IM1 when the detection target TW is detected is expanded from the visual field range VR2 to the visual field range VR2S.

As described above, in the HMD 100a of the second embodiment, the image setting unit 165a changes the display mode of the corresponding image by using the saccade as the visual characteristic that is the human eye movement. Therefore, in the HMD 100a of the second embodiment, as in the first embodiment, in the area outside the visual field range VR2S, in order to display the corresponding image associated with the detection target TW in the image display maximum area PN. Reduce the processing required. Further, in the area of the visual field range VR2S, since the corresponding image IM1 is displayed in the maximum image display area PN, even if the gazing point POR changes to the outside of the visual field range VR2 of the first embodiment due to the saccade, If it is inside, there is no need to delay the display of the corresponding image. This improves the usability of the HMD 100a for the user.

C. Third embodiment:
In the HMD 100b of the third embodiment, when the image setting unit 165b sets the corresponding image to be displayed in the image display maximum area PN, in addition to the line-of-sight direction ED and the detection target considered in the above-described embodiment, the user gazes. The display mode of the corresponding image is set in consideration of the motion characteristics between the visible target being operated and the range other than the visible target. The motion characteristics to be considered include, for example, the feeling that the user feels when the visual target is stationary and the background of the visual target is moving with respect to the visual target. Specifically, while the user on the train is viewing the advertisement on the window of the train as a target for viewing, the scenery outside the running train is perceived by the user as moving. When it is done. Thus, perceiving that a person is moving with respect to himself is called a motion stimulus. On the other hand, what humans perceive to be stationary with respect to themselves is called a static stimulus. It is generally known that human beings perceive that they are moving when receiving a static stimulus as a visual target and a motion stimulus as a background around the visual target. In addition, when a human receives a motion stimulus from near the visual target (especially before and after the depth is used as a reference), the human perceives that he is moving. On the other hand, when a motion stimulus is received as a visual target and a static stimulus is received as a background, the user perceives that he is not moving. In the HMD 100b of the third embodiment, by giving or eliminating a self-kinematic sensation that the user feels that he is exercising, the user has a sense of presence or the presence of the user is suppressed. In the present specification, a background or the like that the user of the HMD 100b recognizes as moving with respect to a visual target that is recognized as stationary is also referred to as a moving background.

FIG. 14 is an explanatory diagram of a visual field VR visually recognized by a user who is driving an automobile in the third embodiment. As shown in FIG. 14, in the third embodiment, the user visually recognizes the road in the forest as the outside scene SC and the corresponding images IM7 and IM8 displayed in the image display maximum area PN. .. The user operates the steering wheel HD of the automobile with the right hand RH and the left hand LH to see the gazing point POR. The corresponding image IM7 is an image associated with the car navigation (hereinafter, also simply referred to as “car navigation”) mounted on the automobile that is the detection target TWa. IM7 is an image that prompts the user to go straight as a car navigation route to the destination. The corresponding image IM7 is displayed in the image display maximum area PN as an image (mark) of an arrow in which information such as characters is deleted. The corresponding image IM8 is an image associated with the road that is the detection target TWb included in the outside scene SC. The corresponding image IM8 is a character image of "animal caution" that prompts the user to pay attention to wild animals that may jump out from the side of the road.

In the third embodiment, the direction specifying unit 166 is used by using the captured image of the right eye RE captured by the right eye imaging camera 37 and the captured image of the left eye LE captured by the left eye imaging camera 38. In addition to the line-of-sight direction ED of the person, the distance to the visual target OJ1 can be specified. In the example illustrated in FIG. 14, the distance to the visual target OJ1 specified by the direction specifying unit 166 is 10 meters (m). That is, the user is gazing at the road included in the outside scene SC of the automobile as the visual target OJ1. In this case, when the automobile is running, the user receives a motion stimulus from the road that he recognizes to be moving. In addition, the user receives a motion stimulus from the background near the visual target OJ1 by perceiving that the background near the road of the visual target OJ1 is also moving. Therefore, the user perceives that he is exercising.

The distance measuring unit 168b according to the third embodiment uses the captured images of the outside scene captured by the cameras 61 and 62 at a plurality of time points (for example, moving images), and identifies a specific target in the captured images with respect to the cameras 61 and 62. It can be determined whether or not it is moving. The image setting unit 165b sets the display mode of the corresponding image IM7 and the display mode of the corresponding image IM8 based on the presence or absence of movement of each of the imaged objects determined by the distance measuring unit 168b.

Since the user receives the motion stimulus from the visual target OJ1, the image setting unit 165b also includes the corresponding image IM8 associated with the detection target TWb that similarly gives the motion stimulus to the user, including information such as characters. The image is displayed in the image display maximum area PN. On the other hand, the image setting unit 165b gives a static stimulus to the user because the detection target TWa arranged inside the vehicle in which the user is riding moves at the same speed as the user. The image setting unit 165b sets the corresponding image IM7 associated with the detection target TWa that gives the user a static stimulus different from the motion stimulus given to the user by the visual target OJ1 as an image representing a simple mark in the image display maximum area PN. Display it. Briefly, in the HMD 100b of the third embodiment, the image setting unit 165b, based on the relationship between the distance to the visual target OJ1 as the user's line-of-sight direction ED and the speed with respect to the visual target OJ1, the corresponding image IM7, Set the display mode of IM8.

FIG. 15 is an explanatory diagram of a visual field VR visually recognized by a user who is driving an automobile in the third embodiment. As shown in FIG. 15, the user visually recognizes the outside scene SC and the corresponding images IM8 and IM9. In the example shown in FIG. 15, the user is gazing in the same direction as the line-of-sight direction ED of the user shown in FIG. In this case, in the example shown in FIG. 14 and the example shown in FIG. 15, the user visually recognizes the same area in the image display maximum area PN in which the image can be displayed. That is, the positions of the detection target TWa and the detection target TWb in FIG. 15 are the same as the positions of the detection target TWa and the detection target TWb in FIG.

In FIG. 15, the visual target OJ2 that the user gazes at is the windshield of the automobile, which is 50 cm (cm) away from the user. In other words, the user is gazing at the visual target OJ2 that seems to be stationary by moving with him. In the example shown in FIG. 15, the user receives a static stimulus from the visual target OJ2, a static stimulus from the detection target TWa, and a motion stimulus from the detection target TWb. Therefore, the image setting unit 165b sets an image including character information as a display mode of the corresponding image IM9 associated with the detection target TWa that gives the user the same static stimulus as the visual recognition target OJ2. The corresponding image IM9 is a character image of "go straight as it is" to prompt the car navigation to go straight to guide the user to the destination. On the other hand, the image setting unit 165b deletes character information as a display mode of the corresponding image IM10 associated with the detection target TWb that gives the user a motion stimulus different from the static stimulus of the visual target OJ2, and is surprised to call attention. Only the mark "!" is set. The simple display mode in the claims means that when two display images displayed in the maximum image display area PN are compared, the shape included in the display image is not more complicated (for example, simple “○” or “ The shape of “×”), the number of colors used for the display image is smaller, the area of the display image is smaller (for example, display with fewer characters), and the like.

As described above, in the HMD 100b of the third embodiment, the direction specifying unit 166b specifies the distance to the visual targets OJ1 and OJ2, and the image setting unit 165b uses the visual stimulus OJ1 and OJ2 to receive the static stimuli and the user. The display mode of the corresponding images IM7, IM8, IM9, IM10 associated with the detection targets TWa, TWb is set based on the motion characteristics such as the motion stimulus. Therefore, in the HMD 100b of the third embodiment, the corresponding image in consideration of the user's motion characteristics is displayed in the image display maximum area PN, so that the corresponding image can be used to adjust the user's sense of presence. Further, when displaying an image that is easily visible to the user and a corresponding image that is difficult to see in the image display maximum area PN, by reducing the information of the corresponding image that is difficult to view, the load of the processing executed by the HMD 100b can be reduced. ..

Further, in the HMD 100b of the third embodiment, as compared with the example shown in FIG. 14 and the example shown in FIG. 15, the image setting unit 165b determines that the detection target located in the same area of the outside scene with respect to the line-of-sight direction ED. Even in TWa and TWb, the display mode of the corresponding image is changed depending on the motion characteristics of the visual target. Therefore, in the HMD 100b of the third embodiment, it is possible to further adjust the sense of presence that the user feels by using the corresponding image.

D. Fourth Embodiment:
In the HMD 100c of the fourth embodiment, the moving speed of the imaged target is specified, and the image setting unit 165c determines the moving speed of the specified target with respect to the user wearing the image display unit 20c and the line of sight of the user. The display mode of the corresponding image is set based on the direction ED. In addition, in the HMD 100c, the wireless communication unit 132c performs wireless communication to acquire the detection target data and the image data that is the basis of the corresponding image associated with the detection target from another server. The image data obtained by the wireless communication unit 132c is classified into a plurality of categories, and the wireless communication unit 132c also obtains these categories in association with the image data. The categories include, for example, “guidance” that teaches the user how to reach the destination, and “warning” that calls the user's attention due to a sudden event such as a natural disaster. The wireless communication unit 132c corresponds to the image data acquisition unit in the claims.

FIG. 16 is an explanatory diagram of a visual field VR visually recognized by a user who is a pedestrian in the fourth embodiment. As shown in FIG. 16, the user visually recognizes the outside scene SC including the building BD4 and the like, and the corresponding image IM11 and the corresponding image IM12 displayed in the image display maximum area PN. In the example shown in FIG. 16, the user visually recognizes the entrance EN of the building BD4 as the gazing point POR. Note that FIG. 16 shows the visual field range VR2 described in the first embodiment centering on the point of interest POR. The corresponding image IM11 is an image displayed in association with the moving object CA1 as the detection target. The moving object CA1 is a circulating bus that passes (left and right) across the road in front of the building BD4 so as to cross the user. Corresponding image IM11 is a character image "Circle bus in the city" that represents that moving object CA1 is a bus circulating in the city.

The corresponding image IM12 is an image displayed in association with the moving object CA2 as a detection target. The moving object CA2 is an automobile traveling in a direction toward the user. The image setting unit 165c determines the display mode of the corresponding image IM12 based on the image data acquired via the wireless communication unit 132c, and displays the corresponding image IM12 in the image display maximum area PN. In the third embodiment, the direction identifying unit 166c measures the moving speed of the target included in the captured images captured by the cameras 61 and 62 with the image display unit 20c as a reference. In the corresponding image IM12, the category associated with the original image data is “warning”. In this case, when the image setting unit 165c determines that the moving object CA2 approaches the user (image display unit 20c) at a speed equal to or higher than a preset speed, regardless of the positional relationship between the gazing point POR and the visual field range VR2. The corresponding image IM12 is displayed in the maximum image display area PN larger than other corresponding images such as the corresponding image IM11. For this reason, as shown in FIG. 16, the font of the corresponding image IM12 of the character image “coming in!”, which indicates the approach of the moving object CA2 to the user, is larger than the font of the corresponding image IM11 in the image display maximum area PN. Is displayed. The direction specifying unit 166c corresponds to the moving object specifying unit in the claims.

As described above, in the HMD 100c of the fourth embodiment, in addition to the line-of-sight direction ED, the image setting unit 165c sets the speed at which the moving object CA2 as the detection target approaches the user wearing the image display unit 20c. Based on this, the display mode of the corresponding image IM12 associated with the moving object CA2 is set. Therefore, in the HMD 100c of the fourth embodiment, when there is a moving object that approaches the user at a high speed, the user can be notified of the danger, and the convenience of the user is improved.

E. Modification:
The present invention is not limited to the above-described embodiment, and can be implemented in various modes without departing from the scope of the invention, and for example, the following modifications are possible.

E-1. Modification 1:
FIG. 17 is an explanatory diagram showing an example of the visual field VR visually recognized by the user in the modified example. FIG. 17 shows the visual field VR after the gazing point POR of the user is changed from the building BD1 to the building BD2, as compared with FIG. 12 of the first embodiment. In the example shown in FIG. 17, since the gazing point POR of the user is the building BD2, the image setting unit 165 determines that the position of the detection target TW in the images captured by the cameras 61 and 62 belongs to the visual field range VR3, The corresponding image IM4 as a display image is displayed in the image display maximum area PN. The corresponding image IM4 is an image displayed as a three-dimensional image including (B) distance information in addition to the corresponding image IM3 shown in FIG. In other words, the corresponding image IM4 is an image including (B) the content of the three-dimensional image including the distance information and (C) the RGB color information. The visual field range VR4 corresponds to the fourth visual field range in the claims, and the visual field range inside the visual field range VR4 corresponds to the third visual field range in the claims.

As described above, in the HMD 100 of this modification, the image setting unit 165 of the image display unit 20 displays the corresponding image IM4 as the three-dimensional image including the distance information in the visual field range VR3 based on the image data of the corresponding image IM1. The image is displayed in the maximum image display area PN. Therefore, in the HMD 100 of this modified example, a three-dimensional image is displayed based on the same image data in a range where the user can recognize it as a three-dimensional image, and a two-dimensional image is displayed in a range where the user cannot easily recognize the three-dimensional image. The image is displayed. As a result, the speed of the process of displaying the corresponding image can be further improved without reducing the ease of recognition by the user who visually recognizes the corresponding image.

FIG. 18 is an explanatory diagram showing an example of the visual field VR visually recognized by the user in the modified example. FIG. 18 shows the visual field VR after the user's gazing point POR changes from the building BD1 to the building BD3, as compared with FIG. 12 of the first embodiment. In the example illustrated in FIG. 18, since the gazing point POR of the user US is the building BD3, the image setting unit 165 determines that the position of the detection target TW in the images captured by the cameras 61 and 62 belongs to the visual field range VR5. The corresponding image IM5 as the display image is displayed in the image display maximum area PN. The corresponding image IM5 is an image that is represented by shading of white and black and is created based on the same image data as the corresponding image IM1. Corresponding image IM5 does not include RGB color information, and in order to improve the user's visibility, a dark and light shaded rectangular image is used as a background and the light and shaded letters "HP" are displayed. It is an image. In other words, the corresponding image IM5 is an image that does not include (C) RGB color information with respect to the corresponding image IM3. The visual field range VR5 corresponds to the sixth visual field range in the claims, and the visual field range inside the visual field range VR5 corresponds to the fifth visual field range in the claims.

As described above, in the HMD 100 of this modification, the image setting unit 165 displays the corresponding image IM5 that does not include RGB color information in the image display unit 20 in the visual field range VR5 based on the image data of the corresponding image IM1. The maximum area PN is displayed. Therefore, in the HMD 100 of this modification, a corresponding image using the RGB color information is displayed in the range where the user can recognize various color information including RGB based on the same image data, and the user displays the color information. In the range where it is difficult to recognize, the corresponding image having only the color information of white and black is displayed. As a result, the speed of the process of displaying the corresponding image can be further improved without reducing the ease of recognition by the user who visually recognizes the corresponding image.

E-2. Modification 2:
FIG. 19 is an explanatory diagram showing an example of the visual field VR visually recognized by the user when the corresponding image IM5 belongs to the visual field range VR5. FIG. 19 shows the visual field VR in the case where the corresponding image IM6 which is the image of the hat is superimposed and displayed at the position of the head on the imaged human subject OB. The subject OB is a person who is trying on the ski wear on the upper body and the lower body, and does not actually try on the hat on the head. As shown in FIG. 19, the gazing point POR of the user of the HMD 100 is a position distant from the subject OB, and in this case, the area in which the corresponding image IM6 is displayed in the image display maximum area PN is in the visual field range VR5. Belong to The image data that is the basis of the corresponding image IM6 also includes color information such as the pattern of the hat displayed as the corresponding image IM6. However, since the display position of the corresponding image IM6 belongs to the visual field range VR5, the image setting unit 165 does not include color information such as a pattern or RGB in the corresponding image IM6 and displays only the outer frame of the hat in dark and light shades of black. The displayed image is displayed in the image display maximum area PN. In other words, the image setting unit 165 displays an image of only the outer frame of the hat image in the maximum image display area PN in order to allow the user to recognize the contour of the corresponding image IM6.

As described above, in the HMD 100 of this modification, the image setting unit 165 displays the corresponding image IM6 of only the contour of the display image in the image display maximum area PN of the image display unit 20 in the visual field range VR5 based on the image data. Let Therefore, in the HMD 100 of this modification, only the outer frame of the image is displayed in the range in which the user is difficult to recognize the color information, so that the recognizability of the user who visually recognizes the corresponding image is not reduced. The speed of the processing for displaying the image can be further improved.

E-3. Modification 3:
In the above embodiment, the display image is displayed in a part of the image display maximum area PN, but the display image may be an image displayed in the entire image display maximum area PN, or the images of the cameras 61 and 62. The image may be different from the corresponding image associated with the detection target detected from the captured image. For example, the display image may be a moving image such as a movie that is played by content or the like regardless of the detection target. Further, the moving image as the display image may be displayed in the entire image display maximum area PN. When the image is a moving image displayed in the entire image display maximum region PN, the image setting unit 165 hides the moving image when it is determined that the line-of-sight direction ED of the user is not on the image display maximum region PN. Good. In this way, the image setting unit 165 variously modifies and sets the display mode of the display image based on the image displayed in the image display maximum area PN and the specified line-of-sight direction ED of the user.

In the above embodiment, the line-of-sight direction ED of the user is specified based on the images captured by the eye imaging cameras 37 and 38, but may be specified by another method. For example, the line-of-sight direction ED of the user may be specified by the orientation of the image display unit 20. Specifically, the line-of-sight direction ED of the user who wears the image display unit 20 on his/her head is specified to be the front direction of the image display unit 20, and the center of the image display maximum area PN is always set to the user. May be set as the gazing point POR. According to the HMD 100 of this modification, the user's line-of-sight direction ED can be specified easily and inexpensively without using a device such as the eye imaging cameras 37 and 38. Further, a gyro sensor may be mounted on the image display unit 20, and the orientation of the image display unit 20 may be specified based on the angular velocity detected by the gyro sensor. Further, the line-of-sight direction of the user may be specified by a myoelectric potential detected by an electrode arranged in a portion that comes into contact with the temples or the nose of the user.

Further, in the above embodiment, the range for the user's gazing point POR such as the visual field range VR2 is set according to the visual field of the human and the degree of visual recognition by the human in relation to the visual field, but various modifications are possible depending on these ranges. is there. The visual field range VR2 and the like may be set depending on the brightness such as the brightness of the outside scene SC, the visual acuity of the user of the HMD 100, the resolution of the displayed image, and the like.

In the above-described embodiment, the distance to the target object included in the captured images is measured based on the captured images of the cameras 61 and 62. However, the method of measuring the distance to the target object and the presence/absence of the measurement are variously modified. It is possible. Also, the distance to the object does not necessarily have to be measured.

In the above-described embodiment, the user wearing the image display unit 20 can see the transparent outside scene SC, but a so-called closed HMD in which the user cannot see the outside scene SC may be used.

Further, in the above-described embodiment, the stereo camera including the first camera 61 and the second camera 62 is used as the image capturing unit that captures the outside scene SC, but one camera may be used or 3 More than one camera may be used. Further, the position where the camera is arranged does not necessarily have to be included in the image display unit 20, and the camera may be configured separately from the image display unit 20.

In addition, since the visual characteristics of the user vary from person to person, the correspondence between the visual field range and the image displayed on the image display unit 20 may be changed by the operation unit 135 receiving a predetermined operation.

In the above embodiment, the method of forming binocular parallax is used in order to make the user recognize as a three-dimensional image, but the method of making the user recognize as a three-dimensional image can be variously modified. For example, as a method for allowing the user to recognize the image as a three-dimensional image, an image in which vergence movement is performed is displayed on the image display unit 20, a shadow is added to the image, or the size of the image is changed to express a sense of depth. May be. Also, the two-dimensional image may be an image displayed on the image display unit 20 as infinity that does not include distance information, and can be variously modified.

As shown in FIG. 18, in the visual field range VR5, the corresponding image IM5 that does not include the RGB color information is displayed in the image display maximum area PN, but the example in which the color information is not included is not limited to this, and various modifications are possible. It is possible. For example, an image in which light and shade is given to a single color such as red (RED) may be displayed in the image display maximum area PN. The image with reduced color information may simply be an image displayed in gray scale of a specific color.

In the above embodiment, as shown in FIG. 11, the corresponding image IM2, which is a circular image, is displayed in the image display maximum area PN as a simplified image, but the simplified image can be variously modified. For example, it may be an image of a triangle or a square other than a circle, or may be a shape of some symbol mark associated with the image data, which makes it easy to associate the image data. The simplified image is not complicated as compared with the image displayed based on the image data (for example, the corresponding image IM1), and has a degree of causing the user to recognize that the simplified image exists therein. Any image will do.

In the above-described embodiment, the aspect of the image displayed in the image display maximum area PN is changed based on the relationship between the visual line direction of the user and the visual field range of the user. May change. For example, when included in the visual field range VR2, the image is displayed in the entire image display maximum area PN, and when it is outside the visual field range VR5, it is simplified to a part of the image display maximum area PN. The displayed image may be displayed.

E-4. Modification 4:
In the fourth embodiment, the moving object CA2 is preset as the detection target, but a new detection is made when an object that is not preset as the detection target is determined to approach the user at the above speed or higher. It may be set as a target. When the corresponding image IM12 in FIG. 16 is included in the visual field range VR2 near the gazing point POR, the corresponding image IM12 may be displayed in the image display maximum area PN with the same font as the corresponding image IM11. This is because it is determined that the user is gazing at the corresponding image IM12 when the corresponding image IM12 that calls attention to the user is displayed near the gazing point POR in the image display maximum area PN. Is. In addition, as another embodiment, the wireless communication unit 132c does not acquire the image data that is the basis of the corresponding image from another server, but the HMD 100c has a storage unit that stores these image data as a database. Good. Further, the image setting unit 165c of the HMD 100c may acquire image data and the like from a database connected via the interface 180.

As described above, in the MD 100c of the modified example, the image setting unit 165c approaches the user wearing the image display unit 20c at a certain speed or higher, even if the target is not stored in advance as the detection target TW. In this case, the target is determined as the detection target TW, and the corresponding image IM12 such as a preset warning is displayed in the image display maximum area PN. Therefore, in the HMD 100c of the fourth embodiment, the user can be more appropriately notified of the danger, and the convenience of the user is improved.

E-5. Modification 5:
In this modification, as compared with the image setting unit 165c of the HMD 100c of the fourth embodiment, the image setting unit 165d of the modification changes the display position of the corresponding image according to the category associated with the image data. Is different. The image setting unit 165d weights the priorities to be displayed according to the category, and displays the display image based on the image data associated with the category with the high priority near the gazing point POR. The category associated with the image data corresponds to the type of image data in the claims.

FIG. 20 is an explanatory diagram of display positions of the corresponding images IM13 and IM14 in the visual field VR of the user in the modification. As shown in FIG. 20, the user wearing the image display unit 20d on his/her head visually recognizes the shopping street as the outside scene SC, the corresponding image IM14, and the corresponding image IM15. The user's gaze POR is looking at the signboard of the store "XX Pharmacy" in the shopping district. In the example shown in FIG. 20, the cameras 61 and 62 image the signboard of "0000 pharmacy" as the detection target TWd1 and the automobile as the detection target TWd2. The corresponding image IM13 is an image displayed in the image display maximum area PN in association with the detection target TWd1. The corresponding image IM13 is displayed in the upper center of the image display maximum area PN. The category associated with the image data of the corresponding image IM13 is “advertisement”. The corresponding image IM14 is an image displayed in the image display maximum area PN in association with the detection target TWd1. The corresponding image IM14 is displayed in the center of the image display maximum area PN. The category associated with the image data of the corresponding image IM14 is "warning".

The image setting unit 165d of the modified example, based on the category linked to the image data of the corresponding image, when a plurality of detection targets overlap at positions with different distances in the line-of-sight direction ED of the user, The display mode including the display position of the corresponding image is set so that the display areas of the corresponding images of No. In FIG. 20, the signboard of the detection target TWd1 and the automobile of the detection target TWd2 overlap in the vicinity of the same gazing point POR. In such a case, the image setting unit 165d displays the corresponding image IM14 whose category is "warning" in the central portion of the image display maximum area PN so that it is more visible than the corresponding image IM13 whose category is "advertisement". Let Further, the image setting unit 165d displays the corresponding image IM14 larger than the corresponding image IM13.

As described above, in the modified HMD 100d, the image setting unit 165d displays the corresponding image associated with the detection target in the image display maximum area PN in accordance with the category associated with the image data of the corresponding image. To set. Therefore, in the HMD 100d of the modified example, the image that the user wants to see more is preferentially displayed in the image display maximum area PN, and the usability for the user is improved.

E-6. Modification 6:
In the above embodiment, the operation unit 135 is formed in the control unit 10, but the mode of the operation unit 135 can be variously modified. For example, there may be a mode in which there is a user interface that is the operation unit 135 separately from the control unit 10. In this case, the operation unit 135 is a separate body from the control unit 10 in which the power source 130 and the like are formed, so that the operation unit 135 can be downsized and the operability for the user is improved. Further, although the cameras 61 and 62 are arranged in the image display unit 20, the camera 61 may be configured separately from the image display unit 20 and can capture the outside scene SC. Further, the HMD 100 in which the CPU 140 and the power source 130 included in the control unit 10 are all mounted on the image display unit 20 may be used. Since the HMD 100 does not have a controller that is separate from the image display unit 20, the HMD 100 can be further downsized. Further, the CPU 140 may be installed in each of the control unit 10 and the image display unit 20, so that the control unit 10 may be used as a single controller and the image display unit 20 may be used as a single display device.

For example, the image light generation unit may be configured to include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image light generation section may use an LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like instead of the LCD. Further, for example, the present invention can be applied to a laser retinal projection type HMD 100. Further, the image display maximum area PN may be configured by a MEMS shutter type display that opens and closes the MEMS shutter formed in each pixel.

Further, the HMD 100 may be a HMD 100 that employs a scanning optical system using a MEMS mirror as the image light generation unit and uses the MEMS display technology. As a specific configuration, in the HMD 100, a virtual image is formed by a signal light forming unit, a scanning optical system having a MEMS mirror that scans light emitted by the signal light forming unit, and light scanned by the scanning optical system. An optical member may be provided as the light emitting unit. When the HMD 100 has this configuration, the light emitted by the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided to the optical member, and reaches the virtual image forming surface (for example, a reflecting surface). When the MEMS mirror scans light, a virtual image is formed on the virtual image forming surface, and the user visually recognizes the formed virtual image, whereby the user recognizes the image.

Further, for example, the HMD 100 may be a head mounted display in which the optical image display unit covers only a part of the user's eye, in other words, the optical image display unit does not completely cover the user's eye. Further, the HMD 100 may be a so-called monocular type head mounted display. Further, instead of the HMD 100, a handheld display, which is not mounted on the head of the user and whose position is fixed manually by the user like binoculars, may be used as the image display device. Further, the HMD 100 is described as a binocular optical transmission type, but the present invention can be similarly applied to other types of head-mounted display devices such as a video transmission type.

Further, the HMD 100 may be used as a display device only for displaying an image based on an image signal received from another device. Specifically, it is used as a display device corresponding to a monitor of a desktop PC, and an image is displayed in the maximum image display area PN of the image display unit 20 by receiving an image signal from the desktop PC, for example. May be.

The HMD 100 may also be used to function as a part of the system. For example, the HMD 100 may be used as a device for executing a part of the functions of a system including an aircraft, and the system in which the HMD 100 is used is not limited to a system including an aircraft, and a system including an automobile, a bicycle, or the like. It may be.

The earphone may be of an ear-hook type or a headband type, and may be omitted. Further, for example, it may be configured as a head-mounted display device mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display device built in a body protector such as a helmet.

E-7. Modification 7:
The configuration of the HMD 100 in the above embodiment is merely an example, and can be variously modified. For example, the direction key 16 provided in the control unit 10 may be omitted, or in addition to the direction key 16 and the track pad 14, another operation interface such as an operation stick may be provided. Further, the control unit 10 may have a configuration capable of connecting an input device such as a keyboard and a mouse, and may accept an input from the keyboard and the mouse.

Further, as the image display unit, instead of the image display unit 20 worn like glasses, an image display unit of another system such as an image display unit worn like a hat may be adopted. Moreover, the earphones 32 and 34 can be appropriately omitted. Further, in the above-described embodiment, the LCD and the light source are used as the configuration for generating the image light, but instead of these, another display element such as an organic EL display may be adopted.

FIG. 21 is an explanatory diagram showing the external configuration of the HMD in the modification. In the case of the example of FIG. 21A, the difference from the HMD 100 shown in FIG. 1 is that the image display unit 20x includes a right optical image display unit 26x instead of the right optical image display unit 26, and a left optical image. That is, a left optical image display unit 28x is provided instead of the display unit 28. The right optical image display unit 26x is formed smaller than the optical member of the above-described embodiment, and is arranged diagonally above the right eye of the user when the HMD 100x is worn. Similarly, the left optical image display unit 28x is formed smaller than the optical member of the above embodiment, and is arranged diagonally above the left eye of the user when the HMD 100x is worn. In the case of the example of FIG. 21B, the difference from the HMD 100 shown in FIG. 1 is that the image display unit 20y includes a right optical image display unit 26y instead of the right optical image display unit 26, and a left optical image. In other words, a left optical image display unit 28y is provided instead of the display unit 28. The right optical image display unit 26y is formed smaller than the optical member of the above embodiment, and is arranged diagonally below the right eye of the user when the head mounted display is mounted. The left optical image display unit 28y is formed smaller than the optical member of the above-described embodiment, and is arranged diagonally below the left eye of the user when the head mounted display is mounted. As described above, it is sufficient if the optical image display unit is arranged near the eyes of the user. Further, the size of the optical member forming the optical image display unit is also arbitrary, the optical image display unit covers only a part of the user's eyes, in other words, the optical image display unit completely covers the user's eyes. It can be realized as the HMD 100 in an uncovered form.

Further, in the above embodiment, the HMD 100 may guide image light representing the same image to the left and right eyes of the user to allow the user to visually recognize a two-dimensional image, or the HMD 100 may display different images for the left and right eyes of the user. The image light may be guided so that the user can visually recognize the three-dimensional image.

Further, in the above embodiment, part of the configuration realized by hardware may be replaced with software, and conversely, part of the configuration realized by software may be replaced with hardware. Good. For example, in the above embodiment, the image processing unit 160 and the audio processing unit 170 are realized by the CPU 140 reading and executing a computer program, but these functional units may be realized by hardware circuits. Good.

When some or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the "computer-readable recording medium" is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but various internal RAMs, ROMs, etc. in a computer, a hard disk, etc. It also includes an external storage device fixed to the computer.

Further, in the above-described embodiment, as shown in FIGS. 1 and 2, the control unit 10 and the image display unit 20 are formed as separate configurations. However, regarding the configuration of the control unit 10 and the image display unit 20, However, the present invention is not limited to this, and various modifications are possible. For example, all or some of the components formed in the control unit 10 may be formed inside the image display unit 20. In addition, the power supply 130 in the above-described embodiment may be formed independently and replaceable, or the configuration formed in the control unit 10 may be duplicated in the image display unit 20. For example, the CPU 140 shown in FIG. 2 may be formed in both the control unit 10 and the image display unit 20, or the functions performed by the CPU 140 formed in the control unit 10 and the CPU formed in the image display unit 20. May be separately configured.

The present invention is not limited to the above-described embodiments and modified examples, and can be realized with various configurations without departing from the spirit of the invention. For example, the technical features in the embodiments and modified examples corresponding to the technical features in each mode described in the section of the summary of the invention are provided in order to solve some or all of the above-mentioned problems, or In order to achieve some or all of the effects, it is possible to appropriately replace or combine them. If the technical features are not described as essential in this specification, they can be deleted as appropriate.

10... Control part, 11... Enter key, 12... Illuminating part, 13... Display switching key, 14... Track pad, 15... Luminance switching key, 16... Direction key, 17... Menu key, 18... Power switch, 20... Image Display unit (image display unit), 21... right holding unit, 22... right display driving unit, 23... left holding unit, 24... left display driving unit, 26... right optical image display unit, 28... left optical image display unit, 30... Earphone plug, 32... Right earphone, 34... Left earphone, 37... Right eye imaging camera (eye imaging unit, line-of-sight direction specifying unit), 38... Left-eye imaging camera (eye imaging unit, line-of-sight direction specifying unit), 40 ... connection section, 42... right code, 44... left code, 46... connecting member, 48... body code, 51, 52... transmission section, 53, 54... reception section, 61... first camera (outside view imaging section), 62 ...Second camera (outside view image pickup section), 100...HMD (head mounted display device), 120...storage section, 130...power supply, 132...wireless communication section, 135...operation section, 140...CPU, 150...operating system , 160... Image processing unit, 165... Image setting unit (image changing unit), 166... Direction specifying unit (line-of-sight direction specifying unit), 168... Distance measuring unit, 170... Voice processing unit, 180... Interface, 190... Display control Section, 201... right BL control section, 202... left BL control section, 211... right LCD control section, 212... left LCD control section, 221... right backlight, 222... left backlight, 241... right LCD, 242... left LCD, 251... Right projection optical system, 252... Left projection optical system, 261... Right light guide plate, 262... Left light guide plate, AP... Tip part, BD1, BD2, BD3, BD4... Building, ED... User's line-of-sight direction , EL... Edge of left optical image display section, ER... Edge of right optical image display, HSync... Horizontal sync signal, VSync... Vertical sync signal, PCLK... Clock signal, IL... Illumination light, IM1, IM2, IM3 IM4, IM5, IM6, IM7, IM8, IM9, IM10, IM11, IM12, IM13, IM14... Corresponding image, LE... Left eye, RE... Right eye, MN... Month, MP... Left/right eyeball center, OA... External device, OB... object, PL... image light, PN... image display maximum area (display area), POR... gazing point, SC... outside scene, TW, TWa, TWb, TWd1, TWd2... detection target (specific object), US... use Person, VR... visual field, VR1, VR2, VR3, VR4, VR5, VR6, VR2S... visual field range, VRh... vertical visual field, VRw... horizontal visual field, OJ1, OJ2... visual target, RH... right hand, LH... left hand, HD... Handle, CA1, CA2... Moving object, EN... Entrance

Claims (17)

A head-mounted display device,
An image display unit capable of displaying an image in the display area and transmitting an outside scene ,
A line-of-sight direction specifying unit that specifies the line-of-sight direction of the user,
An image changing unit that changes the display image displayed in the display area according to the identified line-of-sight direction,
An outside scene image capturing section for capturing the outside scene,
Image data that is the source of the display image associated with a specific object, an image data acquisition unit that identifies and acquires for each type of the image data ,
The image changing unit detects a specific object from the imaged image of the outside scene when the outside scene image capturing unit picks up an image of the specific object, and displays the specific object in association with the detected position of the specific object. Based on the type of the image data acquired by the image data acquisition unit and the position of the imaged specific object with respect to the display area, the position of the image is set, and the image is associated with the specific object. A head-mounted display device for determining a mode of the display image . The head-mounted display device according to claim 1 , further comprising:
A moving object specifying unit that specifies a moving object that is moving in the outside scene,
The head-mounted display device, wherein the image changing unit sets a display mode of the display image associated with the specific object based on a speed at which the moving object approaches the image display unit. A head-mounted display device,
An image display unit capable of displaying an image in the display area and transmitting an outside scene ,
A line-of-sight direction specifying unit that specifies the line-of-sight direction of the user,
An image changing unit that changes the display image displayed in the display area according to the identified line-of-sight direction,
An outside scene image capturing section for capturing the outside scene,
A moving object specifying unit that specifies a moving object that is moving in the outside scene ,
The image changing unit detects a specific object from the captured image of the outside scene, sets the position of the display image in association with the position of the detected specific object, and the moving object displays the image. A head-mounted display device , wherein a display mode of the display image associated with the specific object is set based on the speed of approaching the part . The head-mounted display device according to claim 2 or 3 ,
When the moving object other than a predetermined area centered on the line-of-sight direction approaches the image display unit at a preset speed or higher, the image changing unit sets the moving object as the specific target object in advance. A head-mounted display device, which is determined to be the specific object even if not, and sets the display image set in advance to the determined specific object. The head-mounted display device according to any one of claims 1 to 4 ,
The line-of-sight direction specifying unit further specifies the distance to the visual target visually recognized by the user,
The image changing unit is associated with the specific target object based on the visual target and a motion characteristic based on a moving background that is in a range other than the visual target and that is moving with respect to the visual target. A head-mounted display device for setting a display mode of the display image. A head-mounted display device,
An image display unit capable of displaying an image in the display area and transmitting an outside scene ,
A line-of-sight direction specifying unit that specifies the line-of-sight direction of the user,
An image changing unit that changes the display image displayed in the display area according to the identified line-of-sight direction,
An outside scene image capturing section for capturing the outside scene ,
The line-of-sight direction specifying unit further specifies the distance to the visual recognition target visually recognized by the user,
The image changing unit detects a specific target object from the captured image of the outside scene, sets the position of the display image in association with the detected position of the specific target object, and the visual recognition target and the visual recognition target. A display mode of the display image associated with the specific target object is set based on a motion characteristic based on a moving background that is moving with respect to the visual target in a range other than the target, the head. Wearable display device. The head-mounted display device according to claim 5 or 6 , wherein
When the specific object is the moving background, the image changing unit is associated with the specific object as compared with the case where the specific object is not moving with respect to the visual target. A head-mounted display device for setting the display mode of the display image to a simple mode. The head-mounted display device according to any one of claims 1 to 7 ,
The head-mounted display device, wherein the image changing unit changes the display image in accordance with the visual characteristic of the user that changes according to the specified line-of-sight direction. The head-mounted display device according to claim 8 ,
The visual characteristic is a characteristic related to saccade,
The head-mounted display device, wherein the image changing unit sets a range including a viewing angle of a saccade as a visual field range of the user. The head-mounted display device according to any one of claims 1 to 9 ,
The head-mounted display device, wherein the image changing unit changes the display image in accordance with the position of the display image and the line-of-sight direction when the display image is displayed in a part of the display area. The head-mounted display device according to any one of claims 1 to 10 ,
The line-of-sight direction specifying unit has a display unit direction detection unit that detects the direction of the image display unit, and specifies the line-of-sight direction based on the detected direction of the image display unit, head-mounted display device .. The head-mounted display device according to any one of claims 1 to 11 ,
The head-mounted display device, wherein the line-of-sight direction identifying unit includes an eye image capturing unit that captures an image of the user's eye, and identifies the line-of-sight direction based on the captured image of the user's eye. The head-mounted display device according to any one of claims 1 to 12 ,
When the display image includes a character image, the image changing unit displays the character image as the display image in the display area in the first visual field range including the specified line-of-sight direction, The head-mounted display device, wherein the character image is not displayed as the display image in the display area in a second visual field range that is larger than the visual field range and does not include the first visual field range. The head-mounted display device according to any one of claims 1 to 13 ,
In the eighth visual field range which is larger than the seventh visual field range including the specified visual line direction and does not include the seventh visual field range, the image changing unit displays the display image in the seventh visual field range. A head-mounted display device, which is displayed in the display area as a simple image that is simpler than the image displayed in the visual field range. A computer program for a head-mounted display device,
An image display function that can display an image in the display area and that can see through the outside scene ,
A line-of-sight direction specifying function that specifies the line-of-sight direction of the user,
An image changing function for changing a display image displayed in the display area according to the identified line-of-sight direction,
An outside scene image capturing function for capturing the outside scene,
The image data that is the source of the display image associated with the specific object, an image data acquisition function of identifying and acquiring for each type of the image data, and realizing the head-mounted display device ,
The image changing function detects a specific object from the captured image of the external scene when the external scene imaging function images the specific object, and displays the specific object in association with the detected position of the specific object. Based on the type of the image data acquired by the image data acquisition function and the position of the imaged specific object with respect to the display area, the position of the image is set, and the image is associated with the specific object. A computer program that determines the aspect of the displayed image . A computer program for a head-mounted display device,
An image display function that can display an image in the display area and that can see through the outside scene ,
A line-of-sight direction specifying function that specifies the line-of-sight direction of the user,
An image changing function for changing a display image displayed in the display area according to the identified line-of-sight direction,
An outside scene image capturing function for capturing the outside scene,
A moving object identification function of identifying a moving object that is moving in the outside scene, and a head-mounted display device ,
The image changing function detects a specific object from the captured image of the outside scene, sets the position of the display image in association with the detected position of the specific object, and the moving object is the display area. A computer program that sets the display mode of the display image associated with the specific object based on the speed of approaching . A computer program for a head-mounted display device,
An image display function that can display an image in the display area and that can transmit the outside scene ,
A line-of-sight direction specifying function that specifies the line-of-sight direction of the user,
An image changing function for changing a display image displayed in the display area according to the identified line-of-sight direction,
The head mounted display device is realized with an outside scene image capturing function for capturing the outside scene ,
The line-of-sight direction specifying function further specifies the distance to the visual target visually recognized by the user,
The image changing function detects a specific object from the captured image of the outside scene, sets the position of the display image in association with the position of the detected specific object, and determines the visual target and the visual confirmation. A computer program that sets a display mode of the display image associated with the specific target object based on a motion characteristic based on a moving background that is moving with respect to the visual target in a range other than the target. ..

Images