JP2016161734A - Display device, method for controlling display device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display device that can effectively combine an environment visible for a user with display contents, a method for controlling display device, and a program.SOLUTION: A head-mounted display device 100 is to be used by mounting on a user's body; and includes an image display unit 20 that transmits outside scenery and displays an image to be visible together with the scenery, and a camera 61 that captures scenery in a visual line of the user. A control unit 140 of the head-mounted display device 100 includes: an image generation unit 177 that generates an image for display from the image captured by the camera 61; and an image display control unit 176 that controls the image display unit 20 to display the image for display generated by the image generation unit 177.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a display device, a display device control method, and a program.

  2. Description of the Related Art Conventionally, wearable display devices are known that perform display while being superimposed on an external field of view (see, for example, Patent Document 1). The device described in Patent Document 1 has a configuration in which characters are superimposed on an external field of view, and the display attributes such as font size and character color are changed for some characters or words to be displayed. Makes it easy to identify some characters.

JP 2014-56217 A

Patent Document 1 describes a display method when a user who uses a wearable display device sees either a character displayed on the display device or an external field of view. On the other hand, there has been no proposal for a display method that considers that the user visually recognizes with the outside world.
The present invention has been made in view of the above-described circumstances, and provides a display device, a display device control method, and a program capable of effectively combining an external world visible to a user and display contents. With the goal.

In order to achieve the above object, a display device according to the present invention is a display device that is used by being worn on a user's body, and displays an image so that the image can be seen through the outside scene and visible together with the outside scene. An imaging unit that captures the direction of the user's line of sight, an image generation unit that generates a display image from a captured image of the imaging unit, and a display image generated by the image generation unit is displayed on the display unit And an image display control unit.
ADVANTAGE OF THE INVENTION According to this invention, an image for a display based on the picked-up image which imaged the user's gaze direction can be displayed so that it may overlap with the target object seen as an outside scene. For this reason, the external scene outside the display device and the display contents can be effectively improved by supplementing the appearance of the external scene viewed by the user based on a captured image obtained by capturing the external scene, or presenting another view. In combination, it is possible to provide a new method for utilizing the display by the display device.

The display device may further include a detection unit that detects a use state of the display device, and the image display control unit may detect a display mode of the display image on the display unit. It adjusts corresponding to the use condition of the said display apparatus.
ADVANTAGE OF THE INVENTION According to this invention, aspects, such as a display size of an image displayed on a display part and a display position, can be adjusted corresponding to the use environment of a display apparatus, or a user's usage. For this reason, it is possible to perform display according to the way the user visually recognizes the outside scene.

In the display device according to the aspect of the invention, the detection unit may detect the position of the display device, and the image display control unit may display the display corresponding to the position of the display device detected by the detection unit. Adjusting the display mode of the image for use.
ADVANTAGE OF THE INVENTION According to this invention, based on the position of a display apparatus, it can display according to the appearance that a user visually recognizes an outside scene.

According to the present invention, in the display device, the image display control unit avoids a center of the display line of the display image corresponding to the position of the display device detected by the detection unit. In addition, at least one of a display position and a display size of the display image is adjusted.
ADVANTAGE OF THE INVENTION According to this invention, the image imaged with the imaging part can be displayed so that a user may not become the obstacle which looks at an outside scene.

The display device may further include an operation detection unit that detects an operation of the user based on a captured image of the imaging unit, and the image display control unit is detected by the operation detection unit. The display mode of the display image is adjusted in accordance with the operation.
According to the present invention, the user can adjust the display mode of the display image by performing an operation so that the imaging unit can capture an image.

The display device may further include a sound detection unit that detects sound, and the image display control unit adjusts a display mode of the display image according to a sound detection result of the sound detection unit. It is characterized by doing.
ADVANTAGE OF THE INVENTION According to this invention, the display mode of the image for a display can be adjusted because a user utters an audio | voice.

In the display device according to the present invention, the image generation unit generates the display image by extracting and processing a part of the captured image of the imaging unit.
According to the present invention, since a captured image obtained by capturing the user's line-of-sight direction is used for display, the user can be shown an image obtained by processing a part of the outside scene together with the outside scene.

In the display device according to the aspect of the invention, an operation detection unit that detects an operation of the user based on a captured image of the imaging unit may be provided, and the image generation unit may detect the operation from the captured image of the imaging unit. An area specified based on the user's action detected by the section is extracted and processed to generate the display image.
According to the present invention, a user can determine a range to be extracted from a captured image by performing an operation so that the imaging unit can capture an image.

In the display device according to the present invention, the image generation unit generates the display image by cutting out and enlarging a part of a captured image of the imaging unit.
According to the present invention, since a part of the captured image obtained by capturing the user's line-of-sight direction is enlarged and displayed, the user can view the line-of-sight direction while viewing the outside scene.

Further, the present invention is characterized in that, in the display device, the image generation unit generates the display image by cutting out a region corresponding to a preset condition in a captured image of the imaging unit. To do.
According to the present invention, a captured image corresponding to a preset condition can be displayed. For this reason, for example, when a captured image corresponding to the user's preference or interest is captured, the captured image can be displayed on the display unit and shown to the user.

The display device may further include a target detection unit that identifies a direction corresponding to the user's line-of-sight direction and detects an object in the user's line-of-sight direction from a captured image of the imaging unit. The image generation unit extracts a region including an image of the object detected by the target detection unit from a captured image of the imaging unit, processes the region, and generates the display image. .
According to the present invention, when there is an object in the user's line-of-sight direction, an image of the object can be shown to the user.

Moreover, the present invention is characterized in that, in the display device, the target detection unit detects the visual line direction of the user and detects the target object in the detected visual line direction of the user.
According to the present invention, a portion corresponding to a user's line-of-sight direction in a captured image can be used as a display image.

Moreover, the present invention is characterized in that, in the display device, the target detection unit detects a line-of-sight direction of an eye corresponding to a dominant eye among the eyes of the user.
According to the present invention, a portion corresponding to the line-of-sight direction of the user's dominant eye in the captured image can be used as a display image.

According to the present invention, in the display device, when the target detection unit detects a predetermined motion of the user by the motion detection unit, the target detection unit corresponds to the line-of-sight direction of the user based on the predetermined motion. It is characterized by specifying the direction to perform.
According to the present invention, a part to be extracted as a display image from a captured image can be designated by a user's operation.

In the display device according to the aspect of the invention, the target detection unit may detect the motion of the user based on the detected motion when the motion detection unit detects a motion of at least one of the user's limbs. A direction corresponding to the line-of-sight direction is specified.
According to the present invention, it is possible to specify a portion to be extracted as a display image from a captured image by the user moving the limbs.

The display device may further include a storage unit that stores the feature amount of the image of the object, and the target detection unit may convert the captured image of the imaging unit into a feature amount stored by the storage unit. The object is detected by detecting a suitable image.
According to the present invention, an object can be easily detected based on a captured image.

Further, according to the present invention, in the display device, the feature quantity stored in the storage unit is a feature quantity of an image of a specific object and a general-purpose feature quantity corresponding to a plurality of similar objects. Any one of these is included.
According to the present invention, a process for detecting a specific object or a process for detecting any one of a plurality of similar objects can be executed based on the stored feature amount.

The display device may further include a distance detection unit that detects a distance between the object detected by the target detection unit and the user, and the image display control unit includes the distance detection unit. The display image is generated by enlarging a part of the captured image of the imaging unit with an enlargement ratio corresponding to the detected distance.
ADVANTAGE OF THE INVENTION According to this invention, the image of a target object can be shown to a user with the expansion ratio matched with the distance with the target object of a user's gaze direction.

According to the present invention, in the display device, the image generation unit acquires a plurality of captured images captured when different by the imaging unit, and an area corresponding to a preset condition in the plurality of captured images. The display image is generated by specifying.
According to the present invention, more complicated conditions can be set for a plurality of captured images, and images can be shown to the user based on the captured images corresponding to the conditions.

Further, according to the present invention, in the display device, the image generation unit acquires a plurality of captured images captured at different times by the imaging unit, and a difference between the images of the plurality of captured images is preset. When the condition is met, the acquired captured image is cut out to generate the display image.
According to the present invention, since a display image can be generated and displayed based on a difference between a plurality of captured images, for example, a display image is generated based on a change in the image for a moving image including a plurality of continuous images. Can be displayed.

In order to achieve the above object, the present invention includes a display unit that transmits an outside scene and displays an image so as to be visible together with the outside scene, and controls a display device that is used by being worn on a user's body. A method of capturing a user's line-of-sight direction, cutting out a part of the captured image and enlarging the generated image for display, and displaying the generated display image on the display unit. And
ADVANTAGE OF THE INVENTION According to this invention, an image for a display based on the picked-up image which imaged the user's gaze direction can be displayed so that it may overlap with the target object seen as an outside scene. For this reason, the external scene outside the display device and the display contents can be effectively improved by supplementing the appearance of the external scene viewed by the user based on a captured image obtained by capturing the external scene, or presenting another view. In combination, it is possible to provide a new method for utilizing the display by the display device.

In order to achieve the above object, the present invention includes a display unit that transmits an outside scene and displays an image so that the outside scene can be viewed with the outside scene, and controls a display device that is used by being worn on the user's body. An image generation unit that generates a display image by cutting out and enlarging a part of a captured image obtained by capturing the user's line-of-sight direction. It is a program for functioning as an image display control unit that displays an image generated by a generation unit on the display unit.
ADVANTAGE OF THE INVENTION According to this invention, an image for a display based on the picked-up image which imaged the user's gaze direction can be displayed so that it may overlap with the target object seen as an outside scene. For this reason, the external scene outside the display device and the display contents can be effectively improved by supplementing the appearance of the external scene viewed by the user based on a captured image obtained by capturing the external scene, or presenting another view. In combination, it is possible to provide a new method for utilizing the display by the display device.

Explanatory drawing which shows the external appearance structure of the head mounted display apparatus of 1st Embodiment. The block diagram which shows the functional structure of a head mounted type display apparatus. Explanatory drawing which shows the example of the data memorize | stored in a memory | storage part. The flowchart which shows operation | movement of a head mounting | wearing type display apparatus. It is explanatory drawing which shows the typical example of application of a head-mounted display apparatus, (A) shows the example of a user's visual field, (B) and (C) show the example of the image cut out from a captured image , (D) shows an example of the visual field of the user when the display image is displayed. The flowchart which shows operation | movement of a head mounting | wearing type display apparatus. It is a figure which shows the example of operation | movement of a head mounting type display apparatus, (A) shows the example of the use environment of a head mounting type display apparatus, (B) and (C) show a display example. The block diagram which shows the functional structure of the head mounted display apparatus of 2nd Embodiment. It is a figure which shows the principal part structure of an image display part, (A) is the principal part perspective view which looked at the image display part from the user's head side, (B) is explanatory drawing of a gaze direction. The flowchart which shows operation | movement of a head mounting | wearing type display apparatus. The figure which shows the example of a display of a head mounting | wearing type display apparatus. The figure which shows the example of a display of a head mounting | wearing type display apparatus.

[First Embodiment]
Embodiments to which the present invention is applied will be described below.
FIG. 1 is an explanatory diagram showing an external configuration of a head-mounted display device 100 according to the first embodiment. The head-mounted display device 100 is a display device that is mounted on the head, and is also referred to as a head mounted display (HMD). The head-mounted display device 100 of the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. In this specification, a virtual image visually recognized by the user with the head-mounted display device 100 is also referred to as a “display image” for convenience. Moreover, emitting image light generated based on image data is also referred to as “displaying an image”.

  The head-mounted display device 100 includes an image display unit 20 that allows a user to visually recognize a virtual image when mounted on the user's head, and a control device 10 that controls the image display unit 20. The control device 10 also functions as a controller for the user to operate the head-mounted display device 100. The image display unit 20 is also simply referred to as a “display unit”.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. (Imaging unit) and a microphone 63. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. Hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”.

The display drive units 22 and 24 include liquid crystal displays 241 and 242 (Liquid Crystal Display, hereinafter also referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 (see FIG. 2). Details of the configuration of the display driving units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 2) and a light control plate 20A. The light guide plates 261 and 262 are formed of a light transmissive resin or the like, and guide the image light output from the display driving units 22 and 24 to the user's eyes. The dimming plate 20A is a thin plate-like optical element, and is arranged so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. As the light control plate 20A, various ones such as one having almost no light transmission, one having near transparency, one having a light amount attenuated to transmit light, one having a specific wavelength attenuated or reflected, and the like are used. Can do. By appropriately selecting the optical characteristics (light transmittance, etc.) of the light control plate 20A, the external light amount incident on the right optical image display unit 26 and the left optical image display unit 28 from the outside is adjusted to visually recognize the virtual image. You can adjust the ease. In the present embodiment, a case will be described in which at least a light control plate 20A having such a light transmittance that a user wearing the head-mounted display device 100 can visually recognize an outside scene is used. The light control plate 20A protects the right light guide plate 261 and the left light guide plate 262, and suppresses damage to the right light guide plate 261 and the left light guide plate 262, adhesion of dirt, and the like.
The light control plate 20A may be detachable with respect to the right optical image display unit 26 and the left optical image display unit 28, and a plurality of types of light control plates 20A may be exchanged and may be omitted.

The camera 61 is disposed at the end ER that is the other end of the right optical image display unit 26. The camera 61 captures an outside scene that is an external scenery in a direction opposite to the user's eye side, and acquires an outside scene image. The camera 61 of this embodiment shown in FIG. 1 is a monocular camera, but may be a stereo camera.
The shooting direction of the camera 61, that is, the angle of view, is taken from the front side direction of the head-mounted display device 100, in other words, at least a part of the outside scene in the user's viewing direction when the head-mounted display device 100 is mounted. Direction. In addition, although the angle of view of the camera 61 can be set as appropriate, the imaging range of the camera 61 includes the outside world (outside scene) that the user visually recognizes through the right optical image display unit 26 and the left optical image display unit 28. A range is preferable. Furthermore, it is more preferable that the imaging range of the camera 61 is set so that the entire field of view of the user through the light control plate 20A can be captured.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control device 10. The connection unit 40 includes a main body cord 48 connected to the control device 10, a right cord 42, a left cord 44, and a connecting member 46. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24.

  The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. A microphone 63 is provided in the vicinity of the earphone plug 30. The earphone plug 30 to the microphone 63 are combined into one cord, and the cord branches from the microphone 63 and is connected to each of the right earphone 32 and the left earphone 34.

  The specific specifications of the microphone 63 are arbitrary, and may be a directional microphone or an omnidirectional microphone. Examples of directional microphones include unidirectionality (Cardioid), narrow directivity (Super cardioid), sharp directivity (Hypercardioid), and superdirectivity (Ultra Cardioid). In the case where the microphone 63 has directivity, the sound from the line of sight of the user wearing the head-mounted display device 100 may be particularly well collected and detected. In this case, in order to ensure the directivity of the microphone 63, the microphone 63 or a part that accommodates the microphone 63 may have a structural feature. For example, in the example of FIG. 1, the microphone 63 and the connecting member 46 are designed so that the sound collection unit of the microphone 63 faces the user's line-of-sight direction with the user wearing the right earphone 32 and the left earphone 34. It only has to be. Alternatively, the microphone 63 may be embedded in the right holding unit 21 or the left holding unit 23. In this case, if a sound collecting hole is formed in the front side of the right holding unit 21 or the left holding unit 23, that is, the surface aligned with the right optical image display unit 26 and the left optical image display unit 28, the direction of the user's line of sight The directivity corresponding to can be given. The line-of-sight direction of the user is, for example, the direction that the right optical image display unit 26 and the left optical image display unit 28 face, and the center of the field of view that the user sees through the right optical image display unit 26 and the left optical image display unit 28. In other words, the shooting direction of the camera 61, and the like. Note that the direction of directivity of the microphone 63 may be variable. In this case, it may be configured to detect the user's line-of-sight direction and adjust the line-of-sight direction of the microphone 63 so as to face the direction.

  It is also possible to combine the right code 42 and the left code 44 into one code. Specifically, the lead wire inside the right cord 42 is drawn into the left holding portion 23 side through the inside of the main body of the image display unit 20, and is covered with a resin together with the lead wire inside the left cord 44 to be combined into one cord. Also good.

  The image display unit 20 and the control device 10 transmit various signals via the connection unit 40. A connector (not shown) that fits each other is provided at each of the end of the main body cord 48 opposite to the connecting member 46 and the control device 10. By fitting / releasing the connector of the main body cord 48 and the connector of the control device 10, the control device 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control device 10 is a device for controlling the head-mounted display device 100. The control device 10 includes switches including an enter key 11, a lighting unit 12, a display switching key 13, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. In addition, the control device 10 includes a track pad 14 that is touched by a user with a finger.

  The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control device 10. The lighting unit 12 notifies the operation state of the head-mounted display device 100 by its light emission state. Examples of the operating state of the head-mounted display device 100 include power ON / OFF. As the lighting unit 12, for example, an LED (Light Emitting Diode) is used. The display switching key 13 detects a pressing operation and outputs a signal for switching the display mode of the content moving image between 3D and 2D, for example.

  The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the head-mounted display device 100 by detecting a slide operation of the switch.

FIG. 2 is a functional block diagram of each unit constituting the display system 1 according to the embodiment.
As shown in FIG. 2, the display system 1 includes an external device OA and a head-mounted display device 100. Examples of the external device OA include a personal computer (PC), a mobile phone terminal, and a game terminal. The external device OA is used as an image supply device that supplies an image to the head-mounted display device 100.

The control device 10 of the head-mounted display device 100 includes a control unit 140, an operation unit 135, an input information acquisition unit 110, a storage unit 120, a power supply 130, an interface 180, a transmission unit (Tx) 51, and And a transmission unit (Tx) 52.
The operation unit 135 detects an operation by the user. The operation unit 135 includes the determination key 11, the display switching key 13, the track pad 14, the luminance switching key 15, the direction key 16, the menu key 17, and the power switch 18 shown in FIG.

The input information acquisition unit 110 acquires a signal corresponding to an operation input by the user. As a signal corresponding to the operation input, for example, there is an operation input to the track pad 14, the direction key 16, and the power switch 18.
The power supply 130 supplies power to each part of the head-mounted display device 100. As the power supply 130, for example, a secondary battery can be used.

The storage unit 120 stores various computer programs. The storage unit 120 is configured by a ROM, a RAM, or the like. The storage unit 120 may store image data to be displayed on the image display unit 20 of the head-mounted display device 100.
The storage unit 120 stores detection feature data 124 referred to by a target detection unit 171 described later, and replacement image data 125 processed by the image display control unit 176.

  The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control device 10. As the interface 180, for example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, or a memory card interface can be used.

The control unit 140 reads out and executes the computer program stored in the storage unit 120, thereby realizing the function of each unit. That is, the control unit 140 includes an operating system (OS) 150, an image processing unit 160, an audio processing unit 170, a target detection unit 171, a distance detection unit 173, an image display control unit 176, an image generation unit 177, and a display control unit. It functions as 190.
A three-axis sensor 113, a GPS 115, and a communication unit 117 are connected to the control unit 140. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 can acquire the detection value of the triaxial sensor 113. The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and obtains the current position of the control device 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. Further, the GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140 of the control device 10.

The communication unit 117 executes wireless data communication conforming to a standard such as a wireless LAN (WiFi (registered trademark)), Miracast (registered trademark), or Bluetooth (registered trademark).
When the external device OA is wirelessly connected to the communication unit 117, the control unit 140 acquires content data from the communication unit 117 and performs control for displaying an image on the image display unit 20. On the other hand, when the external device OA is wired to the interface 180, the control unit 140 acquires content data from the interface 180 and performs control for displaying an image on the image display unit 20. Therefore, the communication unit 117 and the interface 180 are hereinafter collectively referred to as a data acquisition unit DA.
The data acquisition unit DA acquires content data to be displayed by the head-mounted display device 100 from the external device OA. The content data includes image data to be described later.

The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). Thereafter, the image processing unit 160 stores the converted digital image signal as image data (Data in the figure) of the target image in the DRAM in the storage unit 120 for each frame. This image data is, for example, RGB data.
Note that the image processing unit 160 may execute image processing such as various tone correction processing such as resolution conversion processing, brightness and saturation adjustment, and keystone correction processing on the image data as necessary. .

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. To do. The image data Data transmitted via the transmission unit 51 is also referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left eye image data”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control device 10 and the image display unit 20.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 controls driving of the right LCD 241 by the right LCD control unit 211, driving ON / OFF of the right backlight 221 by the right backlight control unit 201, and left LCD control unit according to control signals. The left LCD 242 driving ON / OFF by 212, the left backlight 222 driving ON / OFF by the left backlight control unit 202, and the like are individually controlled. Thus, the display control unit 190 controls the generation and emission of image light by the right display driving unit 22 and the left display driving unit 24, respectively. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light.

  The display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a camera 61. And a vibration sensor 65 and a nine-axis sensor 66.

  The vibration sensor 65 is configured using an acceleration sensor, and is arranged inside the image display unit 20 as shown in FIG. In the example of FIG. 1, the right holding unit 21 is built in the vicinity of the end ER of the right optical image display unit 26. When the user performs an operation of hitting the end ER (knock operation), the vibration sensor 65 detects vibration caused by this operation and outputs the detection result to the control unit 140. Based on the detection result of the vibration sensor 65, the control unit 140 detects a knocking operation by the user.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, the movement of the user's head is detected. Since the orientation of the image display unit 20 is known from the detected movement of the user's head, the control unit 140 can estimate the user's line-of-sight direction.

  The right display driving unit 22 includes a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, a right LCD control unit 211 that functions as a display element, and a right An LCD 241 and a right projection optical system 251 are provided. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as display elements. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control device 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data1 input through the reception unit 53. . The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The right projection optical system 251 and the right light guide plate 261 are collectively referred to as “light guide unit”.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, a left LCD control unit 212 and a left that function as a display element. An LCD 242 and a left projection optical system 252 are provided. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as display elements. The left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are also collectively referred to as “image light generation unit”. The left projection optical system 252 is configured by a collimating lens that converts the image light emitted from the left LCD 242 into a light beam in a parallel state. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the image light along a predetermined optical path. The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “light guide unit”.

The head-mounted display device 100 displays an image based on image data so as to overlap the outside scene when the user views the outside scene through the right optical image display unit 26 and the left optical image display unit 28. To do.
The target detection unit 171 performs control for causing the camera 61 to perform imaging, and acquires a captured image. The captured image is output from the camera 61 as color image data or monochrome image data. The camera 61 outputs an image signal, and the target detection unit 171 generates image data that conforms to a predetermined file format from the image signal. May be.
The object detection unit 171 analyzes the acquired captured image data and detects an object reflected in the captured image data. The target object is an object or a person that exists in the imaging direction of the camera 61, that is, the user's line-of-sight direction.

The target detection unit 171 searches the captured image for an image that matches the detection feature data 124 and detects the matching image as an image of the target object.
FIG. 3 is a diagram illustrating a configuration example of the detection feature data 124 and the replacement image data 125 stored in the storage unit 120.
The detected feature data 124 is data of the feature amount of the image detected from the captured image. In the present embodiment, the detected feature data 124 includes single image feature data 124a and moving image feature data 124b. The single image feature data 124a is feature amount data applied to one captured image. When the single image feature data 124a is used, the target detection unit 171 performs matching processing for detecting a portion that matches the single image feature data 124a for one captured image.

  The moving image feature data 124b is feature amount data applied to a plurality of captured images. When the moving image feature data 124b is used, the target detection unit 171 takes a captured image of the camera 61 a plurality of times, and obtains a difference between the captured plurality of captured images. The camera 61 can capture images at predetermined time intervals, and can output captured image data every time it captures images. That is, the camera 61 functions as a moving image camera that captures images at a frame rate corresponding to the time interval. In this case, the target detection unit 171 can obtain a change in the moving image by obtaining a difference between a plurality of captured images captured by the camera 61 at different timings. The moving image feature data 124b can be referred to as feature amount data applied to changes in moving images. The target detection unit 171 performs matching processing for detecting a portion that matches the moving image feature data 124b in the difference between the captured images.

  In the present embodiment, whether the detection feature data 124 used by the target detection unit 171 is the single image feature data 124a or the moving image feature data 124b is set in advance. When the target detection unit 171 is set to use the single image feature data 124a, the target detection unit 171 acquires one captured image of the camera 61 and performs matching processing. On the other hand, when the moving image feature data 124b is set to be used, the target detection unit 171 temporarily stores the captured image of the camera 61 in the storage unit 120 every time it acquires the captured image. Then, the target detection unit 171 obtains a plurality of captured images stored in the storage unit 120, obtains a difference, and performs a matching process based on the moving image feature data 124b for the difference.

  The object detection unit 171 identifies an object that appears in the captured image captured by the camera 61 through matching processing. That is, when the target detection unit 171 detects a region that matches the single image feature data 124a in the captured image or a region that matches the moving image feature data 124b in the difference between the captured images, the target detection unit 171 reflects the detected region. The subject is the target.

The distance detection unit 173 obtains the distance to the object detected by the object detection unit 171. For example, the distance detection unit 173 obtains the distance to the object based on the size of the image of the object detected by the object detection unit 171 in the captured image of the camera 61.
The head-mounted display device 100 may include a distance meter that detects the distance to the object using laser light or ultrasonic waves. For example, the distance meter includes a light source of laser light and a light receiving unit that receives reflected light of the laser light emitted from the light source, and detects a distance to an object based on a light receiving state of the laser light. The distance meter may be an ultrasonic distance meter, for example. That is, a distance meter that includes a sound source that emits ultrasonic waves and a detection unit that detects ultrasonic waves reflected on the object, and detects the distance to the object based on the reflected ultrasonic waves may be used. Further, the distance meter can be configured by combining a distance meter using laser light and a distance meter using ultrasonic waves. Such a distance meter is preferably provided in the right holding unit 21 or the right display driving unit 22 of the image display unit 20, and may be installed, for example, facing the front side on the surface aligned with the light control plate 20A. The direction in which the distance meter measures the distance is preferably the user's line-of-sight direction, similar to the imaging direction of the camera 61.
The distance detection unit 173 detects the distance from the camera 61 or the distance meter to the object, and this distance can be regarded as the distance from the user of the head-mounted display device 100 to the object.
When the target detected by the target detection unit 171 is not a specific object, the distance detection unit 173 detects the distance to this target, for example, using the center of the captured image as a virtual target.

The image generation unit 177 generates a display image by extracting and processing a part of the captured image of the camera 61. In the present embodiment, the image generation unit 177 generates a display image by cutting out an area of a predetermined size including the feature detected by the target detection unit 171 from the captured image. When the object detection unit 171 detects an object from a plurality of captured images using the moving image feature data 124b, the image generation unit 177 includes the object detected by the object detection unit 171 in the latest captured image. Cut out the area.
The display image generated by the image generation unit 177 is displayed in the displayable area of the image display unit 20 under the control of the image display control unit 176. The display size of the display image is set in advance, and is, for example, a level that does not hinder the user when viewing the outside scene. Therefore, the image generation unit 177 cuts out the captured image with a size that matches the display size of the display image.

The image generation unit 177 generates a display image so that a part of the captured image including the feature detected by the target detection unit 171 is displayed more easily. Therefore, as described above, the process executed by the image generation unit 177 is not limited to the process of enlarging the area extracted from the captured image. For example, a display image may be obtained by surrounding the object in a region extracted from the captured image with a frame such as a circle or a rectangle. In addition, for example, a decoration image such as a balloon or an arrow may be given to the object to generate a display image that can easily recognize the object. The decoration in this case is an image stored in advance for decoration, and can be called an auxiliary image or an enhancement image. Further, the luminance and color tone of the region including the object may be changed. For example, the object and its surroundings may be a high-luminance image.
Further, when a decoration is given to an object detected based on the single image feature data 124a or the moving image feature data 124b, the data related to the decoration is used as the single image feature data 124a and the moving image feature data 124b. May be included.

  Further, the image generation unit 177 may cut out the target object and its surrounding images when cutting out an image including the target object detected by the target detection unit 171. In this case, it can be clearly seen that the image is an enlarged image of a part of the captured image, so that the user can enlarge and view a portion that is difficult to see, for example. In addition, the image generation unit 177 may cut out the image along the contour of the image of the object. In this case, the image to be cut out is combined with another background image and displayed, or the cut out image is displayed in 3D ( It is possible to display the image on the image display unit 20 as a (stereo) image.

  Furthermore, the image generation unit 177 determines the enlargement ratio based on the distance detected by the distance detection unit 173. The image generation unit 177 makes the user see an enlarged image when a part of the image cut out from the captured image is displayed under the control of the image display control unit 176. Therefore, the image generation unit 177 determines the size of the image to be cut out from the captured image based on the size (resolution or the number of image dots) for displaying the display image in the displayable area of the image display unit 20 and the enlargement ratio. To do. Then, the image generation unit 177 executes enlargement processing (or resolution conversion processing) for converting the size of the cut-out image according to the display size. Thereby, since the image cut out from the captured image is in an enlarged state, the user can see the enlarged image.

  The size at which the display image enlarged by the image generation unit 177 is visually recognized by the user is affected by the specifications of the image display unit 20. In order to appropriately manage and adjust the enlargement ratio at which the image generation unit 177 enlarges the image cut out from the captured image, for example, when the image cut out from the captured image has the same size as the outside scene in the user's field of view (that is, It is preferable to set the size of an image with an enlargement ratio of 1) in advance. That is, the size of the image cut out by the image generation unit 177 when the user visually recognizes the display image with a size that is neither enlarged nor reduced is set in advance. With this size as a reference, the image generation unit 177 cuts out and enlarges the image in accordance with the target enlargement ratio.

The enlargement ratio in the process of enlarging the image by the image generation unit 177 may be determined according to the distance detected by the distance detection unit 173, for example.
FIG. 3 shows an example in which the display setting data 126 stored in the storage unit 120 includes enlargement ratio setting data 126 a that associates the distance to the object detected by the distance detection unit 173 and the enlargement ratio. The enlargement ratio setting data 126a is data for determining the distance and the enlargement ratio detected by the distance detection unit 173. For example, a table for obtaining the enlargement ratio step by step, or an arithmetic expression or parameter for calculating the enlargement ratio from the distance. Including. Based on the enlargement factor setting data 126a, the image generation unit 177 can obtain the enlargement factor from the distance detected by the distance detector 173. Further, the case where the image generation unit 177 does not enlarge may be set in the enlargement ratio setting data 126a, or the case where the image generation unit 177 does not generate a display image may be set. For example, when the distance to the object detected by the distance detection unit 173 is closer than the set distance, it can be set not to generate and display the display image. When the image generation unit 177 does not generate and output display image data, the image display control unit 176 does not display the display image.
The display setting data 126 includes, in addition to the enlargement ratio setting data 126a, data set in advance with respect to image display, such as setting value data relating to the default display size in the image display unit 20, and setting value data relating to display color adjustment. May be included.

  The image display control unit 176 controls the display control unit 190 to cause the image display unit 20 to display an image based on the display image data generated by the image generation unit 177.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker in the left earphone 34 (not shown) connected to the connecting member 46. (Not shown). For example, when the Dolby (registered trademark) system is adopted, processing is performed on the audio signal, and different sounds with different frequencies or the like are output from each of the right earphone 32 and the left earphone 34, for example.

  The sound processing unit 170 acquires sound collected by the microphone 63, converts it into digital sound data, and performs processing related to sound. For example, the speech processing unit 170 performs speaker recognition that identifies a person who speaks for each voice by recognizing voices of a plurality of persons separately by extracting and modeling features from the acquired speech. May be.

A three-axis sensor 113, a GPS 115, and a communication unit 117 are connected to the control unit 140. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 can detect the detection value of the triaxial sensor 113 and detect the movement and the direction of movement of the control device 10.
The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and obtains the current position of the control device 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. Further, the GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140 of the control device 10.
The communication unit 117 performs wireless data communication conforming to a wireless LAN (WiFi (registered trademark)) or Bluetooth (registered trademark) standard.

The interface 180 is an interface for connecting various image supply apparatuses OA that are content supply sources to the control apparatus 10. The content supplied by the image supply device OA includes a still image or a moving image, and may include sound. Examples of the image supply device OA include a personal computer (PC), a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.
Here, it is also possible to connect the image supply device OA to the control device 10 by a wireless communication line. In this case, the image supply device OA performs wireless communication with the communication unit 117 and transmits content data using a wireless communication technique such as Miracast (registered trademark).

FIG. 4 is a flowchart showing the operation of the head-mounted display device 100. In this operation, when the user views the outside scene through the right optical image display unit 26 and the left optical image display unit 28, a part of the captured image obtained by capturing the outside scene is displayed by the image display unit 20.
FIG. 5 is an explanatory diagram showing how the head-mounted display device 100 displays an image by the operation shown in FIG. 5A shows an example of the user's visual field VR, FIGS. 5B and 5C show examples of images cut out from the captured image, and FIG. 5D shows a case where a display image is displayed. An example of the visual field VR of the user is shown.

  In the control unit 140 of the head-mounted display device 100, the target detection unit 171 causes the camera 61 to perform imaging (step S11) and obtains a captured image (step S12). The target detection unit 171 acquires the single image feature data 124a or the moving image feature data 124b from the storage unit 120 (step S13), and the acquired detection feature data target is a single captured image or a plurality of captured images. Is determined (step S14). This determination can be made based on whether the single image feature data 124a or the moving image feature data 124b is designated as the detection feature data to be used.

When processing a single captured image (step S14; YES), the target detection unit 171 performs matching processing on the captured image acquired in step S12 using the single image feature data 124a (step S15). ), The process proceeds to step S18.
When processing a plurality of captured images (step S14; NO), the target detection unit 171 includes the captured image acquired in step S12 and the captured image acquired in the past and temporarily stored in the storage unit 120. The difference is obtained (step S16). The target detection unit 171 executes matching processing based on the moving image feature data 124b for the obtained difference (step S17), and proceeds to step S18.

  In step S18, the target detection unit 171 determines an area to be cut out from the captured image. The region determined here is a region including the object detected by the object detection unit 171, but the detailed cutout range is determined by the image generation unit 177 in step S <b> 21 described later. Therefore, the region determined in step S18 is a temporary region, and may be, for example, the center position of the region to be cut out or the contour of the object.

  The example shown in FIG. 5A shows an example where the user is watching a soccer game from the audience seat. In this example, a soccer field F and a plurality of players FP playing in the field F are visible. It can be seen that one player FP has the ball B. In this example, since the user is far away from the field F, the player P and the ball B in the field of view VR appear small.

The imaging range (view angle) of the camera 61 is, for example, the same range as the visual field VR, and the field F, the player FP, and the ball B are captured in the captured image.
The object detection unit 171 performs matching processing on the captured image based on the single image feature data 124a, and detects the image of the ball B as the object. In this case, the target detection unit 171 detects the ball B in step S15, and in step S18, an area including the ball B is determined as a cut-out area.

Subsequently, the distance detection unit 173 detects the distance to the target object included in the region determined by the target detection unit 171 in step S18 (step S19).
Then, the image generation unit 177 determines the enlargement rate based on the distance detected by the distance detection unit 173 and the enlargement rate setting data 126a stored in the storage unit 120 (step S20). Based on the determined enlargement factor, the image generation unit 177 determines the size and position of the image to be cut out from the captured image, cuts out the image, and outputs the cut out image as a display image (step S21). In step S21, the image generation unit 177 may perform the above-described size conversion or resolution conversion processing. The image generation unit 177 does not output a display image when the setting of the enlargement ratio setting data 126a corresponding to the distance detected by the distance detection unit 173 is a setting for not displaying.

  5B and 5C show examples of display images output from the image generation unit 177. FIG. The image generation unit 177 cuts out an area including the image of the ball B from the captured image in accordance with contents set in advance with respect to the cutout of the image. The contents set in advance for image cropping are the aspect ratio of the clipped image and the size of the image when the enlargement ratio is 1. In the example of FIG. 5B, an area in which the whole body of the ball B and the player FP having the ball B is shown is cut out and output as a display image. In the example of FIG. 5C, the ball B and a region of a predetermined size including the ball B are cut out and output as a display image. In the example of FIG. 5C, only part of the body of the player FP having the ball B is included. When the single image feature data 124a is feature amount data for detecting the ball B, whether the whole body of the player FP or a part of the body is included in the image cut out by the image generation unit 177 is: It depends on the setting such as magnification.

  The image display control unit 176 stores the display image output from the image generation unit 177 in the storage unit 120 (step S22). The storage unit 120 stores display images in time series. Therefore, when the process shown in steps S11 to S24 in FIG. 4 is repeatedly executed, a plurality of display images are stored in the storage unit 120 in chronological order. The plurality of display images can be reproduced as moving images by displaying them continuously. For example, in accordance with an instruction operation on the operation unit 135, the control unit 140 can reproduce and display the display image stored in the storage unit 120.

The image display control unit 176 outputs the display image output from the image generation unit 177 in step S21 to the display control unit 190 and displays it on the image display unit 20 (step S23).
Thereafter, the control unit 140 determines whether or not to end the display (step S24). When the display is continued (step S24; NO), the control unit 140 returns to step S1. When the display is terminated according to the operation detected by the operation unit 135 (step S24; YES), the control unit 140 stops the display by the display control unit 190 and ends the present process.

  FIG. 5D shows a state in which a display image is displayed by the image display unit 20. In the field of view VR, as in FIG. 5A, the field F is seen as an outside scene, but the image IV is displayed so as to be superimposed on a part of the field of view VR. This image IV is a display image displayed by the image display unit 20. Since the image IV is an image obtained by enlarging a part of the outside scene, the user can view the outside scene viewed through the image display unit 20 and a part of the outside scene at the same time.

  In addition, the object detection part 171 may perform the process which tracks the target once detected, when performing the process of FIG.4 S11-S17 repeatedly. That is, the target detection unit 171 detects a target object based on the single image feature data 124a or the moving image feature data 124b, and then monitors the change in the captured image of the camera 61 to detect the movement of the detected target object. Chase. Thereby, the target detection unit 171 can detect the target object from the captured image of the detection target without performing the matching processing in steps S15 and S17. Such a method is, for example, processing load when a moving image is captured by the camera 61 at a predetermined frame rate (for example, 30 frames / second), and the object detection unit 171 detects an object for each frame. Can be mitigated and effective.

  Further, the single image feature data 124a and the moving image feature data 124b are not limited to feature amounts for detecting a specific object. For example, the single image feature data 124a can be a feature amount for detecting a captured image including a specific feature, and the moving image feature data 124b is a feature amount for detecting a moving image having a specific change. can do. In this case, when a feature that matches the single image feature data 124a or the moving image feature data 124b is detected in the captured image, the target detection unit 171 sets a portion having the feature as a target. In this case, the target is not a specific object but a part of the captured image. The image generation unit 177 may cut out a part of the captured image detected as the target by the target detection unit 171 and use it as a display image.

The single image feature data 124a and the moving image feature data 124b may include feature amount data for a specific object for detecting one specific object or a small number of objects, or a plurality of similar types of data. General-purpose feature amount data that can detect the target object may be included.
For example, in the example of FIGS. 5B and 5C, the target detection unit 171 detects the ball based on the single image feature data 124a. The single image feature data 124a may be feature amount data for a specific object for detecting a ball of a specific design (including a color and a shape of the design), or a general-purpose data for detecting a sphere. It may be feature amount data. In the case of general-purpose feature amount data, for example, a feature amount that identifies only a shape, a feature amount that identifies a shape and a color, or a feature amount that identifies one of a plurality of colors It may be. In addition to the feature amount data, the single image feature data 124a may include data defining a range of difference from the allowable feature amount. The same applies to the moving image feature data 124b.
Specifically, if the data of general-purpose features that allow for differences in a specific person's face, hairstyle, clothes (uniforms, shoes, hats, bibs, etc.), not only can multiple objects be detected, It is possible to cope with changes in color in captured images due to sunlight, lighting conditions, environmental brightness, and sunlight. Further, the target detection unit 171 may track the target in the captured image data that is subsequently captured after the target is detected. In this case, when the object image in the captured image data no longer corresponds to the feature amount while tracking the target object, the feature amount may be calibrated (calibrated). As a result, when using feature amount data for a specific object and when using general-purpose feature amount data, the color and shape of the object in the captured image can be determined from the feature amount due to factors such as sunlight. Even if the vehicle deviates, the object can be detected and the object can be detected continuously.

  FIG. 6 is a flowchart showing the operation of the head-mounted display device 100, and shows the operation of adjusting the display in accordance with the position of the user wearing the head-mounted display device 100. 7 is an explanatory view of the same operation as FIG. 6, FIG. 7A shows an example of the usage environment of the head-mounted display device, and FIG. 7B and FIG. A display example is shown.

The operation in FIG. 6 is processing for adjusting the position and size at which the image display unit 20 displays the image IV in accordance with the position of the user. As described above, the head-mounted display device 100 is a device that allows the user to view the captured image of the camera 61 with the image display unit 20 while looking at the outside scene. The direction of the outside scene that the user sees through the image display unit 20 is strongly influenced by the relative position between the user and the object that the user sees.
For example, as shown in FIG. 7A, a user wears and uses the head-mounted display device 100 in a stadium ST having a spectator seat S of multiple levels (four levels in FIG. 7A). Think about the case. When the user is on an upper floor such as the fourth floor or the third floor, the user looks down to see the field F. Therefore, the direction that the user wants to see is below the display areas of the right light guide plate 261 and the left light guide plate 262. In this case, if the display position of the display image by the image display control unit 176 is set, for example, in the upper part of the display area, avoiding the direction that the user wants to see, the user will not be disturbed. The example shown in FIG. 7B is an example in which the user is looking at the player FP on the field F from the upper floor, and the image display control unit 176 displays the image IV at the upper part of the display area. In this case, in the user's visual field VR, the player FP can be seen without being disturbed by the image IV. 7C is an example in which the user is looking at the player FP on the field F from the lower floor, and the image display control unit 176 displays the image IV at the lower part of the display area. In this case, since the user's line of sight is close to the front, the image display control unit 176 displays a small image IV at a position avoiding the front. Therefore, the player FP can be seen in the visual field VR of the user without being disturbed by the image IV.

The operation in FIG. 6 can be executed at an arbitrary timing when a preset condition is satisfied after the display process is started in step S23 in FIG. 4 or when the user operates the control device 10.
The image display control unit 176 determines a relative position between the object included in the outside scene viewed by the user and the head-mounted display device 100 (step S31). The position of the head-mounted display device 100 may be determined based on the position detected by the GPS 115, for example. Alternatively, the distance detection unit 173 may make the determination based on the distance to the object detected from the captured image data of the camera 61. Furthermore, in a state where a position specifying system using WiFi (registered trademark) is operating in the environment in which the head-mounted display device 100 is used, the position may be specified by the communication unit 117 performing WiFi communication. Alternatively, when the position of the head-mounted display device 100 is designated by an input operation on the operation unit 135, the designated position may be acquired.

  The image display control unit 176 refers to the display setting data 126 stored in the storage unit 120, and acquires the setting related to the display corresponding to the position determined in step S31 (step S32). This setting is a display position of the image IV, a specified value or a maximum value of the display size, and the like. The image display control unit 176 adjusts the display size and display position of the image IV according to the setting acquired in step S32 (step S33), and determines whether or not to end the display (step S34).

  While the display operation is not ended (step S34; NO), the control unit 140 repeatedly executes the processes of steps S31 to S33, and ends the present process together with the end of display (step S34; YES).

  As described above, the head-mounted display device 100 according to the first embodiment to which the present invention is applied is a display device that is used while being worn on the user's body, and transmits the outside scene and the outside scene. In addition, an image display unit 20 that displays an image so that the user can view the image and a camera 61 that captures the direction of the user's line of sight are provided. The control unit 140 of the head-mounted display device 100 generates an image for display from an image captured by the camera 61 and an image for causing the image display unit 20 to display the display image generated by the image generation unit 177. A display control unit 176. As a result, the display image based on the captured image obtained by capturing the user's line-of-sight direction can be displayed so as to overlap the object that appears as an outside scene. For this reason, the external scene outside the display device and the display contents can be effectively improved by supplementing the appearance of the external scene viewed by the user based on a captured image obtained by capturing the external scene, or presenting another view. In combination, it is possible to provide a new method for utilizing the display by the display device.

The head-mounted display device 100 includes a detection unit such as a distance detection unit 173 that detects the usage state of the head-mounted display device 100. The image display control unit 176 adjusts the display mode of the display image on the image display unit 20 in accordance with the usage state of the head-mounted display device 100. For this reason, it is possible to adjust aspects such as the display size and display position of the image displayed on the image display unit 20 in accordance with the usage environment of the head-mounted display device 100 and the usage by the user. For this reason, it is possible to perform display according to the way the user visually recognizes the outside scene.
In addition, the image display control unit 176 detects the position of the head-mounted display device 100 using the distance detection unit 173 and the like, and displays the display image in accordance with the position of the head-mounted display device 100 to be detected. Adjust. For this reason, based on the position of the head-mounted display device 100, it is possible to perform display according to how the user visually recognizes the outside scene. Specifically, the image display control unit 176 adjusts at least one of the display position and the display size of the display image so that the display image avoids the center in the user's line-of-sight direction. The center in the line-of-sight direction refers to, for example, the intersection of the central axis of the line of sight when the user views the outside scene through the image display unit 20 and the display area where the image is displayed on the image display unit 20 and the vicinity thereof. . In this case, the display image is not displayed directly in front of the user's line of sight, and the display image is visually recognized by the user at a position that avoids the front (center) of the line of sight. For this reason, according to the present invention, an image captured by the camera 61 can be displayed so that the user does not become an obstacle to see the outside scene. For example, the display image can be displayed so as not to hide the target object that the user pays attention to.

Moreover, since the image generation unit 177 generates a display image by cutting out and enlarging a part of the captured image of the camera 61, the captured image obtained by capturing the user's line-of-sight direction can be enlarged and displayed. For this reason, the user can enlarge the line of sight while viewing the outside scene.
In addition, the image generation unit 177 generates a display image by cutting out a region corresponding to a preset condition in the captured image of the camera 61, and thus can display a captured image corresponding to the preset condition. For example, when a captured image corresponding to the user's preference or interest is captured, the captured image can be displayed by the image display unit 20 and shown to the user.

The head-mounted display device 100 includes a target detection unit 171 that detects an object in the direction of the user's line of sight from an image captured by the camera 61. The image generation unit 177 generates a display image by cutting out a region including the image of the target detected by the target detection unit 171 from the captured image of the camera 61. Thereby, when there is an object in the direction of the user's line of sight, the image of the object can be shown to the user.
The head-mounted display device 100 includes a distance detection unit 173 that detects the distance between the object detected by the object detection unit 171 and the user. The image generation unit 177 enlarges a part of the captured image of the camera 61 at an enlargement rate corresponding to the distance detected by the distance detection unit 173, and generates a display image. Thereby, the image of the object can be shown to the user at an enlargement rate that matches the distance from the object in the direction of the user's line of sight.
In addition, the image generation unit 177 acquires a plurality of captured images captured at different times by the camera 61, and specifies a region corresponding to a preset condition in the plurality of captured images, thereby generating a display image. Generate. Thereby, more complicated conditions can be set for a plurality of captured images, and images can be shown to the user based on the captured images corresponding to the conditions.

  In addition, the image generation unit 177 acquires a plurality of captured images captured at different times by the camera 61, and when the difference between the images of the plurality of captured images satisfies a preset condition, the acquired imaging A display image is generated by cutting out the image. Accordingly, since a display image can be generated and displayed based on a difference between a plurality of captured images, for example, a display image can be generated and displayed based on a change in image for a moving image including a plurality of continuous images. .

In addition, the image display control unit 176 stores the display image generated by the image generation unit 177 in the storage unit 120 in time series, and when the operation unit 135 detects an operation instructing reproduction, the image display control unit 176 stores the display image in the storage unit 120. The stored display image is reproduced and displayed. Here, the image display control unit 176 can display a moving image by sequentially reproducing a plurality of display images stored in time series at a predetermined frame rate. In this case, the user can see a moving image in which an image with an enlarged line-of-sight direction is recorded. Here, when the display image stored in the storage unit 120 is reproduced and displayed, the image display control unit 176 may change the magnification of the display image. For example, when an operation for designating an enlargement ratio is detected by the operation unit 135, the enlargement ratio may be changed according to this operation. Further, in FIG. 5D, the display image is displayed so as to overlap a part of the visual field VR. However, when the display image stored in the storage unit 120 is reproduced and displayed, the entire displayable area of the image display unit 20 is displayed. A display image may be displayed on the screen.
Further, the image generation unit 177 may adjust the transparency of the image IV. In the example of FIG. 5D, the transparency is zero, and the outside scene is not visible in the portion overlapping the image IV, but the image generation unit 177 sets the transparency based on the brightness of the outside scene overlapping the image IV, the brightness of the image IV, and the like. You may adjust.

  In the above embodiment, the case has been described in which the object detection unit 171 recognizes an object using the single image feature data 124a and the moving image feature data 124b stored in the storage unit 120. The single image feature data 124a is not only stored in the storage unit 120 in advance, but may be configured to be generated or changed by a user operation while the head-mounted display device 100 is in use. For example, a configuration in which the head-mounted display device 100 can perform an operation of selecting a part of the display image by operating the track pad 14 while displaying the captured image of the camera 61 on the image display unit 20 is conceivable. . With this configuration, when the user selects a part of the display image, the target detection unit 171 detects an image of a person or an object from the selected part, and the single image feature data 124a or the like is detected from the detected image. The moving image feature data 124b may be generated. Alternatively, the head-mounted display device 100 may be configured to be operable by a voice command. In this case, the control unit 140 detects the user's voice with the microphone 63, analyzes the detected voice with the voice processing unit 170, and recognizes the voice command. When the head-mounted display device 100 displays a captured image of the camera 61 on the image display unit 20, when the user instructs to select a part of the display image with a voice command, the target detection unit 171 Alternatively, an image of a person or an object may be detected from the selected portion, and single image feature data 124a or moving image feature data 124b may be generated from the detected image. Specifically, when a soccer game is shot and displayed as shown in FIG. 5A, the object is detected when the user instructs “Track No. 11” with a voice command. The unit 171 detects the number “11” from the captured image of the camera 61, extracts a portion including the detected number, obtains a feature amount, and generates single image feature data 124 a and moving image feature data 124 b. May be.

[Second Embodiment]
FIG. 8 is a block diagram illustrating a functional configuration of a head-mounted display device 100B according to the second embodiment. The head-mounted display device 100B includes the image display unit 20B instead of the image display unit 20 in the head-mounted display device 100 described in the first embodiment. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals and description thereof is omitted. FIG. 9 is a diagram illustrating a configuration of a main part of the image display unit 20B. FIG. 9A is a perspective view of a main part when the image display unit 20B is viewed from the user's head side, and FIG. FIG.

The image display unit 20B includes a distance sensor 64 and a line-of-sight sensor 68 in addition to the configuration of the image display unit 20 (FIG. 2). In other respects, the head-mounted display device 100 and the head-mounted display device 100B are common, and the operation of the head-mounted display device 100 in the first embodiment is the same as the head-mounted display device. 100B can be executed.
261A and 262A shown in FIG. 9B are half mirrors formed on the right light guide plate 261 and the left light guide plate 262, respectively, and correspond to display areas that display images so that the outside scene can be transmitted. The half mirrors 261A and 262A are also present in the image display unit 20 of the first embodiment.

The distance sensor 64 is disposed at the boundary between the right optical image display unit 26 and the left optical image display unit 28 as shown in FIG. In a state where the user wears the image display unit 20B, the position of the distance sensor 64 is substantially in the middle between both eyes of the user in the horizontal direction and above the both eyes of the user in the vertical direction. The distance sensor 64 detects the distance to the measurement object located in the preset measurement direction.
For example, as shown in FIG. 9B, the distance sensor 64 detects the distance to the object OB ahead of the line of sight RD of the user's right eye RE and the line of sight LD of the left eye LE. In this case, the direction 64A in which the distance sensor 64 detects is the front of the image display unit 20B.

The distance sensor 64 includes, for example, a light source such as an LED or a laser diode, and a light receiving unit that receives reflected light that is reflected from the light to be measured. In this case, the distance sensor 64 performs a triangular distance measurement process or a distance measurement process based on a time difference according to the control of the distance detection unit 173. The distance sensor 64 may include a sound source that emits ultrasonic waves and a detection unit that receives ultrasonic waves reflected by the measurement object. In this case, the distance sensor 64 performs a distance measurement process based on a time difference until the reflection of the ultrasonic wave according to control of the position detection unit 162 described later. The distance sensor 64 may include a light source, a light receiving unit, or a sound source and a detection unit, and the distance detection unit 173 may perform a distance measurement process.
The measurement direction of the distance sensor 64 of the present embodiment is the front side direction of the head-mounted display device 100 and overlaps with the imaging direction of the camera 61.

  Further, as shown in FIG. 9A, the line-of-sight sensor 68 is disposed on the surface on the user side of the image display unit 20B. The line-of-sight sensor 68 corresponds to the right eye RE (FIG. 9B) and the left eye LE (FIG. 9B) of the user to display the right optical image display unit 26 and the left optical image display, respectively. A pair is provided at a central position with the portion 28. The line-of-sight sensor 68 includes, for example, a pair of cameras that respectively capture the right eye RE and the left eye LE of the user. The line-of-sight sensor 68 captures images in accordance with the control of the control unit 140 (FIG. 4), and the control unit 140 detects reflected light and pupil images on the eyeball surfaces of the right eye RE and the left eye LE from the captured image data. Identify the direction. In addition, since the line-of-sight sensor 68 captures the left and right eyes of the user, when the user closes the eyes, the line-of-sight sensor 68 can detect that the eyes are closed.

  As shown in the block diagram of FIG. 8, the distance sensor 64 and the line-of-sight sensor 68 are connected to the control unit 140 through the connection unit 40. The distance detection unit 173 can detect the distance to the object by controlling the distance sensor 64. The distance detection unit 173 may be configured to detect the distance to the object based on the captured image data of the camera 61 as described in the first embodiment.

The head-mounted display device 100 </ b> B according to the second embodiment can perform an operation based on the actions of the user by imaging the actions of the user's hands, feet, eyes, and the like with the camera 61. The user's hand can be imaged by the camera 61, and the eye can be imaged or detected by the eye sensor 68. Further, the user's foot can be imaged by providing the camera 61 with a wide-angle camera or a foot camera (not shown) provided facing the lower side of the image display unit 20B.
The image display control unit 176 detects the movement of the user based on the captured image data of the camera 61, the captured image data of the foot camera, or the detection result of the visual line sensor 68. Further, the image display control unit 176 detects this movement as an operation when the detected movement of the user corresponds to a preset movement.
In this case, the image display control unit 176 functions as an operation detection unit, an image display control unit, and a sound detection unit.

  The user's movement detected as an operation by the image display control unit 176 is included in the detection feature data 124 stored in the storage unit 120. In the detected feature data 124, feature amount data for extracting an image of the user's hand or foot from the captured image of the camera 61, and feature amount data of the motion of the extracted user's hand or foot image. Etc. are included.

  Further, the image display control unit 176 may be configured to be able to detect an operation in which a user emits sound. In this case, the sound collected by the microphone 63 is converted into digital sound data by the sound processing unit 170, and the image display control unit 176 processes the digital sound data. That is, the image display control unit 176 generates a voice feature quantity from the digital voice data having a predetermined time length, and whether or not this feature quantity corresponds to a voice recognition feature quantity included in the detected feature data 124. Based on the above, it detects voice operations. Here, the image display control unit 176 performs voice recognition on the digital voice data to generate text data of the utterance content, and even when an operation is detected when the text data includes a preset character string. Good. In this case, the detection feature data 124 may include dictionary data for speech recognition and data related to a character string detected as an operation.

  The operation of the image display control unit 176 corresponding to the user's operation will be described with reference to FIGS. FIG. 10 is a flowchart showing the operation of the head-mounted display device 100B, and FIG. 11 shows a display example on the image display unit 20B.

The operation of FIG. 10 can be executed at an arbitrary timing when a preset condition is satisfied after the display process is started in step S23 of FIG. 4 or when the user operates the control device 10.
The control unit 140 shifts to a “user operation mode” that is an operation mode in which an operation is performed based on the user's movement, and starts detecting the user's movement (step S51). As described above, the motion is detected based on the captured image data of the camera 61, the detection result data of the line-of-sight sensor 68, or the digital audio data generated by the audio processing unit 170.

  The image display control unit 176 determines whether or not the detected user movement corresponds to a movement set in advance as the “first movement” (step S52). The first operation is, for example, an operation in which the user closes (collapses) one eye, an operation that aligns both feet, or an operation that speaks “start zoom” by voice. In addition, the first operation may be an operation in which the user forms a ring with the thumb (mother finger) and the index finger (indicating finger) or the middle finger. Moreover, the operation | movement which puts out a user's both hands face to the imaging range of the camera 61 may be sufficient.

  While the operation corresponding to the first operation is not detected (step S52; NO), the image display control unit 176 waits for an operation. When an operation corresponding to the first operation is detected (step S52; YES), the image display control unit 176 starts zoom adjustment (step S53). In the zoom adjustment, first, an indicator is displayed on the image display unit 20B by the function of the image display control unit 176 (step S54). The indicator includes characters and images indicating that zoom operation is possible for the user, and is displayed so that the user can see the image superimposed on the outside scene. This indicator may include characters and images that guide or explain an operation performed by the user to instruct zoom adjustment.

The image display control unit 176 identifies the marker position that is the center of zoom adjustment, and displays the marker at the identified position (step S55). The marker is an image or symbol indicating the center of zoom adjustment to the user, and is displayed by the image display unit 20B.
As a method for specifying the marker position, for example, when the user closes one eye in the first operation, there is a method for detecting the line of sight of the eye that is not closed and setting the line-of-sight direction as the marker position. Here, when the user opens both eyes, there is a possibility that the line-of-sight direction does not necessarily match and the marker position cannot be specified as one. If the eye or the left eye is set as the dominant eye, the line-of-sight direction of the set dominant eye may be set as the marker position. In this case, the dominant eye may be automatically determined.
Further, when the user performs an operation of making a ring with the thumb and forefinger or middle finger in the first operation, the center of the ring can be set as the marker position. Further, in the first operation, when the user performs an operation of putting both hands facing each other and bringing them into the imaging range of the camera 61, the center of both hands may be set as the marker position.

  In addition, when the image display control unit 176 detects a plurality of objects in the user's line-of-sight direction from the captured image of the camera 61 and can detect the characteristics of each detected object, the user designates the object. The marker position may be specified by uttering a sound. That is, the user speaks the characteristics of the person or the object that is the object captured in the image captured by the camera 61, and the image display control unit 176 selects one object based on this one voice, and this object An object may be specified as the marker position. In this case, for example, the user can specify the marker position by speaking a word specifying a person, a place, a person, etc. such as “ball”, “goal”, “number 10”.

  FIG. 11A shows a display example in a state where the marker is recognized. In FIG. 11A, an indicator IND is displayed by the image display unit 20B, and the center of the ring is specified as the marker position and the marker M is displayed in response to the user's operation of making a ring with a finger. The indicator IND and the marker M are visually recognized in the user's visual field, and the user's hand H is seen as an outside scene.

The image display control unit 176 starts the operation of following the object at the specified marker position, moving the image, and holding the marker position at the center of the display image (step S56).
Here, the image display control unit 176 detects the second operation for designating the zoom magnification or designating the change of the zoom magnification (step S57). The second operation may be, for example, an operation in which the user makes a ring with the thumb and forefinger or middle finger, and rotates the hand while maintaining the ring. In this case, the image display control unit 176 increases the zoom magnification when the rotation direction is clockwise, and decreases the zoom magnification when the rotation direction is counterclockwise. This change in zoom magnification can be a unit amount set in advance. In this case, the user can adjust to a desired zoom magnification by repeatedly performing the second operation.

  In the example of FIG. 11A, the user puts his right hand into the imaging range of the camera 61 and makes a ring with his finger. In this state, when the hand is moved so as to rotate the ring made with the right hand clockwise (CW in the figure), as shown in FIG. growing. That is, a larger image IV is displayed with the marker M as the center. In the state of FIG. 11A, when the user moves the right hand so as to rotate the ring of the right hand counterclockwise (CCW in the figure), as shown in FIG. Zoom magnification decreases. In FIG. 11C, the image IV is visually reduced with the marker M as the center.

Whether or not to make a ring, when the user puts one hand in the imaging range of the camera 61, the operation of bringing the hand closer to the camera 61 may be a second operation corresponding to an increase in zoom magnification. it can. Further, the operation of moving the hand away from the camera 61 can be a second operation corresponding to the reduction of the zoom magnification.
In addition, for example, as shown in FIG. 11D, when the user performs an operation of facing both hands as the first operation, the center of both hands may be used as the marker M. In this case, as shown by an arrow in FIG. 11D, the user can widen or narrow the distance between both hands as the second motion. In this case, the image display control unit 176 performs a process of increasing the zoom magnification when the user widens the distance between both hands, and decreasing the zoom magnification when the distance between both hands is decreased.

As illustrated in FIGS. 11A to 11D, the target detection unit 171 detects an object corresponding to one of the user's extremities from the captured image of the camera 61, and the movement and position of the detected object Based on the above, the position of the marker M is specified. The detected object is not limited to the user's hand, arm, or foot, but may be a part of the limb such as a finger or a palm.
Specifically, a configuration is possible in which a thumb and an index finger appear in the imaging range of the camera 61 and a marker M is displayed between the thumb and the index finger. Further, an operation for narrowing the distance between the thumb and the index finger may be assigned to an instruction for reducing the zoom magnification, and an operation for increasing the distance between the thumb and the index finger may be assigned to an instruction for increasing the zoom magnification. Other fingers may be assigned to the above operation, and three or more fingers and movements of these fingers may be detected. For example, three or more fingers may be detected at the same time, and movements of each finger, changes in the relative positions of a plurality of fingers, and the like may be detected as movements. The processing corresponding to the movement of the finger is not limited to the specification of the marker M and the change in the zoom magnification, but may be volume adjustment of output sound and light adjustment adjustment of the displayed image.

Further, the second operation may be an operation by a user's foot. The operation in which the user puts the right foot ahead of the left foot can be a second action corresponding to an increase in the zoom magnification, and the zoom magnification is increased according to the time during which the right foot is put out before the left foot. Also good. In this case, the operation in which the user moves the right foot behind the left foot can be a second operation corresponding to the reduction of the zoom magnification, and the zoom magnification according to the time during which the right foot is positioned behind the left foot. May be reduced. In addition, the zoom magnification may be changed according to the amount of the right foot placed before or after the left foot.In this case, the indicator has a scale indicating the change in the zoom magnification and the correspondence between the zoom magnification and the amount of the foot. It may be displayed.
Further, the second operation may be speech of the user's voice, and the image display control unit 176 adjusts the zoom magnification by speaking a word that specifies zoom magnification, reduction, or zoom magnification with a numerical value or the like. May be.

  While the operation corresponding to the second operation is not detected (step S57; NO), the image display control unit 176 waits for an operation. When an operation corresponding to the second operation is detected (step S57; YES), the image display control unit 176 changes the zoom magnification (step S58). Thereafter, the image display control unit 176 detects the third operation.

The third operation corresponds to an operation for instructing the start of recording. The third operation can be, for example, an operation in which the user puts one hand into the imaging range of the camera 61 and tilts, an operation in which the user puts the left foot forward of the right foot, or an operation in which the user closes both eyes. Moreover, the operation | movement which utters the word which a user instruct | indicates the recording start may be sufficient. When the third operation is detected (step S59; YES), the image display control unit 176 starts recording and records the captured image of the camera 61 and the digital audio data of the microphone 63 generated by the audio processing unit 170. Start (step S60).
Thereafter, the image display control unit 176 detects the fourth operation (step S61). The fourth operation corresponds to an operation for instructing the end of recording. For example, when the user performs an operation of tilting one hand out of the imaging range of the camera 61 in the third operation, the operation of returning the tilt of the hand can be the fourth operation. Further, when the user performs an operation of putting the left foot forward from the right foot in the third operation, the operation of lowering the left foot back can be set as the fourth operation. Moreover, it is good also considering the operation | movement which a user closes both eyes as a 4th operation | movement. Alternatively, the user may utter a word that instructs the end of recording. When the fourth operation is detected (step S61; YES), the image display control unit 176 ends the recording (step S62).

Thereafter, the image display control unit 176 determines whether or not to end the zoom adjustment operation (step S63). Even when the third operation is not detected within the predetermined time in step S59 (step S59; NO), the image display control unit 176 performs the determination in step S63.
When the user's operation on the operation unit 135 or the end of the zoom operation is instructed by hand, foot, eye (eyelid), voice or the like as in the first to fourth actions (step S63; YES), the image The display control unit 176 ends the user operation mode, stops displaying the indicator, shifts to normal display (step S64), and ends the process. If it is determined that the zoom operation is not finished (step S63; NO), the image display control unit 176 returns to step S57.

  As described above, the head-mounted display device 100B can be operated by operations such as the user's hand, foot, eye (eyelid), and speech. Accordingly, various instruction operations can be performed without directly operating the control device 10. For example, the control device 10 and the image display unit 20 are attached to the body or clothes, and both hands are free, so-called hands. It can be operated for free. In addition, since it is possible to perform operations with eyes, voice, or feet, the head-mounted display device 100B can be operated even when both hands are used for work and the like and cannot be performed with hands.

  Also, in steps S59 and S61, when a hand or a foot is put out in the imaging range of the camera 61 as the third operation and the fourth operation, the image display control unit 176 does not include the hand or foot in the recorded image. May perform processing. In other words, when the hand and foot motions are detected as the third and fourth motions, the image display control unit 176 performs a process of removing the hand and foot images detected as the third and fourth motions from the recorded images. May be. For example, hand and foot images can be removed by performing a trimming process on a frame of an image captured by the camera 61.

Further, prior to the operation of FIG. 10, the control unit 140 may perform calibration that associates the line-of-sight direction detected by the line-of-sight sensor 68 with the imaging range of the camera 61. The calibration result may be included in the detection feature data 124 and stored in the storage unit 120, for example.
Further, in the operation of FIG. 10, the zoom magnification is adjusted in steps S53 to S58, and then the recording is performed in steps S59 to S62. However, only the recording may be performed. Further, when the first operation is detected in step S52, the operation of step S60 may be executed when an operation preset as the third operation is detected.
Furthermore, in addition to the movements of the hands and feet, it is also possible to detect movements by an indicator other than the hands and legs as the first to fourth movements. In this case, the indicator only needs to have a feature that the image display control unit 176 can extract from the captured image of the camera 61 in appearance, and may have a specific shape, color, and pattern. Further, if the camera 61 can receive and detect light outside the visible region such as infrared light and ultraviolet light, a device that emits such light can be used as an indicator. An operation of moving these indicators so as to draw a predetermined trajectory in the imaging range of the camera 61 or simply putting them in the imaging range of the camera 61 can be the first to fourth operations. If the indicator is a wearable device that can be worn on the user's body or clothes like a wristwatch, the operability can be further improved.

Furthermore, as described above, the operations listed as the first to fourth operations or the operations for specifying the marker position are not only for adjusting the zoom magnification and recording instructions, but also for operations at the time of reproducing recorded images, for example. Applicable.
FIG. 12 shows a display example in the case of instructing playback of a recorded moving image. The example of FIG. 12 is displayed when reproduction is instructed by operating the control device 10, for example. In FIG. 12, a timeline TL showing the entire recorded moving image is displayed in the visual field VR. For a frame at a specific time on the timeline TL, the image display control unit 176 generates and displays thumbnail images P1 to P3. The time for generating and displaying the thumbnail image may be instructed by a user's specific operation such as the first to fourth operations during recording, and corresponds to the time length of the recorded image. The image display control unit 176 may automatically select the time. Further, the image display control unit 176 may automatically select the time based on the change of the image. In the strip on which the display of FIG. 12 is performed, for example, an operation of selecting a time to start reproduction from the thumbnail images P1 to P3 can be performed according to the viewing direction.

As described above, according to the second embodiment to which the present invention is applied, the image display control unit 176 detects the user's operation based on the captured image of the camera 61, and displays it according to the detected user's operation. The display mode of the image for use is adjusted. For example, the zoom magnification is adjusted. Therefore, the user can adjust the display mode of the display image by performing an operation so that the camera 61 can capture an image. In addition, the image display control unit 176 may detect the sound collected by the microphone 63 and adjust the display mode of the display image in accordance with the sound detection result.
The image display control unit 176 detects the user's action based on the captured image of the camera 61. Then, the image display control unit 176 can extract and process a region specified based on the user's action, for example, a region centered on the marker M, and the image generation unit 177 can generate a display image.

In addition, the image display control unit 176 identifies a direction corresponding to the user's line-of-sight direction, and detects an object in the user's line-of-sight direction from the captured image of the camera 61. Then, the image generation unit 177 may extract and process a region including the image of the object to generate a display image. Further, when a direction corresponding to the user's line-of-sight direction is specified, the marker M may be displayed at a position corresponding to the specified direction.
Furthermore, the direction corresponding to the user's line of sight may be obtained by detecting the line of sight of the user's dominant eye. Further, the movement of the user's hand or foot may be detected from the captured image of the camera 61, and the position of the marker M may be specified based on this movement.

In addition, this invention is not restricted to the structure of said each embodiment, In the range which does not deviate from the summary, it can be implemented in a various aspect.
In the first embodiment, as an example in which the image generation unit 177 extracts a region corresponding to a preset condition from the captured image of the camera 61, a region in which the target detection unit 171 performs the matching process to detect the target object. The example which cuts out was demonstrated. The present invention is not limited to this. For example, the image generation unit 177 may detect and cut out a region corresponding to a preset condition in the captured image to generate a display image. For example, when the position and size of the region are set as the conditions for the region to be cut out from the captured image with respect to the image generation unit 177, the image generation unit 177 does not depend on the detection result of the target detection unit 171 in the captured image. Cut out the area of the set position and size. In addition, the condition of the region to be cut out may be a condition for determining a color or luminance in the captured image, and does not necessarily require processing by the target detection unit 171.
Further, the process performed when the image generation unit 177 generates a display image from the captured image is not limited to the process of cutting out and enlarging a part of the captured image. For example, the entire captured image may be used as it is or as a display image after resolution conversion. Further, for example, the image generation unit 177 may cut out a plurality of regions from the captured image to generate a plurality of display images, and simultaneously display the plurality of images IV in the visual field VR. Also, the position where the display image is displayed may be changed or adjusted as appropriate by the image generation unit 177 or the image display control unit 176.

  In the first and second embodiments, the imaging direction of the camera 61 may be movable with respect to the image display units 20 and 20B. For example, the camera 61 is attached to the frame of the image display units 20 and 20B via a rotatable arm, link, or flexible arm, and the orientation of the camera 61 is changed without moving the image display units 20 and 20B. It may be possible. In this case, it is good also as a structure which can detect the angle of the optical axis of the camera 61 with respect to the image display parts 20 and 20B, or an imaging direction. Specifically, a rotation sensor may be provided in a link or arm rotation unit that instructs the camera 61, or a gyro sensor or an acceleration sensor may be provided in the camera 61 itself. In this case, the line-of-sight direction of the user may be obtained based on the direction of the optical axis of the camera 61. When the optical axis of the camera 61 can be moved, the user directs the optical axis of the camera 61 in the direction he wants to see, so the direction of the optical axis of the camera 61 can be regarded as the direction in which the user is gazing or related. . Therefore, based on the direction of the optical axis of the camera 61, for example, the position of the head-mounted display device 100 can be determined in step S31 of FIG. Further, based on the direction of the optical axis of the camera 61, the position of the marker can be specified in step S55 of FIG.

  Further, the image display control unit 176 may control the movement of the image displayed by the image display unit 20. For example, the image display control unit 176 may control the image displayed on the image display unit 20 based on the movement of the image display unit 20 that can be detected from the detection values of the 9-axis sensor 66 and / or the GPS 115. The image display control unit 176 displays an image to be displayed on the image display unit 20 when the acceleration or speed of the movement of the image display unit 20 exceeds a threshold value while displaying an image generated from the captured image of the camera 61. The image displayed in the direction opposite to the movement of the image display unit 20 may be shifted. In this case, the image display control unit 176 may change the display image generated by the image generation unit 177 or may display a part of the display image. According to this configuration, when the movement of the head of the user wearing the head-mounted display device 100 is fast and the change in the display image becomes severe, the movement of the image displayed by the image display unit 20 is suppressed, User discomfort can be reduced. This function is called a stabilization function. The stabilization function can be switched on / off by an operation on the operation unit 135. For example, as described with reference to FIGS. 10 to 11, the stabilization function may be switched on / off by a user's hand, foot, eye movement, or voice.

  The stabilization function can also be applied to a case where the object detected by the object detection unit 171 moves at a high speed in the captured image. For example, when a ball is detected as an object as illustrated in FIGS. 5A to 5C, a moving speed is obtained from a change in the position of the ball in the captured image, and the acceleration or speed is equal to or greater than a threshold value. The change of the image to be displayed or the movement of the ball at the displayable position of the image display unit 20 may be suppressed. Alternatively, even when the moving speed or acceleration obtained from the change in the position of the ball in the captured image is greater than or equal to the threshold value, the image generation unit 177 may temporarily stop the process of extracting the target image and generating the display image. Good. Further, even if the image generation unit 177 decreases the enlargement ratio of the extracted image, changes the angle of view of the camera 61, or reduces the zoom magnification when the camera 61 has a zoom mechanism or a digital zoom function. Good. Thus, by reducing the display size of the image of the object displayed by the image display unit 20, an effect that the object can be easily seen even when the moving speed of the object is fast can be expected.

  Further, for example, in the above-described embodiment, an example has been described in which the image generation unit 177 determines the enlargement rate, and the image generation unit 177 cuts out and enlarges the captured image by image processing according to the enlargement rate. For example, the camera 61 may have a zoom function, and the image generation unit 177 may control the zoom magnification of the camera 61. In this case, the image generation unit 177 may acquire a captured image captured at a zoom magnification specified by the image generation unit 177, and the image may be cut out to be a display image.

  Further, as the image display unit, instead of the image display unit 20, another type of image display unit such as an image display unit worn like a hat may be adopted, corresponding to the left eye of the user. What is necessary is just to provide the display part which displays an image, and the display part which displays an image corresponding to a user's right eye. The display device of the present invention may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Moreover, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet, or may be a head-up display (HUD) used for a windshield of an automobile. Furthermore, the user's eyeball forms an image on the retina such as a so-called contact lens type display that is worn on the user's eyes (for example, on the cornea) and an implant type display that is embedded in the eyeball. The display may be the image display unit 20.

In addition, the present invention may be applied to the user's body, and can be applied regardless of whether support by another method is required. For example, a binocular-type hand held display (Hand Held Display) that the user uses with both hands may be adopted as the image display unit 20 of the present application. This type of display needs to be held by the user in order to keep it attached to the user, but should be attached to the user's head or face when the user looks at the display. Therefore, it is included in the display device of the present invention. Furthermore, even if the display device is fixed to the floor or wall surface by a support leg or the like, as long as it can be attached to the user's head or face when the user views the display on the display, the display of the present invention Included in the device.
Further, only the image display unit 20 or a display unit including a configuration related to image display in the image display unit 20 is mounted on the user's body, and the control device 10 other than the display unit, or the control device 10 and the control unit. The control system including 140 may be physically configured as a separate body. For example, an image display unit 20 or a display unit that is a part of the image display unit 20 may be wirelessly connected to a device having another control system, and the display device may be similar to the head-mounted display device 100. As an apparatus provided with this control system, a conventional electronic device such as a smartphone, a mobile phone, a tablet computer, or a personal computer of another shape can be used. Of course, this type of display device can also be applied.

Furthermore, in the above embodiment, the image display units 20 and 20B and the control device 10 are separated and connected by way of the connection unit 40. However, the control device 10 and the image display unit 20 It is also possible to adopt a configuration in which is integrally formed and is mounted on the user's head.
Further, the control device 10 and the image display units 20 and 20B are connected by a longer cable or a wireless communication line. As the control device 10, a notebook computer, a tablet computer or a desktop computer, a game machine, a portable phone, A portable electronic device including a smartphone or a portable media player, other dedicated devices, or the like may be used.

  Further, for example, as a configuration for generating image light in the image display unit 20, a configuration including an organic EL (Organic Electro-Luminescence) display and an organic EL control unit may be used, or an LCOS (Liquid crystal) may be provided. On silicon and LCoS are registered trademarks), and digital micromirror devices can also be used. Further, for example, the present invention can be applied to a laser retinal projection type head mounted display. That is, the image generation unit includes a laser light source and an optical system that guides the laser light source to the user's eye, makes the laser light enter the user's eye, scans the retina, and forms an image on the retina. A configuration that allows the user to visually recognize an image may be employed. When a laser retina projection type head-mounted display is employed, the “image light emitting area in the image light generation unit” can be defined as an image area recognized by the user's eyes.

  As an optical system that guides image light to the user's eyes, an optical member that transmits external light that is incident from the outside toward the apparatus and that enters the user's eyes together with the image light can be employed. Moreover, you may use the optical member which is located ahead of a user's eyes and overlaps a part or all of a user's visual field. Further, a scanning optical system that scans a laser beam or the like to obtain image light may be employed. Further, the optical member is not limited to guiding the image light inside the optical member, and may have only a function of guiding the image light by refracting and / or reflecting it toward the user's eyes.

  Further, the present invention can be applied to a display device that employs a scanning optical system using a MEMS mirror and uses MEMS display technology. That is, as an image display element, a signal light forming unit, a scanning optical system having a MEMS mirror that scans light emitted from the signal light forming unit, and an optical member on which a virtual image is formed by light scanned by the scanning optical system May be provided. In this configuration, the light emitted from the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided through the optical member, and reaches the virtual image forming surface. When the MEMS mirror scans the light, a virtual image is formed on the virtual image forming surface, and the user recognizes the virtual image with the eyes, thereby recognizing the image. The optical component in this case may be one that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above embodiment, and may use a half mirror surface. .

  The display device of the present invention is not limited to a head-mounted display device, and can be applied to various display devices such as a flat panel display and a projector. The display device of the present invention may be any device that allows an image to be visually recognized by external light and image light. For example, a configuration in which an image by image light is visually recognized by an optical member that transmits external light is exemplified. Specifically, the above-described head-mounted display includes an optical member that transmits external light, and a light-transmitting plane or curved surface (glass) that is fixedly or movably installed at a position away from the user. It is also applicable to a display device that projects image light onto a transparent plastic or the like. As an example, a configuration of a display device that projects image light onto a window glass of a vehicle and allows a user who is on board or a user outside the vehicle to visually recognize the scenery inside and outside the vehicle together with an image by the image light. It is done. In addition, for example, image light is projected onto a transparent, semi-transparent, or colored transparent display surface that is fixedly installed such as a window glass of a building, and is displayed together with an image by the image light to a user around the display surface. A configuration of a display device that visually recognizes a scene through the surface can be given.

In addition, at least a part of the functional blocks shown in FIGS. 2 and 8 may be realized by hardware, or may be realized by cooperation of hardware and software. As described above, the present invention is not limited to a configuration in which independent hardware resources are arranged.
The operation of the control unit 140 described in the first and second embodiments can be realized as a program. That is, the control unit 140 includes a CPU, a ROM, and a RAM, and when the CPU executes a program, for example, the object detection unit 171, the distance detection unit 173, the image display control unit 176, and the image generation unit 177 A function may be realized. Further, the CPU of the control unit 140 may execute the functions of the operating system 150, the image processing unit 160, the sound processing unit 170, and the display control unit 190 by executing a program. Some or all of these programs may be realized as hardware. For example, each of the processing units included in the control unit 140 may be configured using an ASIC (Application Specific Integrated Circuit), or a PLD (Programmable Logic Array) such as an FPGA (Field-Programmable Gate Array). Device) may be programmed. Further, it may be realized as a circuit including a plurality of semiconductor devices. When the hardware executes the program to realize the above function, the program may be stored in the ROM of the control unit 140 or the storage unit 120 or another storage device in the control device 10. Alternatively, the control unit 140 may acquire and execute a program stored in an external device via the communication unit 117 or the interface 180.

  Of the configurations formed in the control device 10, only the operation unit 135 may be formed as a single user interface (UI), or the power source 130 in the above embodiment is formed independently and is replaceable. It may be a configuration. Further, the configuration formed in the control device 10 may be formed in the image display unit 20 in an overlapping manner. For example, the control unit 140 illustrated in FIG. 2 may be formed in both the control device 10 and the image display unit 20, or the control unit 140 formed in the control device 10 and the CPU formed in the image display unit 20. It is good also as a structure by which the function to perform is divided | segmented separately.

  DESCRIPTION OF SYMBOLS 10 ... Control apparatus 20, 20B ... Image display part (display part), 21 ... Right holding part, 22 ... Right display drive part, 23 ... Left holding part, 24 ... Left display drive part, 26 ... Right optical image display part 28 ... Left optical image display unit, 61 ... Camera (imaging unit), 63 ... Microphone, 64 ... Distance sensor, 68 ... Gaze sensor, 100, 100B ... Head-mounted display device (display device), 117 ... Communication unit , 120 ... storage section, 140 ... control section, 150 ... operating system, 160 ... image processing section, 170 ... audio processing section, 171 ... target detection section, 173 ... distance detection section, 176 ... image display control section, 177 ... image Generation unit, 180 ... interface, 190 ... display control unit, 201 ... right backlight control unit, 202 ... left backlight control unit, 211 ... right LCD control unit, 212 ... left LCD control unit, 221 ... right Backlight, 222 ... left backlight, 241 ... right LCD, 242 ... left LCD, 251 ... right projection optical system, 252 ... left projection optical system, 261 ... Migishirubekoban, 262 ... Hidarishirubekoban, DA ... data acquisition unit.

Claims (22)

A display device used by being worn on a user's body,
A display unit that displays an image so that it can be seen through the outside scene together with the outside scene;
An imaging unit that images the user's line-of-sight direction;
An image generation unit that generates a display image from a captured image of the imaging unit;
An image display control unit that causes the display unit to display the display image generated by the image generation unit;
A display device comprising: A detection unit for detecting a use state of the display device;
The image display control unit adjusts a display mode of the display image on the display unit corresponding to a use state of the display device detected by the detection unit;
The display device according to claim 1. The detection unit performs detection related to the position of the display device, and the image display control unit adjusts the display mode of the display image corresponding to the position of the display device detected by the detection unit;
The display device according to claim 2. The image display control unit corresponds to the position of the display device detected by the detection unit, so that the display image avoids the center in the user's line-of-sight direction. Adjusting at least one of the sizes,
The display device according to claim 3. An operation detection unit for detecting the operation of the user based on a captured image of the imaging unit;
The image display control unit adjusts a display mode of the display image corresponding to the user's operation detected by the operation detection unit;
The display device according to claim 2, wherein: It has a voice detector that detects voice,
The image display control unit adjusts a display mode of the display image in accordance with a sound detection result of the sound detection unit;
The display device according to claim 1. The image generation unit generates the display image by extracting and processing a part of the captured image of the imaging unit;
The display device according to claim 1, wherein: An operation detection unit for detecting the operation of the user based on a captured image of the imaging unit;
The image generation unit extracts and processes a region specified based on the user's motion detected by the motion detection unit from a captured image of the imaging unit, and generates the display image.
The display device according to claim 7. The image generation unit generates the display image by cutting out and enlarging a part of the captured image of the imaging unit;
The display device according to claim 7 or 8. The image generation unit generates the display image by cutting out a region corresponding to a preset condition in the captured image of the imaging unit;
The display device according to claim 7, wherein: A target detection unit that identifies a direction corresponding to the user's line-of-sight direction and detects an object in the user's line-of-sight direction from a captured image of the imaging unit;
The image generation unit extracts and processes a region including an image of the object detected by the target detection unit from a captured image of the imaging unit, and generates the display image;
The display device according to claim 1, wherein: The object detection unit detects the user's line-of-sight direction, and detects the object in the detected line-of-sight direction of the user;
The display device according to claim 11. The target detection unit detects a line-of-sight direction of an eye corresponding to a dominant eye among both eyes of the user;
The display device according to claim 12. The target detection unit, when the predetermined motion of the user is detected by the motion detection unit, to specify a direction corresponding to the visual line direction of the user based on the predetermined motion;
The display device according to claim 11. The target detection unit, when the motion detection unit detects a motion by at least one of the limbs of the user, to specify a direction corresponding to the visual line direction of the user based on the detected motion;
The display device according to claim 14. A storage unit for storing the feature amount of the image of the object;
The target detection unit detects the target object by detecting an image suitable for a feature amount stored in the storage unit from a captured image of the imaging unit;
The display device according to claim 11, wherein: The feature quantity stored in the storage unit includes any one of a feature quantity of an image of the specific object and general-purpose feature quantities corresponding to a plurality of similar objects.
The display device according to claim 16. A distance detection unit for detecting a distance between the object detected by the object detection unit and the user;
The image display control unit generates a display image by enlarging a part of the captured image of the imaging unit at an enlargement rate according to the distance detected by the distance detection unit;
The display device according to claim 11, wherein: The image generation unit acquires a plurality of captured images captured at different times by the imaging unit, and identifies the region corresponding to a preset condition in the plurality of captured images, thereby obtaining the display image. Generating,
The display device according to claim 1, wherein: The image generation unit acquires a plurality of captured images captured at different times by the imaging unit, and when the difference between the images of the plurality of captured images satisfies a preset condition, the acquired captured image And generating the display image,
The display device according to claim 19. A control method for a display device that includes a display unit that transmits an outside scene and displays an image so that the outside scene can be visually recognized, and is used by being worn on the user's body,
Image the user's line of sight,
Generate a display image by cutting out and enlarging part of the captured image,
Displaying the generated display image on the display unit;
A control method of a display device characterized by the above. A program that can be executed by a computer that controls a display device that is mounted on a user's body and includes a display unit that transmits an outside scene and displays an image so as to be visible together with the outside scene.
The computer,
An image generation unit that generates a display image by cutting out and enlarging a part of the captured image obtained by capturing the user's line-of-sight direction;
An image display control unit that causes the display unit to display an image generated by the image generation unit;
Program to make it function.

Images