JP2016034091A - Display device, control method of the same and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enhance convenience by displaying a captured image by an appropriate method by a display device capable of displaying the image together with an outside scene.SOLUTION: A head-mounted type display device 100 comprises: an image display section 20 displaying an image so as to allow the image to be transmitted through an outside scene and to be visually identified together with the outside scene; a camera 61; an imaging processing section 163 causing imaging to be executed by the camera 61; a distance acquisition section 161 acquiring a distance to a subject captured by the camera 61; and a preview control section 165 displaying a captured image as a reference image and adjusting a display form of the reference image such that as the distance to the subject acquired by the distance acquisition section 161 is shorter, visual identification of the reference image is more facilitated.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a display device, a display device control method, and a program.

  2. Description of the Related Art Conventionally, there has been known an apparatus in which an imaging device is connected to a display device mounted on a user's head and an image captured by the imaging device is displayed on the display device (see, for example, Patent Document 1). The display device of Patent Document 1 has a glasses shape that is worn in front of a user, and includes a shutter that can be opened and closed. In the imaging mode in which an image captured by the imaging device is displayed, the shutter is closed and the outside scene cannot be seen. In this imaging mode, an image captured by the imaging apparatus is displayed and an operation display is performed so that the user can operate while viewing the captured image even when the outside cannot be seen.

JP 2014-42279 A

In the conventional display device, since the shutter is closed in the imaging mode, there is no opportunity to view the outside scene and the captured image at the same time. If a captured image is displayed in a state where the outside scene is visible, it is necessary that the captured image display does not interfere with the field of view of the outside scene and that the captured image is displayed in an easy-to-view manner, but there is no method that satisfies this conflicting request. .
The present invention has been made in view of the above-described circumstances, and an object of the present invention is to display a captured image by a display apparatus that can display an image together with the outside scene in an appropriate manner so that the captured image can be seen along with the outside scene.

In order to achieve the above object, a display device according to the present invention includes a display unit that displays an image so as to be visible through the outside scene and the outside scene, an imaging unit, and an imaging processing unit that causes the imaging unit to perform imaging. A distance acquisition unit that acquires a distance from the subject imaged by the imaging unit, a captured image of the imaging unit is displayed as a reference image on the display unit, and a display form of the reference image on the display unit is A reference image control unit that adjusts the reference image so that the reference image becomes easier to visually recognize as the distance to the subject acquired by the distance acquisition unit is shorter.
According to the present invention, the display device makes it easy to see the reference image when the distance from the subject is short and the difference between the angle of view of the imaging unit and the appearance of the outside scene visible to the user is likely to occur. For this reason, the user who uses the display device can easily adjust the imaging direction while viewing the captured image. Therefore, the captured image can be displayed in an appropriate display form, and the visibility of the outside scene and the captured image can be compatible and the convenience can be enhanced.

Here, the imaging unit may capture a user's visual field direction, and more specifically, the imaging direction of the imaging unit is a direction including at least a part of the user's visual field in the imaging range. May be. In this case, a captured image including at least a part of the user's visual field is displayed on the display unit as a reference image. Therefore, a captured image obtained by capturing a part of the outside scene that the user sees through the display unit is displayed as a reference image on the outside scene. Therefore, the user can easily adjust the imaging direction while viewing the captured image when capturing the outside scene.
Alternatively, the imaging direction of the imaging unit may be a direction that does not include the user's field of view in the imaging range. In this case, a direction different from the outside scene that the user sees through the display unit is imaged, and the captured image is displayed on the display unit. For this reason, the visual field which permeate | transmits a display part can be supplemented by imaging the range which a user cannot see as an outside scene, and displaying a captured image. Furthermore, by changing the display form of the reference image according to the distance to the captured subject, the subject in a direction that the user cannot see through the display unit is displayed in a display form corresponding to the distance to the subject. Can be shown to the user.

In the display device according to the aspect of the invention, the reference image control unit may increase the size of the reference image as the distance from the subject acquired by the distance acquisition unit is reduced. It adjusts so that it may become large, It is characterized by the above-mentioned.
According to the present invention, the reference image is displayed in a large size when the distance to the subject is short, and the difference between the angle of view of the imaging unit and the appearance of the outside scene visible to the user is likely to occur, so the user views the captured image. However, it is possible to adjust the imaging direction and the like more easily.

Further, in the display device according to the present invention, the reference image control unit may determine whether the reference image is larger than the size of the reference image when the distance from the subject acquired by the distance acquisition unit is a first distance. The reference image when the distance is a second distance closer than the first distance is enlarged.
According to the present invention, when the distance to the subject is short, the reference image displayed on the display unit is large, and when the distance to the subject is long, the reference image is small. It becomes easy and the convenience can be enhanced.

The display device may further include an imaging processing unit configured to detect the subject that satisfies a preset condition from a captured image of the imaging unit, and the reference image control unit may include the imaging processing unit. When the subject is detected, the display form of the reference image is further changed.
According to the present invention, it is possible to further change the display form of the reference image when a subject corresponding to a preset condition is detected. Here, the preset subject includes, for example, a one-dimensional code such as a barcode, a two-dimensional code such as a QR code (registered trademark), or other machine-recognizable markers. Thereby, when making a display apparatus recognize a marker etc., for example, adjustment concerning recognition operation becomes easy. The detected subject is not limited to a marker or the like. For example, when data related to feature points and feature quantities of a subject detected from a captured image is set as a condition, by detecting feature points and feature quantities corresponding to the conditions from the captured image, a three-dimensional object, a landscape, a planar image, etc. Can be detected and recognized as a subject.

Further, according to the present invention, in the display device, the reference image control unit detects distortion of the image of the subject detected by the imaging processing unit in the captured image of the imaging unit, and according to the detected distortion state. The reference image is displayed in a different display form.
According to the present invention, it is possible to notify the user of the occurrence of distortion by reflecting in the display form that the image of the subject in the captured image is distorted. In addition, for example, the display form of the reference image can be changed so as to compensate for the distortion, thereby improving the visibility of the image of the subject.

Further, the present invention is characterized in that, in the display device, the distance acquisition unit obtains a distance from the subject based on a size of the subject in a captured image captured by the imaging unit.
According to the present invention, the distance to the subject can be easily obtained.

Further, the present invention is characterized in that, in the display device, the distance acquisition unit obtains a distance to the subject based on at least two images captured by the imaging unit.
According to the present invention, the distance to the subject can be determined easily and more accurately based on the captured image.

Further, the present invention is characterized in that, in the display device, the imaging unit can capture at least two images at the same time.
According to the present invention, an image can be easily obtained in a configuration in which a distance from a subject is obtained based on two or more images.

The display device may further include a distance measurement unit that obtains a distance to the subject optically or using sound waves or ultrasonic waves, and the distance acquisition unit is measured by the distance measurement unit. Acquiring a distance from the subject.
According to the present invention, the distance to the subject can be quickly obtained.

Further, the present invention is characterized in that, in the display device, the imaging unit is arranged so that a direction corresponding to the line of sight of the user is an imaging direction.
According to the present invention, it is possible to capture a direction corresponding to the user's line-of-sight direction and display the captured image as a reference image.

In order to achieve the above object, the present invention provides a control method for a display device that includes a display unit that displays an image so that it can be seen through the outside scene and visible together with the outside scene. The distance to the subject is acquired, and the captured image is displayed as a reference image on the display unit. The display form of the reference image on the display unit is such that the reference image becomes easier to visually recognize as the distance to the subject decreases. It is characterized by adjusting to.
According to the present invention, the display device displays the reference image in an easy-to-see manner when the distance from the subject is short and the difference between the angle of view of the imaging unit and the appearance of the outside scene visible to the user is likely to occur. For this reason, the user who uses the display device can easily adjust the imaging direction while viewing the captured image. Therefore, the captured image can be displayed in an appropriate display form, and the visibility of the outside scene and the captured image can be compatible and the convenience can be enhanced.

In order to achieve the above object, the present invention is a program executable by a computer that controls a display device that includes a display unit that displays an image so that the image can be viewed through the outside scene and the outside scene. An image processing unit that causes the imaging unit to perform imaging, a distance acquisition unit that acquires a distance between the subject captured by the imaging unit, and a captured image of the imaging unit displayed on the display unit as a reference image; The display form of the reference image on the display unit is caused to function as a preview control unit that adjusts the reference image so that the reference image becomes easier to see as the distance to the subject acquired by the distance acquisition unit is closer.
According to the present invention, the display device displays the reference image in an easy-to-see manner when the distance from the subject is short and the difference between the angle of view of the imaging unit and the appearance of the outside scene visible to the user is likely to occur. For this reason, the user who uses the display device can easily adjust the imaging direction while viewing the captured image. Therefore, the captured image can be displayed in an appropriate display form, and the visibility of the outside scene and the captured image can be compatible and the convenience can be enhanced.

It is explanatory drawing which shows the external appearance structure of a head mounted type display apparatus. It is a block diagram which shows the functional structure of a head mounted display apparatus. It is a schematic diagram which shows the relative position of a camera and a user's eyes. It is a flowchart which shows operation | movement of a head mounting type display apparatus. It is explanatory drawing which shows the example of the preview display of a head mounting | wearing type display apparatus. It is explanatory drawing which shows another example of the preview display of a head mounted display apparatus. It is a block diagram which shows the principal part structure of the head mounted display apparatus in a modification.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is an explanatory diagram showing an external configuration of the head-mounted display device 100. The head-mounted display device 100 is a display device mounted on the head, and is also called a head mounted display (HMD). The head-mounted display device 100 of the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. In this specification, a virtual image visually recognized by the user with the head-mounted display device 100 is also referred to as a “display image” for convenience. Moreover, emitting image light generated based on image data is also referred to as “displaying an image”.

  The head-mounted display device 100 includes an image display unit 20 (display unit) that allows the user to visually recognize a virtual image when mounted on the user's head, and a control device (controller) 10 that controls the image display unit 20. And.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. (Imaging unit) and a microphone 63. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. Hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”.

The display driving units 22 and 24 include liquid crystal displays 241 and 242 (hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 (see FIG. 2). Details of the configuration of the display driving units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 2). Moreover, you may equip the front side of the right optical image display parts 26 and 28 with a light control board (not shown). The light guide plates 261 and 262 are formed of a light transmissive resin or the like, and guide the image light output from the display driving units 22 and 24 to the user's eyes. The light control plate is a thin plate-like optical element, and may be disposed so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. Various types of light control plates are used, such as those that have almost no light transmission, those that are nearly transparent, those that attenuate the amount of light and transmit light, and those that attenuate or reflect light of a specific wavelength. be able to. By appropriately selecting the optical characteristics (light transmittance, etc.) of the light control plate, the amount of external light incident on the right optical image display unit 26 and the left optical image display unit 28 from the outside is adjusted to facilitate visual recognition of the virtual image. You can adjust the height. In the present embodiment, a case will be described in which at least a light control plate having such a light transmittance that a user wearing the head-mounted display device 100 can visually recognize an outside scene is used. This light control plate protects the right light guide plate 261 and the left light guide plate 262, and is useful for suppressing damage to the right light guide plate 261 and the left light guide plate 262, adhesion of dirt, and the like.
The light control plate may be detachable with respect to the right optical image display unit 26 and the left optical image display unit 28, or a plurality of types of light control plates may be exchanged and may be omitted.

The imaging direction (angle of view) of the camera 61 is arbitrary, and may be a direction corresponding to the user's line of sight, or the imaging direction may be determined regardless of the user's line of sight.
In the present embodiment, first, an example in which the imaging direction of the camera 61 is set to the imaging direction of the front side direction of the head-mounted display device 100 will be described.
As shown in FIG. 1, the camera 61 is disposed at an end ER that is the other end of the right optical image display unit 26. In the configuration of FIG. 1, the photographing lens of the camera 61 is in the direction of the user's line of sight, and the camera 61 captures an outside scene that is an external scenery in a direction opposite to the user's eye side, and displays the outside scene image. get. The camera 61 of this embodiment shown in FIG. 1 is a monocular camera, but may be a stereo camera.

  In the present embodiment, the angle of view of the camera 61 is a direction in which at least a part of the outside scene in the viewing direction of the user when the head-mounted display device 100 is worn is imaged. In addition, although the angle of view of the camera 61 can be set as appropriate, the imaging range of the camera 61 is a range including the outside world that the user can visually recognize through the right optical image display unit 26 and the left optical image display unit 28. It is preferable. Furthermore, it is more preferable that the imaging range of the camera 61 is set so that the entire field of view of the user through the light control plate can be imaged.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control device 10. The connection unit 40 includes a main body cord 48 connected to the control device 10, a right cord 42, a left cord 44, and a connecting member 46. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24.

  The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. A microphone 63 is provided in the vicinity of the earphone plug 30. The earphone plug 30 to the microphone 63 are combined into one cord, and the cord branches from the microphone 63 and is connected to each of the right earphone 32 and the left earphone 34.

  It is also possible to combine the right code 42 and the left code 44 into one code. Specifically, the lead wire inside the right cord 42 is drawn into the left holding portion 23 side through the inside of the main body of the image display unit 20, and is covered with a resin together with the lead wire inside the left cord 44 to be combined into one cord. Also good.

  The image display unit 20 and the control device 10 transmit various signals via the connection unit 40. A connector (not shown) that fits each other is provided at each of the end of the main body cord 48 opposite to the connecting member 46 and the control device 10. By fitting / releasing the connector of the main body cord 48 and the connector of the control device 10, the control device 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control device 10 is a device for controlling the head-mounted display device 100. The control device 10 includes switches including an enter key 11, a lighting unit 12, a display switching key 13, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. In addition, the control device 10 includes a track pad 14 that is touched by a user with a finger.

  The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control device 10. The lighting unit 12 notifies the operation state of the head-mounted display device 100 by its light emission state. Examples of the operating state of the head-mounted display device 100 include power ON / OFF. For example, an LED (Light Emitting Diode) is used as the lighting unit 12. The display switching key 13 detects a pressing operation and outputs a signal for switching the display mode of the content video between 3D and 2D, for example.

  The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the head-mounted display device 100 by detecting a slide operation of the switch.

FIG. 2 is a functional block diagram of the head-mounted display device 100. FIG. 2 also illustrates the image supply device OA connected to the head-mounted display device 100. The head-mounted display device 100 can constitute a display system together with the image supply device OA.
As shown in FIG. 2, the control device 10 includes a control unit 140, an operation unit 135, an input information acquisition unit 110, a storage unit 120, a power supply 130, an interface 180, a transmission unit (Tx) 51 and a transmission. Part (Tx) 52.
The operation unit 135 detects an operation by the user. The operation unit 135 includes the determination key 11, the display switching key 13, the track pad 14, the luminance switching key 15, the direction key 16, the menu key 17, and the power switch 18 shown in FIG.

The input information acquisition unit 110 acquires a signal corresponding to an operation input by the user. As a signal corresponding to the operation input, for example, there is an operation input to the track pad 14, the direction key 16, and the power switch 18.
The power supply 130 supplies power to each part of the head-mounted display device 100. As the power supply 130, for example, a secondary battery can be used.

The storage unit 120 stores various computer programs. The storage unit 120 is configured by a ROM, a RAM, or the like.
The control unit 140 includes a CPU, a ROM, a RAM, and the like, and controls each unit of the head-mounted display device 100 by executing a program stored in the ROM or the storage unit 120. The control unit 140 functions as an operating system (OS) 150 that is a basic control system of the head-mounted display device 100 by executing the program. Further, the control unit 140 executes the above-described program and functions as an image processing unit 160, a distance acquisition unit 161, an imaging processing unit 163, a preview control unit 165, an audio processing unit 170, and a display control unit 190. These functions may be part of the operating system 150, or may be functions of application programs that operate on the operating system 150.

The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). Thereafter, the image processing unit 160 stores the converted digital image signal in the DRAM in the storage unit 120 frame by frame as image data (Data in the figure) of the target image. This image data is, for example, RGB data.
Note that the image processing unit 160 may execute image processing such as various tone correction processing such as resolution conversion processing, brightness and saturation adjustment, and keystone correction processing on the image data as necessary. .

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. To do. The image data Data transmitted via the transmission unit 51 is also referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left eye image data”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control device 10 and the image display unit 20.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 controls driving of the right LCD 241 by the right LCD control unit 211, driving ON / OFF of the right backlight 221 by the right backlight control unit 201, and left LCD control unit according to control signals. The left LCD 242 driving ON / OFF by 212, the left backlight 222 driving ON / OFF by the left backlight control unit 202, and the like are individually controlled. Thus, the display control unit 190 controls the generation and emission of image light by the right display driving unit 22 and the left display driving unit 24, respectively. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light.

  The display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

  The distance acquisition unit 161 acquires the distance to the subject shown in the captured image of the camera 61. Various methods can be adopted as a method by which the distance acquisition unit 161 acquires the distance. In the present embodiment, the distance acquisition unit 161 estimates the distance from the imaging element of the camera 61 or the lens of the camera 61 to the subject based on the size of the subject image in the captured image of the camera 61, and acquires this distance. To do. When the camera 61 is fixedly attached to the image display unit 20, the distance acquisition unit 161 may acquire the distance from the front surface of the image display unit 20 to the subject.

The imaging processing unit 163 controls the captured image of the camera 61 to perform imaging (capturing), and acquires image data of the captured image. The imaging processing unit 163 outputs the captured image data to the distance acquisition unit 161 to execute acquisition of the distance to the subject.
The imaging processing unit 163 has a function of detecting an image corresponding to a preset condition from the captured image of the camera 61. The preset condition is, for example, a pattern of pixel values of a plurality of pixels. Examples of the image detected by this function include so-called markers such as a two-dimensional code such as a QR code and a one-dimensional barcode. In other words, this image is an image that can be optically read and recognized by a computer and from which the computer can acquire information. The condition regarding the image is stored in the storage unit 120, for example. When the imaging processing unit 163 detects an image corresponding to the condition stored in the storage unit 120 from the captured image of the camera 61, the imaging processing unit 163 performs a preset process on the image. In the present embodiment, the storage unit 120 stores the conditions for detecting the marker, and the imaging processing unit 163 performs processing for detecting the marker from the captured image of the camera 61 and acquiring information represented by the marker. .

  The subject detected by the imaging processing unit 163 is not limited to a marker or the like. For example, when data relating to feature points and feature amounts of a subject to be detected is set as a condition and stored in the storage unit 120, the imaging processing unit 163 detects the corresponding feature points and feature amounts from the captured image data. Thus, it is possible to detect and recognize a three-dimensional object, a landscape, a planar image, etc. as a subject. In this case, when a subject such as a specific background or a specific three-dimensional object is detected from the captured image, processing described later is performed in the same manner as when a marker is detected.

  The head-mounted display device 100 can execute a plurality of operation modes including at least a camera mode that uses the camera 61 and a normal operation mode that does not use the camera 61. In this embodiment, an example in which the normal mode and the camera mode are switched and executed will be described.

  In the camera mode, the imaging processing unit 163 operates the camera 61 to perform imaging and acquire captured image data. In the camera mode, under the control of the imaging processing unit 163, the camera 61 continuously performs imaging at a preset cycle, and the imaging processing unit 163 acquires captured image data every time one frame is captured. The operation of causing the camera 61 to capture an image in this way may be the same as when moving images are captured by the camera 61.

In the camera mode, the imaging processing unit 163 executes imaging with the camera 61 and acquires captured image data of the camera 61. This operation is always executed by the imaging processing unit 163 in the camera mode even when the shutter operation or the recording operation is not performed on the operation unit 135. Further, the imaging processing unit 163 may acquire captured image data of the camera 61 when a predetermined condition is satisfied in the camera mode. The predetermined condition includes, for example, an operation of instructing the start of imaging of the camera 61, which is different from the shutter operation and the recording operation.
In the camera mode, the imaging processing unit 163 temporarily stores captured image data captured by the camera 61 in the storage unit 120 as preview data 121. The preview control unit 165 reads the preview data 121 and causes the display control unit 190 to display it. This display is called a preview display. The preview display displays the captured image data of the camera 61 while the shutter operation or the recording operation is not performed, thereby using the angle of view (imaging range) captured by the camera 61, the imaging direction, the state of the captured image, and the like. The function to inform the person.
As described above, the preview control unit 165 can execute the preview display regardless of either the shutter operation or the recording operation. That is, the preview and preview display in the following description are not limited to the display performed before the operation executed in accordance with the shutter operation or the recording operation. It simply refers to the function of displaying the image data captured by the camera 61 on the image display unit 20 without any shutter operation or recording operation. Of course, it is possible for the user to perform a shutter operation or a recording operation after viewing the contents of the preview display.

When a shutter operation or a recording operation is performed on the operation unit 135, the imaging processing unit 163 stores the preview data 121 in the storage unit 120 as captured image data. The preview data 121 is temporarily stored data, and may be erased or overwritten every predetermined time, for example. The captured image data stored in the storage unit 120 is deleted when a deletion instruction is detected by the operation unit 135.
Note that the configuration is not limited to the configuration in which the preview control unit 165 reads the preview data 121 and displays the preview data. For example, the imaging processing unit 163 outputs the captured image data of the camera 61 to the preview control unit 165, and the captured image data. The preview control unit 165 may display a preview. In this case, the captured image data is temporarily stored in a RAM (not shown) constituting the control unit 140 but is not stored in the storage unit 120.
In the following description, all data that the preview control unit 165 displays as a preview is called captured image data. The captured image data may be either preview data 121 read by the preview control unit 165 from the storage unit 120 or data output by the imaging processing unit 163 to the preview control unit 165.

  In the camera mode, in order to make the preview display easy to see, the head-mounted display device 100 does not display an image such as content input from the image supply device OA described later. For this reason, the user sees the above-described preview display together with the outside scene seen through the image display unit 20.

The preview control unit 165 acquires captured image data of the camera 61 from the imaging processing unit 163 during execution of the camera mode, and causes the display control unit 190 to display a preview image based on the captured image data. The preview control unit 165 sets the display size and display position of the image displayed by the display control unit 190.
During execution of the camera mode, the image display unit 20 performs a preview display under the control of the preview control unit 165. In the camera mode, when recording or imaging execution is instructed by the operation of the control device 10, the imaging processing unit 163 stores the captured image data of the camera 61 in the storage unit 120. Further, when recording or imaging execution is instructed, the camera 61 may perform imaging at a higher resolution and / or higher frame rate than the preview display. In this case, by suppressing the data amount of the captured image data used for the preview display, the imaging processing unit 163 and the preview control unit 165 can reduce the processing load for controlling the preview display, and record or capture a high-quality image. Can record.
In the camera mode, when the camera 61 captures the above-described marker or the like, processing based on information such as the marker is executed.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker in the left earphone 34 (not shown) connected to the connecting member 46. (Not shown). For example, when the Dolby (registered trademark) system is adopted, processing on the audio signal is performed, and different sounds with different frequencies or the like are output from the right earphone 32 and the left earphone 34, for example. In addition, the voice processing unit 170 acquires the voice collected and input by the microphone 63, converts it into digital voice data, and performs processing related to the voice. For example, the speech processing unit 170 performs speaker recognition that identifies a person who speaks for each voice by recognizing voices of a plurality of persons separately by extracting and modeling features from the acquired speech. May be.

A three-axis sensor 113, a GPS 115, and a communication unit 117 are connected to the control unit 140. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 can detect the detection value of the triaxial sensor 113 and detect the movement and the direction of movement of the control device 10.
The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and obtains the current position of the control device 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. Further, the GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140 of the control device 10.
The communication unit 117 performs wireless data communication conforming to a wireless LAN (WiFi (registered trademark)) or Bluetooth (registered trademark) standard.

The interface 180 is an interface for connecting various image supply apparatuses OA that are content supply sources to the control apparatus 10. The content supplied by the image supply device OA includes a still image or a moving image, and may include sound. Examples of the image supply device OA include a personal computer (PC), a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.
Here, it is also possible to connect the image supply device OA to the control device 10 by a wireless communication line. In this case, the image supply device OA performs wireless communication with the communication unit 117 and transmits content data using a wireless communication technique such as Miracast (registered trademark).

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a camera 61. And a vibration sensor 65 and a nine-axis sensor 66.

  The vibration sensor 65 is configured using an acceleration sensor, and is arranged inside the image display unit 20 as shown in FIG. In the example of FIG. 1, the right holding unit 21 is built in the vicinity of the end ER of the right optical image display unit 26. When the user performs an operation of hitting the end ER (knock operation), the vibration sensor 65 detects vibration caused by this operation and outputs the detection result to the control unit 140. Based on the detection result of the vibration sensor 65, the control unit 140 detects a knocking operation by the user.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, the movement of the user's head is detected. For example, the control unit 140 can determine the orientation of the image display unit 20 from the movement of the user's head detected by the 9-axis sensor 66, and can estimate the user's line-of-sight direction.

  The right display driving unit 22 includes a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, a right LCD control unit 211 that functions as a display element, and a right An LCD 241 and a right projection optical system 251 are provided. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as display elements. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control device 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data1 input through the reception unit 53. . The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The right projection optical system 251 and the right light guide plate 261 are collectively referred to as “light guide unit”.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, a left LCD control unit 212 and a left that function as a display element. An LCD 242 and a left projection optical system 252 are provided. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as display elements. The left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are also collectively referred to as “image light generation unit”. The left projection optical system 252 is configured by a collimating lens that converts the image light emitted from the left LCD 242 into a light beam in a parallel state. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the image light along a predetermined optical path. The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “light guide unit”.

  In the head-mounted display device 100, the camera 61 captures an image in the camera mode, and operates using the captured image data. As shown in FIG. 1, the camera 61 is provided in the image display unit 20 attached to the user's head. For this reason, the imaging direction of the camera 61 is the same as the user's line-of-sight direction, which is convenient because the direction in which the user is looking is captured. However, strictly speaking, the imaging direction of the camera 61 and the user's line-of-sight direction are separated from each other. This difference in position may affect the captured image. This point will be described.

  FIG. 3 is a schematic diagram showing the relative position between the imaging direction of the camera 61 provided in the head-mounted display device 100 and the user's line of sight in plan view. FIG. 3 shows a wearing state in which the user wears the image display unit 20. Reference sign CR in the figure indicates the angle of view of the camera 61, and reference sign VR indicates the field of view of the user. An arrow AX is an axis indicating the imaging direction of the camera 61, and specifically indicates the optical axis direction of the imaging lens of the camera 61. The arrow RS indicates the line-of-sight direction when the right eye RE faces the front, and the arrow LS indicates the line-of-sight direction when the left eye LE faces the front.

  As described above, since the right optical image display unit 26 and the left optical image display unit 28 transmit external light, the user can see the outside scene through the image display unit 20. On the other hand, the camera 61 is disposed forward in the image display unit 20, and the imaging direction AX of the camera 61 is substantially parallel to the line-of-sight directions RS and LS. For this reason, since the camera 61 captures substantially the same direction as the direction the user is viewing, the range (imaging range) that falls within the angle of view CR of the camera 61 and the visual field VR of the user substantially overlap.

  For example, when the user is looking at the subject 301 at a distant place, the subject 301 is included in the captured image of the camera 61 because the subject 301 is included in the angle of view CR of the camera 61. As described above, when the user images the subject 301 located far away, the difference between the visual field VR of the user and the captured image of the camera 61 is small. This is because the position of the subject 301 in the visual field VR and the position of the subject 301 in the angle of view CR are substantially the same position.

On the other hand, when imaging a subject close to the image display unit 20, the difference between the angle of view CR of the camera 61 and the visual field VR of the user becomes large, so that the captured image and the outside scene visible to the user are The difference is not negligible.
For example, the subject 302 shown in FIG. 3 is located near the image display unit 20, and the distance L2 from the image display unit 20 to the subject 302 is shorter than the distance L1. The distance L2 is several to several tens of times the distance D between the imaging direction AX and the line-of-sight direction RS, and is less than a hundred times. In this example, the subject 302 is in the field of view VR, but at the angle of view CR, it is positioned at the end and may exit from the angle of view CR. Thus, since the position of the subject 302 in the visual field VR and the position of the subject 302 in the angle of view CR are different, the captured image of the camera 61 is clearly different from the visual field VR seen by the user.
Therefore, when shooting the subject 302, the user can move and adjust the image display unit 20 so that the imaging direction AX faces the subject 302 or the subject 302 is included in the angle of view CR. desirable.

The difference between the case of imaging the subject 301 and the case of imaging the subject 302 is that the distance D between the imaging direction AX of the camera 61 and the user's line of sight RS and LS, and the distance L1 to the subjects 301 and 302, The reason is that the ratio with L2 is different. Further, it is influenced by whether the angle of view of the camera 61 is a wide angle or a narrow angle, and the width of the visual field VR of the user through the right optical image display unit 26 and the left optical image display unit 28 is also affected. Affected. However, an element having a particularly great influence is the distance from the image display unit 20 to the subjects 301 and 302.
The distances L1 and L2 indicate distances from the subjects 301 and 302 to the front surface of the image display unit 20, but may be distances to the light receiving unit of the camera 61.

  The head-mounted display device 100 can recognize and process information indicated by the marker when the camera 61 images a marker or the like by the function of the imaging processing unit 163 or the like described above. Specifically, in a system that provides information for work support to the head-mounted display device 100 when a user performs a work using a marker, the system is provided at or near a work place such as a work desk. A marker is attached. Since the user needs to cause the camera 61 to image the marker or the like, the user is conscious that the image display unit 20 is directed in the direction of the marker. However, the marker is different due to the difference between the visual field VR of the user and the angle of view CR of the camera 61 It is difficult to consider that no image is taken.

  In order to solve this problem, for example, the user's eyes RE, LE and the incident light of the camera 61 coincide with each other so that the difference between the angle of view CR of the camera 61 and the visual field VR of the user becomes small. It is possible to do. However, if the imaging element (light receiving element) of the camera 61 is arranged in front of the user's eyes RE and LE, the visibility of the outside scene is lowered. Further, for example, if an optical member such as a half mirror is used to guide the light incident on the user's eyes RE and LE to the image sensor of the camera 61 to capture an image, the apparatus becomes large. Thus, there was no highly feasible solution.

Therefore, the head-mounted display device 100 of the present embodiment has a function that allows the user to easily confirm the angle of view CR of the camera 61 by a preview display displayed in the camera mode. In this preview display, the preview control unit 165 has a function of adjusting the display form of the preview display in the image display unit 20.
The display form of the preview display as the reference image includes the display size, the display position in the displayable area of the image display unit 20, the display color tone, the display brightness, the shape of the preview display, and the like. Hereinafter, as an example of the display mode, the display size of the preview display will be described as an example.

The preview control unit 165 changes the display size of the preview display displayed by the image display unit 20 according to the distance to the subject. Specifically, when the distance from the image display unit 20 to the subject is short, the difference between the angle of view CR and the visual field VR is likely to appear, so the display size of the preview display is increased. Further, if the display size of the preview display is large, it may be possible to hinder the visual recognition of the outside scene. However, the head-mounted display device 100 according to the present embodiment reduces the size of the preview display when the distance to the subject is large. To do. For this reason, when the user can easily check the angle of view CR of the camera 61 and the necessity of checking the angle of view CR is small, the size of the preview display is reduced and the field of view is not obstructed. Further, the head-mounted display device 100 may stop the preview display when the distance to the subject is large.
Hereinafter, the preview display in the head-mounted display device 100 will be described in detail.

FIG. 4 is a flowchart showing the operation of the head-mounted display device 100.
FIG. 5 shows an example of preview display of the head-mounted display device 100, and FIG. 6 is an explanatory diagram showing another example of preview display.
When the control unit 140 starts the camera mode (step S11), the imaging processing unit 163 causes the camera 61 to start imaging and acquires captured image data (step S12).

Here, the distance acquisition unit 161 detects the image of the subject in the captured image of the camera 61, and acquires the distance to the detected subject (step S13). In the present embodiment, the distance acquisition unit 161 estimates the distance from the lens of the camera 61 to the subject based on the size of the subject image in the captured image of the camera 61.
The preview control unit 165 determines the size of the preview display based on the distance acquired by the distance acquisition unit 161 (step S14), and causes the display control unit 190 to display an image based on the captured image data of the camera 61 with the determined size. (Step S15). Here, the preview control unit 165 may stop the preview display when the distance acquired by the distance acquisition unit 161 is farther than a preset distance. Furthermore, when the distance acquired by the distance acquisition unit 161 is unknown, the preview control unit 165 may determine the display size set in association with the distance unknown.

FIG. 5A shows an example of a preview display when a distant subject is imaged, and FIG. 5B shows an example of a preview display when an object close to the image display unit 20 is imaged. VR in the figure indicates the entire visual field visible to the user. Further, for convenience of understanding, it is assumed that the displayable area in which the image display unit 20 can display an image matches the entire visual field VR.
The preview image PV shown in FIGS. 5A and 5B is displayed on both the right LCD 241 and the left LCD 242, and the right light guide plate 261 and the left light guide plate 262 cause the image light of the preview image PV to be seen by both eyes of the user. Incident. In the present embodiment, the same preview image PV is displayed on the right LCD 241 and the left LCD 242, but for example, an image having a parallax is displayed on the right LCD 241 and the left LCD 242, and the preview image PV of the stereoscopic image is displayed. May be.

  In the example of FIG. 5A, a preview image PV is displayed in the upper left corner of the visual field VR visible to the user. The size of the preview image PV is 1/3 or less in both the vertical and horizontal directions of the visual field VR. In the example of FIG. 5A, since the distance of the subject 310 shown in the image captured by the camera 61 is long, the display size of the preview image PV is small corresponding to this distance.

  On the other hand, in the example of FIG. 5B, the size of the preview image PV is about ½ of the vertical direction and the horizontal direction of the visual field VR. In the example of FIG. 5B, since the distance to the subject 311 shown in the captured image of the camera 61 is short, the display size of the preview image PV is large corresponding to this distance.

There are various methods by which the preview control unit 165 determines the display size of the preview image PV. For example, a configuration in which the size at which the display control unit 190 displays the preview image PV is determined in advance can be given. In this case, the display size and display position of the preview image PV are determined by the pixel coordinates on the right LCD 241 and the left LCD 242. A plurality of display sizes are determined in advance, and each display size is associated with a distance to the subject acquired by the distance acquisition unit 161. The distance to the subject may be divided into a plurality of stages, and one display size may be associated with each stage.
Further, for example, the preview control unit 165 may calculate the display size by calculating the distance to the subject acquired by the distance acquisition unit 161 using an arithmetic expression stored in the storage unit 120.
In any method, the preview control unit 165 determines the display size so that the display size of the preview image PV increases as the distance to the subject decreases.

  Returning to FIG. 4, after the preview display is started, the imaging processing unit 163 detects data corresponding to the feature of the marker image from the captured image data of the distance acquisition unit 161, and determines whether or not it has been detected. (Step S16). In step S16, the imaging processing unit 163 may detect the marker from the captured image data displayed as the preview image in step S15, or may detect from another captured image data.

When marker image data is detected from the captured image data, the imaging processing unit 163 determines whether or not the detected marker image data can be read (step S17). When the image processing unit 163 recognizes the data of the marker image and can acquire the information indicated by the marker, the imaging processing unit 163 determines that the information can be read (step S17; YES), and acquires the information (step S18). The control unit 140 executes data processing based on the information acquired from the marker by the imaging processing unit 163 (step S19), and ends this processing.
Note that the preview control unit 165 may change the display form of the preview image when the data of the marker image is detected. For example, symbols, characters, etc. indicating marker detection may be displayed overlapping the preview display.

  If the imaging processing unit 163 cannot read the detected marker (step S17; NO), the preview control unit 165 detects distortion of the marker image (step S20). For example, the imaging processing unit 163 performs a reading process including, for example, an error correction process on the marker detected in step S16, and determines that reading is not possible when valid data is not obtained. If information cannot be read even though it is detected as a marker, the situation in which the camera 61 images the marker may be inappropriate. Specifically, distortion of the image of the marker in the captured image, brightness and contrast problems of the captured image, a state where a part of the marker is hidden by an obstacle, and the like cause unreadable. In such a case, the head-mounted display device 100 distorts and displays the preview image in accordance with the distortion of the marker, and prompts the user to optimize the imaging direction of the camera 61.

  After detecting the marker distortion, the preview control unit 165 changes the shape of the image to be preview-displayed in accordance with the detected distortion (step S21), and updates the preview display. Thereafter, the control unit 140 determines whether or not there is an instruction to end the camera mode (step S22). If the camera mode is not ended, the process returns to step S12.

FIG. 6A shows an example in which the marker enters the field of view, and shows an example in which the marker 315 is visually recognized together with the tool on the work table in the visual field VR of the user. In this example, a QR code is used as a marker. The marker 315 is printed on paper, for example, and placed on the work table together with the tool. This work table is located at a position close to the image display unit 20 to the extent that the user can reach it.
FIG. 6B shows a captured image 320 of the camera 61. Since the distance to the work table, which is the subject, is short, the user's visual field VR shown in FIG. 6A and the captured image 320 of the camera 61 are different. Since the work table is located obliquely to the left when viewed from the camera 61, the marker image 321 is distorted in the oblique direction in the captured image 320. The marker image 321 can be detected as a marker because the corner has a symbol specific to QR. However, the marker image 321 cannot be decoded by the QR code decoding algorithm due to distortion.

In this case, as described in steps S20 to S21, the preview control unit 165 causes the preview image itself to be distorted and displayed.
FIG. 6C shows an example in which the preview image PV is deformed. The preview image PV in FIG. 6C has a shape distorted in an oblique direction, imitating the shape of the marker image 321 in the captured image 320 as shown in FIG. 6B. By deforming the preview image PV itself, it is possible to promptly notify the user that the captured image 320 of the camera 61 is distorted.
Here, the preview control unit 165 may change the shape of the preview image PV so as to compensate for the distortion of the marker image 321. In this case, the distortion of the marker image 321 is reduced, and the marker image 321 is displayed in a rectangle or the like that is the original marker shape. In this case, since it becomes easy for the user to visually recognize the marker, for example, the user can be made aware that the marker is reflected in the captured image.

In addition, when the imaging processing unit 163 detects the marker image in step S16 and the preview control unit 165 displays the symbols, characters, and the like superimposed on the preview display, if the preview image PV is deformed, the symbols, characters, etc. are displayed. It is possible to deform. In this case, the preview control unit 165 may deform only the symbols and characters according to the distortion of the marker detected in step S20 before deforming the preview image PV. In this case, since the distortion of the marker and the distortion of the symbol and the character are aligned, when the preview image PV is subsequently deformed, both the marker, the symbol and the character are easy to see.
Furthermore, after detecting the distortion of the marker in step S20, the preview control unit 165 may display a symbol, a character, or the like indicating a movement necessary for correcting the distortion so as to overlap the preview image PV. For example, when the imaging direction of the camera 61 and the distance between the marker and the camera 61 can change the image of the marker with a small distortion, the direction in which the camera 61 is moved may be guided by a symbol or character such as an arrow. The user can recognize the marker by moving the head-mounted display device 100 in accordance with the symbols and characters. In this case, the preview image PV does not have to be deformed in step S21. For example, it is preferable to return to step S12 and acquire the captured image data again.

  Furthermore, after executing the data processing based on the marker in step S19, the control unit 140 may determine whether to move to step S22 and end the camera mode. If the camera mode is not ended, step S12 is performed. You may return to Moreover, you may return to step S12 directly after step S19. In this case, the camera mode can be continued after the marker is recognized and the processing based on the marker is performed. For this reason, the head-mounted display device 100 can continuously read other markers.

  As described above, the imaging processing unit 163 may detect a subject such as a specific background or a specific three-dimensional object from the captured image data, in addition to the marker. In this case, the processing after step S17 may be performed in the same manner as when the imaging processing unit 163 detects the subject and detects the marker in step S16.

As described above, the head-mounted display device 100 according to the embodiment to which the present invention is applied includes the image display unit 20 that displays an image so as to transmit the outside scene and be visible together with the outside scene, and the visual field of the user. An imaging processing unit 163 that captures a direction including at least a part by the camera 61, a distance acquisition unit 161 that acquires a distance from the subject imaged by the camera 61, and a preview of the captured image of the camera 61 as a reference image A preview control unit 165 that displays the image on the image display unit 20 and adjusts the display form of the reference image on the image display unit 20 so that the reference image becomes easier to see as the distance to the subject acquired by the distance acquisition unit 161 is closer. Prepare. For this reason, the user can recognize the angle of view of the captured image and adjust the imaging direction of the imaging apparatus to an appropriate direction.
In addition, the preview control unit 165 displays the reference image as a preview, and the preview image PV when the difference between the angle of view of the camera 61 and the appearance of the outside scene is hardly generated is relatively small, and thus the visibility of the outside scene is impaired. Hateful. Thus, by displaying a preview of the captured image of the camera 61 using an appropriate method, the visibility of the outside scene and the captured image can be compatible, and convenience can be enhanced.

  The preview control unit 165 also has a second distance that is closer to the subject than the first distance than the size of the preview when the distance to the subject acquired by the distance acquisition unit 161 is the first distance L1. The preview for L2 is enlarged. For this reason, the preview displayed on the image display unit 20 becomes larger as the distance from the subject is closer and the difference between the angle of view of the camera 61 and the appearance of the outside scene visible to the user is more likely to occur. Can be appropriately displayed together with the outside scene to enhance convenience.

Further, the imaging processing unit 163 detects a subject that satisfies a preset condition from the captured image of the camera 61. The preview control unit 165 changes the display format of the preview when the subject is detected by the imaging processing unit 163. Thereby, when a marker or the like is detected from the captured image of the camera 61, the display form of the preview can be changed, for example, to make it easy to see the subject or to notify that the subject meets a preset condition. Accordingly, it is possible to improve convenience when the head-mounted display device 100 operates by recognizing the marker from the captured image.
Further, the preview control unit 165 detects distortion of the image of the subject detected by the imaging processing unit 163 in the captured image of the camera 61, and displays the preview in a display form corresponding to the detected distortion state. For this reason, since the fact that the image of the subject in the captured image is distorted is reflected in the display form, it is possible to notify the user of the occurrence of distortion. Further, if the image display form is adjusted so as to compensate for distortion, for example, the visibility of the image of the subject can be improved.

Further, since the distance acquisition unit 161 obtains the distance to the subject based on the size of the subject in the captured image captured by the camera 61, the distance to the subject can be easily obtained. For example, when the subject image occupies most of the captured image, the subject can be considered to be near the camera 61.
As described above, the distance acquisition unit 161 extracts the subject image from the captured image, and estimates the distance based on the ratio of the area occupied by the subject image and the ratio between the subject image and the entire captured image size.
In this configuration, the size of the preview display is substantially determined based on the ratio of the area occupied by the subject image and the size ratio of the subject image and the entire captured image by extracting the subject image from the captured image. It can be said that.

  In the example of FIGS. 5A and 5B, the preview control unit 165 changes the display size of the preview image PV in accordance with the distance obtained by the distance acquisition unit 161. The function of the preview control unit 165 is not limited to this, and the display position of the preview image PV may be changed according to the distance obtained by the distance acquisition unit 161. For example, the display position of the preview image PV is closer to the center of the visual field VR as the distance to the subject is smaller (closer), and the display position of the preview image PV is closer to the outer edge of the visual field VR as the distance to the subject is larger (far). You may do it. In this case, when the distance to the subject is smaller than the reference, the preview image PV is arranged at the center of the visual field VR, and when the distance to the subject is larger than the reference, the preview image PV is arranged at the end of the visual field VR. .

Further, the preview control unit 165 may change the display color tone of the preview image PV according to the distance obtained by the distance acquisition unit 161. The display color tone corresponds to the shade of the color of the image of the preview image PV. If the display color tone is dark, the preview image PV can be clearly seen in the visual field VR, and if the display color tone is light, the preview image PV is thin and transmits the preview image PV. Then you can see the outside scene.
For example, the preview control unit 165 may darken the display color tone of the preview image PV as the distance to the subject is smaller (closer), and may decrease the display color tone of the preview image PV as the distance to the subject is greater (far). . In this case, the display color tone may be set to the highest density when the distance to the subject is smaller than the reference, and the color tone of the preview image PV may be set to the lowest density when the distance to the subject is greater than the reference.

The preview control unit 165 may change the display brightness of the preview image PV according to the distance obtained by the distance acquisition unit 161. For example, the preview control unit 165 increases (brightens) the brightness of the preview image PV as the distance to the subject is smaller (closer), and decreases (darkens) the brightness of the preview image PV as the distance to the subject is greater (farther). May be. In this case, the display color tone may be set to the highest luminance when the distance to the subject is smaller than the reference, and the color tone of the preview image PV may be set to the lowest luminance when the distance to the subject is larger than the reference.
The preview control unit 165 may change the shape of the reference image according to the distance according to the distance obtained by the distance acquisition unit 161. For example, the preview control unit 165 may transform the shape of the preview image PV into a circular shape, a rectangular shape, a triangular shape, or another polygonal shape corresponding to the distance to the subject obtained by the distance acquisition unit 161 in a stepwise manner. . Further, when the distance to the subject is smaller than the reference, the shape of the preview image PV is circular, and when the distance to the subject is larger than the reference, the color of the preview image PV is rectangular, and the distance before the subject is between the reference. In such a case, it may be considered that the shape is a combination of a rectangle and a circle.
In any of the above examples, the preview control unit 165 changes the display form so that the visibility of the preview image PV increases as the distance to the subject is shorter, and is easier to see. More preferably, when the distance to the subject is short, the display form is changed so that the visibility of the preview image PV is higher than the outside scene seen through the image display unit 20.
Furthermore, the preview control unit 165 not only changes the above-described display size, display position, display color tone, luminance, shape, and the like as the display form of the preview image PV corresponding to the distance obtained by the distance acquisition unit 161, These changes may be combined. For example, both the display size and the display position may be changed, and the display luminance may be further changed.

In addition, when the imaging processing unit 163 detects a marker in step S16 (FIG. 4), the preview control unit 165 may change the display form of the preview image PV. That is, when the imaging processing unit 163 detects a marker while the preview image PV is displayed, the preview control unit 165 may change the display form of the preview image PV. In this case, by changing the display form of the preview image PV, it is possible to notify the user that the head-mounted display device 100 has detected the marker.
For example, the preview control unit 165 may change the display size, display position, display color tone, brightness, shape, and the like of the preview image PV. In this case, the preview control unit 165 changes the display form so that the visibility of the preview image PV is higher than before the marker is detected, and the preview image PV is easily seen by the user. As described above, the display mode is changed so that the preview image PV is easy to see. The display size is increased, the display position is changed to a position close to the center, the display color is darkened, the brightness is increased, and the shape is circular. It is a change such as getting closer. Moreover, you may combine the change of a some display form.

  As shown in FIG. 3, the camera 61 has an imaging direction AX of the camera 61 corresponding to the user's line of sight, and specifically, parallel to the line-of-sight directions RS and LS. Has been placed. In this configuration, since the user's line-of-sight direction corresponds to the imaging direction, the need to view the preview image is small when the distance to the subject is long, and the need to view the preview image as the distance to the subject is short Is expensive. As described above, the display size when the preview of the captured image is displayed together with the outside scene can be adjusted according to the necessity of the preview, and the convenience can be enhanced.

  In the above embodiment, in the head-mounted display device 100, the distance acquisition unit 161 acquires the distance based on the captured image of the camera 61. However, the method of acquiring the distance is not limited to this, and various modifications are possible. Is possible.

  FIG. 7 is a block diagram illustrating a main configuration of a head-mounted display device according to a modification. 7A shows a configuration including a stereo camera 64 as a first modification, and FIG. 7B illustrates a configuration including a distance meter 67 as a second modification. 7A and 7B show the main configuration of the head-mounted display device 100, and the components that are the same as those of the head-mounted display device 100 shown in FIG. 2 are not shown.

In the first modification, an image display unit 20A including a stereo camera 64 is used instead of the camera 61 included in the image display unit 20. The image display unit 20 </ b> A has the same configuration as the image display unit 20 except for the camera 61. Moreover, the control apparatus 10 is the same as the structure shown in FIG.
The stereo camera 64 includes a plurality of imaging lenses (not shown) and a plurality of imaging elements (not shown) corresponding to the respective imaging lenses, and images are taken simultaneously by the plurality of imaging elements to have a plurality of images having parallax. Output an image. The imaging lenses of the stereo camera 64 are distributed and arranged in the direction in which the user's right eye RE and left eye LE are aligned. For example, there is a configuration in which one imaging lens is arranged at the end ER (FIG. 1) and another imaging lens is arranged at the end EL.

The imaging processing unit 163 controls the stereo camera 64 to execute imaging, and acquires a plurality of captured image data. The distance acquisition unit 161 calculates the distance to the subject by detecting the parallax of the plurality of captured image data acquired by the imaging processing unit 163.
According to this configuration, the distance to the subject can be calculated easily and more accurately based on the parallax of the stereo camera.

In this configuration, the preview control unit 165 displays any one of a plurality of captured image data acquired by the imaging processing unit 163 as a preview image. In this case, for example, one captured image data obtained by capturing an outside scene close to the visual field that can be seen through the image display unit 20 from the plurality of captured image data may be selected and displayed. Further, for example, one captured image data with less distortion of the subject image in the captured image may be selected and displayed.
The preview control unit 165 may newly generate image data for preview display from a plurality of captured image data acquired by the imaging processing unit 163. Alternatively, a plurality of captured image data acquired by the imaging processing unit 163 may be displayed on each of the right LCD 241 and the left LCD 242 to display a stereoscopic image preview.

Further, the present invention is not limited to the configuration using the stereo camera 64. A camera that has three or more imaging elements and outputs three or more captured image data may be provided instead of the stereo camera 64. Further, two or three or more cameras 61 are provided in the head-mounted display device 100, and the plurality of cameras 61 perform imaging simultaneously, and use the captured image data output by each camera 61, You may detect the distance with a to-be-photographed object.
In addition, when the camera 61 or the stereo camera 64 captures images a plurality of times, a plurality of captured image data captured may be used. In this case, the distance acquisition unit 161 detects the distance to the subject based on a plurality of captured image data having different imaging times. In this case, since the process of detecting the distance to the subject based on the captured image data is performed a plurality of times, the detection accuracy can be improved as compared with the case where one captured image data is used.
Furthermore, when the camera 61 or the stereo camera 64 performs imaging a plurality of times at different positions, parallax due to the difference in imaging position occurs in the captured image data. The distance to the subject may be calculated using this parallax. In this case, it is only necessary that the position of the camera 61 or the stereo camera 64 at the time of shooting or the change in position is clear. For example, the position of the camera 61 or the stereo camera 64 at the time of shooting may be obtained based on the output values of the 3-axis sensor 113, the GPS 115, or the 9-axis sensor 66, and the imaging direction may be obtained together.

In addition, the imaging directions of the camera 61 of the embodiment and the stereo camera 64 of the modification are both set so as to capture a range including at least a part of the user's visual field. The present invention is not limited to this, and may be set to image a range that does not include the user's field of view, for example. That is, the imaging direction of the camera 61 or the stereo camera 64 that is the imaging unit of the present invention may be a direction that does not include the user's field of view in the imaging range.
In this case, the camera 61 or the stereo camera 64 captures a direction different from the outside scene viewed by the user through the image display unit 20, and the preview control unit 165 displays the captured image as the preview image PV. In this configuration, a captured image in a range that the user cannot see through the image display unit 20 is displayed, so that the user's visual field can be supplemented.
Further, with such a configuration, as described above, the display form of the preview image PV may be changed according to the distance from the captured subject. In this case, a subject in a direction that the user cannot see through the image display unit 20 can be shown to the user in a display form corresponding to the distance to the subject. Specifically, when the image display unit 20 is positioned in front of the user's eyes, the rear of the user can be imaged with the camera 61 or the stereo camera 64. Then, when the subject behind the user is closer to the user than the predetermined distance, the usage method of alerting the user by changing the display form of the preview image PV can be realized. In addition, when the distance to the subject is equal to or greater than a predetermined distance, the display form of the preview image PV may be changed so that the visibility of the preview image PV is lowered to make it difficult for the user to pay attention.

In the second modification, in addition to the configuration of the image display unit 20 in FIG. 2, an image display unit 20 </ b> B including a distance meter 67 (ranging unit) is used. Moreover, the control apparatus 10 is the same as the structure shown in FIG.
The distance meter 67 is a device that detects a distance to an object using laser light or ultrasonic waves. For example, a light source of laser light and a light receiving unit that receives reflected light of the laser light emitted from the light source are provided, and the distance to the object is detected based on the light receiving state of the laser light.
In addition, the distance meter 67 may be an ultrasonic distance meter. In this case, the distance meter 67 includes a sound source that emits an ultrasonic wave and a detection unit that detects the ultrasonic wave reflected on the object, and reflects the reflected ultrasonic wave. The distance to the object is detected based on the sound wave.
Furthermore, the distance meter 67 may be configured by combining a distance meter using laser light and a distance meter using ultrasonic waves.

In this configuration, the distance acquisition unit 161 causes the distance meter 67 to perform distance measurement at the timing when the imaging processing unit 163 acquires captured image data from the camera 61, and acquires data indicating the measurement result from the distance meter 67. And output to the preview control unit 165.
According to this configuration, the distance to the subject can be obtained quickly and more accurately.

  The method by which the head-mounted display device 100 determines the distance to the subject is not limited to the configuration of the above-described embodiment and each modification. For example, the current position detected by the GPS 115 included in the head-mounted display device 100 Information may be used. Specifically, when the current position of the subject can be detected, the distance between the subject and the head-mounted display device 100 may be obtained based on the position of the subject and the position detected by the GPS 115. In this case, the subject includes a communication unit, the subject communication unit and the communication unit 117 are connected to each other, the position detected by the GPS 115 is transmitted to the subject, and the subject and the head-mounted display device 100 are connected to each other. The distance may be obtained and the obtained distance may be transmitted. Alternatively, the position of the subject may be transmitted to the head-mounted display device 100, and the distance acquisition unit 161 may obtain the distance based on the position of the subject and the position detected by the GPS 115. This operation may be executed when the subject is detected from the captured image data.

  In addition, this invention is not restricted to the structure of the said embodiment and each modification, It can implement in a various aspect in the range which does not deviate from the summary. For example, one of the direction key 16 and the track pad 14 provided in the control device 10 is omitted, or in addition to the direction key 16 or the track pad 14 or in place of the direction key 16 or the track pad 14, an operation stick or the like. An operation interface may be provided. Moreover, the control apparatus 10 is a structure which can connect input devices, such as a keyboard and a mouse | mouth, and is good also as what receives an input from a keyboard or a mouse | mouth.

  As the image display unit, instead of the image display unit 20 worn like glasses, another type of image display unit such as an image display unit worn like a hat may be adopted. In the above embodiment, the head-mounted display device 100 is a binocular optical transmission type. However, the present invention can be applied to other types of head-mounted display such as a video transmission type and a monocular type. The same applies to the apparatus. Further, for example, it may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Moreover, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet, or may be a head-up display (HUD) used for a windshield of an automobile. Further, for example, a binocular-type hand held display (Hand Held Display) that the user uses with both hands may be employed as the image display unit 20 of the present application. Furthermore, the user's eyeball forms an image on the retina such as a so-called contact lens type display that is worn on the user's eyes (for example, on the cornea) and an implant type display that is embedded in the eyeball. The display may be the image display unit 20.

Furthermore, in the above embodiment, the image display unit 20 and the control device 10 are separated and connected by way of the connection unit 40, but the control device 10 and the image display unit 20 are integrated. It is also possible to be configured to be mounted on the user's head.
In addition, the control device 10 and the image display unit 20 are connected by a longer cable or a wireless communication line. As the control device 10, a notebook computer, a tablet computer or a desktop computer, a game machine, a portable phone, a smartphone, A portable electronic device including a portable media player, other dedicated devices, or the like may be used.

  Further, for example, as a configuration for generating image light in the image display unit 20, a configuration including an organic EL (Organic Electro-Luminescence) display and an organic EL control unit may be used, or an LCOS (Liquid crystal) may be provided. On silicon and LCoS are registered trademarks), and digital micromirror devices can also be used. Further, for example, the present invention can be applied to a laser retinal projection type head mounted display. That is, the image generation unit includes a laser light source and an optical system that guides the laser light source to the user's eye, makes the laser light enter the user's eye, scans the retina, and forms an image on the retina. A configuration that allows the user to visually recognize an image may be employed. When a laser retina projection type head-mounted display is employed, the “image light emitting area in the image light generation unit” can be defined as an image area recognized by the user's eyes.

  As an optical system that guides image light to the user's eyes, an optical member that transmits external light that is incident from the outside toward the apparatus and that enters the user's eyes together with the image light can be employed. Moreover, you may use the optical member which is located ahead of a user's eyes and overlaps a part or all of a user's visual field. Further, a scanning optical system that scans a laser beam or the like to obtain image light may be employed. Further, the optical member is not limited to guiding the image light inside the optical member, and may have only a function of guiding the image light by refracting and / or reflecting it toward the user's eyes.

  Further, the present invention can be applied to a display device that employs a scanning optical system using a MEMS mirror and uses MEMS display technology. That is, as an image display element, a signal light forming unit, a scanning optical system having a MEMS mirror that scans light emitted from the signal light forming unit, and an optical member on which a virtual image is formed by light scanned by the scanning optical system May be provided. In this configuration, the light emitted from the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided through the optical member, and reaches the virtual image forming surface. When the MEMS mirror scans the light, a virtual image is formed on the virtual image forming surface, and the user recognizes the virtual image with the eyes, thereby recognizing the image. The optical component in this case may be one that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above embodiment, and may use a half mirror surface. .

  The display device of the present invention is not limited to a head-mounted display device, and can be applied to various display devices such as a flat panel display and a projector. The display device of the present invention may be any device that allows an image to be visually recognized by external light and image light. For example, a configuration in which an image by image light is visually recognized by an optical member that transmits external light is exemplified. Specifically, the above-described head-mounted display includes an optical member that transmits external light, and a light-transmitting plane or curved surface (glass) that is fixedly or movably installed at a position away from the user. It is also applicable to a display device that projects image light onto a transparent plastic or the like. As an example, a configuration of a display device that projects image light onto a window glass of a vehicle and allows a user who is on board or a user outside the vehicle to visually recognize the scenery inside and outside the vehicle together with an image by the image light. It is done. In addition, for example, image light is projected onto a transparent, semi-transparent, or colored transparent display surface that is fixedly installed such as a window glass of a building, and is displayed together with an image by the image light to a user around the display surface. A configuration of a display device that visually recognizes a scene through the surface can be given.

  In addition, at least a part of each functional block shown in FIG. 2 may be realized by hardware, or may be realized by cooperation of hardware and software, as shown in FIG. It is not limited to a configuration in which independent hardware resources are arranged. The program executed by the control unit 140 may be stored in the storage unit 120 or the storage device in the control device 10, or the program stored in an external device is acquired via the communication unit 117 or the interface 180. It is good also as a structure to execute. Of the configurations formed in the control device 10, only the operation unit 135 may be formed as a single user interface (UI), or the power source 130 in the above embodiment is formed independently and is replaceable. It may be a configuration. Further, the configuration formed in the control device 10 may be formed in the image display unit 20 in an overlapping manner. For example, the control unit 140 illustrated in FIG. 2 may be formed in both the control device 10 and the image display unit 20, or the control unit 140 formed in the control device 10 and the CPU formed in the image display unit 20. It is good also as a structure by which the function to perform is divided | segmented separately.

  DESCRIPTION OF SYMBOLS 10 ... Control apparatus 20, 20A, 20B ... Image display part (display part), 21 ... Right holding part, 22 ... Right display drive part, 23 ... Left holding part, 24 ... Left display drive part, 26 ... Right optical image Display unit, 28: Left optical image display unit, 61: Camera (imaging unit), 63 ... Microphone, 64 ... Stereo camera (imaging unit), 65 ... Vibration sensor, 66 ... 9-axis sensor, 67 ... Distance meter (ranging) Part), 100 ... head-mounted display device (display device), 117 ... communication part, 120 ... storage part, 140 ... control part, 150 ... operating system, 160 ... image processing part, 161 ... distance acquisition part, 163 ... Imaging processing unit, 165... Preview control unit, 170 ... audio processing unit, 180 ... interface, 190 ... display control unit, 201 ... right backlight control unit, 202 ... left backlight control unit, 211 ... right LCD control unit 212 ... Left LCD control unit, 221 ... Right backlight, 222 ... Left backlight, 241 ... Right LCD, 242 ... Left LCD, 251 ... Right projection optical system, 252 ... Left projection optical system, 261 ... Right light guide plate, 262 ... Left light guide plate, 301, 302 ... Subject, 310 ... Subject, 311 ... Subject, 315 ... Marker, 320 ... Captured image, 321 ... Marker image, AX ... Capture direction, CR ... Field angle, D ... Distance, EL, ER ... End, HM ... Head-mounted display device, L1 ... Distance (first distance), L2 ... Distance (second distance), OA ... Image supply device, PV ... Preview image, LE ... Left eye, RE ... Right eye, RS, LS ... Gaze direction, VR ... Field of view.

Claims (12)

A display unit that displays an image so that it can be seen through the outside scene together with the outside scene;
An imaging unit;
An imaging processing unit that causes the imaging unit to perform imaging;
A distance acquisition unit that acquires a distance from the subject imaged by the imaging unit;
The captured image of the imaging unit is displayed as a reference image on the display unit, and the display form of the reference image on the display unit is more visually recognized as the distance from the subject acquired by the distance acquisition unit is closer. A reference image control unit that adjusts to facilitate,
A display device comprising: The reference image control unit adjusts the display form of the reference image on the display unit so that the size of the reference image increases as the distance from the subject acquired by the distance acquisition unit decreases.
The display device according to claim 1. The reference image control unit is configured such that the distance to the subject is closer to the first distance than the size of the reference image when the distance to the subject acquired by the distance acquisition unit is a first distance. Enlarging the reference image when the distance is 2;
The display device according to claim 1, wherein: An imaging processing unit that detects the subject that satisfies a preset condition from a captured image of the imaging unit;
The reference image control unit further changes a display form of the reference image when the subject is detected by the imaging processing unit;
The display device according to claim 1 or 3.   The reference image control unit detects distortion of the image of the subject detected by the imaging processing unit in the captured image of the imaging unit, and displays the reference image in a display form according to the detected distortion state. The display device according to claim 4.   The display device according to claim 1, wherein the distance acquisition unit obtains a distance from the subject based on a size of the subject in a captured image captured by the imaging unit.   The display device according to claim 1, wherein the distance acquisition unit obtains a distance to the subject based on at least two images captured by the imaging unit.   The display device according to claim 7, wherein the imaging unit is capable of simultaneously capturing at least two images. A distance measuring unit that obtains the distance to the subject optically or using sound waves or ultrasonic waves,
The display device according to claim 1, wherein the distance acquisition unit acquires a distance from the subject measured by the distance measurement unit.   The display device according to claim 1, wherein the imaging unit is arranged so that a direction corresponding to the line of sight of the user is an imaging direction. A control method for a display device that includes a display unit that transmits an outside scene and displays an image so as to be visible together with the outside scene.
To capture the distance from the subject being imaged,
The captured image is displayed as a reference image by the display unit,
Adjusting the display form of the reference image on the display unit so that the reference image is easier to visually recognize as the distance to the acquired subject is shorter;
A control method of a display device characterized by the above. A program that can be executed by a computer that controls a display device that includes a display unit that transmits an outside scene and displays an image so as to be visible together with the outside scene.
The computer,
An imaging processing unit that causes the imaging unit to perform imaging;
A distance acquisition unit that acquires a distance from the subject imaged by the imaging unit;
The captured image of the imaging unit is displayed as a reference image on the display unit, and the display mode of the reference image on the display unit is more visible as the distance from the subject acquired by the distance acquisition unit is closer. A reference image control unit that adjusts to facilitate,
Program to make it function.

Images