JP2009200691A - Head-mounted video display device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a head-mounted video display device capable of capturing a desired image at all times regardless of the mounting state of the device or the form of a work object. <P>SOLUTION: The head-mounted video display device having a see-through display unit and an electronic camera whose photographing direction is variable is provided with: a camera position and posture calculation unit for calculating the position and posture of the camera on the basis of the information of the image photographed by the camera; a camera position and posture difference calculation unit for calculating a difference between the two positions and postures on the basis of the first position and posture of the camera calculated in the camera position and posture calculation unit on the basis of the information of the image photographed in the state that a prescribed index is positioned at the prescribed position of a see-through screen and the second position and posture of the camera calculated in the camera position and posture calculation unit on the basis of the information of the image photographed in the state that the index photographed by the camera and displayed on the screen of the display unit is positioned at the prescribed position; and a camera direction control unit for controlling the direction of the camera on the basis of the calculated result. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a head-mounted video display device, and more particularly to a head-mounted video display device including a display unit capable of see-through the outside and an electronic camera.

  Conventionally, an image (hereinafter also referred to as an image) that is detachably attached to the observer's head or face and obtained from an image display element such as a small CRT or a liquid crystal display element is displayed on the observer's eyeball by an eyepiece optical system. A head-mounted image display device, that is, an HMD (Head Mount Display) is known in which a virtual image can be observed as if the image is enlarged and projected in the air.

  The HMD is used as a display device for viewing content videos such as DVDs and videos, and for remote operation of industrial equipment or medical equipment, and has a wide variety of uses.

  For remote operation, a so-called see-through type HMD in which various information necessary for the work of an operator is usually displayed superimposed on a natural image (hereinafter also referred to as a see-through image) by external light incident through the display unit of the HMD. Is used.

For example, an operator wears a see-through type HMD equipped with an electronic camera, works while observing a see-through image of the work object, and an instructor displays an image of the work object photographed with the electronic camera at a remote location. For example, a remote work support system is known in which an operator is provided with appropriate information by voice or video while giving a work instruction while observing on a monitor such as a host computer. (For example, refer to Patent Document 1).
JP 2002-132487 A

  By the way, in such a remote operation support system, in order for the instructor to provide appropriate information to the operator, a see-through image of the work object being observed by the operator and an electronic camera being observed by the instructor It is necessary that the image of the work object photographed in step 1 is an image of substantially the same scene. That is, it is necessary for the operator's line-of-sight direction and the photographing direction (optical axis) of the electronic camera to be in the same direction.

  However, in an HMD equipped with an electronic camera such as that disclosed in Patent Document 1, the electronic camera usually has an HMD so that its optical axis is oriented in a certain direction with respect to the optical axis of the eyepiece optical system. It is fixedly attached to the frame. For this reason, depending on the worker's wearing state of the HMD and the position of the work target, for example, on a desk or overhead, the worker's line-of-sight direction and the optical axis of the electronic camera may be directed in different directions. For this reason, the images (scenes) observed by the worker and the instructor are different, and there is a problem that the instructor cannot provide appropriate information to the operator.

  The present invention has been made in view of the above problems, and provides a head-mounted video display apparatus that can always capture a desired image regardless of the state of the apparatus and the mode of the work object. Objective.

  The above object is achieved by the invention described in any one of the following items 1 to 3.

1. A display unit that displays images and can see through the outside world;
A head-mounted image display device comprising: an electronic camera that shoots the outside world and a shooting direction is variably attached;
A camera position / orientation calculation unit that calculates the position / orientation of the electronic camera based on information of an image captured by the electronic camera;
The electronic position calculated by the camera position / orientation calculation unit based on the information of the image captured by the electronic camera when the predetermined index is set at a predetermined position on the see-through screen that can be observed through the screen of the display unit The first position and orientation of the camera;
Calculated by the camera position / orientation calculation unit based on information of an image photographed by the electronic camera when the index photographed by the electronic camera and displayed on the screen of the display unit is located at the predetermined position. A camera position and orientation difference calculating unit that calculates a difference between the two positions and orientations based on the two positions and orientations of the electronic camera.
A head-mounted image display device comprising: a camera orientation control unit that controls an orientation of the electronic camera based on a calculation result of the camera position and orientation difference calculation unit.

  2. The camera orientation control unit compares the difference between the two positions / orientations calculated by the camera position / orientation difference calculation unit with a preset reference value related to the difference between the two positions / orientations, 2. The head-mounted image display device according to 1, wherein the direction of the electronic camera is controlled based on a comparison result.

3. 3. The camera position / posture difference calculation unit calculates X ₁ based on the following equation (1), where X ₁ is a difference between the two positions / postures, Head-mounted video display device.

(1)
However, M ₀ is a matrix indicating the first position / orientation in the simultaneous coordinate system, and M ₁ is a matrix indicating the second position / orientation in the simultaneous coordinate system.

  According to the present invention, the camera direction control unit calculates the difference between the two positions / postures of the first position / posture of the electronic camera and the second position / posture by the calculation result of the camera position / posture difference calculation unit. Based on this, the configuration is such that the orientation of the electronic camera is controlled. Thus, the image (scene) captured when the line of sight is directed toward the index and the image (scene) captured when the direction of the electronic camera is directed toward the index are substantially the same image (scene). The direction can be controlled. That is, the photographing direction (optical axis) of the electronic camera can be made to substantially coincide with the line-of-sight direction regardless of the mounting state of the HMD and the position of the index (work object). As a result, when the HMD according to the present invention is used in, for example, the above-described remote operation support system, the HMD that the instructor observes in the see-through image of the work object that the operator wearing the HMD observes. Since the images of the work object photographed by the electronic camera can be made substantially coincident, the instructor can provide appropriate information to the worker.

  Embodiments of a head-mounted image display device (hereinafter also referred to as HMD) according to the present invention will be described below with reference to the drawings. In addition, although this invention is demonstrated based on embodiment of illustration, this invention is not limited to this embodiment.

  First, the appearance of the HMD will be described with reference to FIG. FIG. 1 is a perspective view showing an appearance of an HMD 1 according to an embodiment of the present invention.

  As shown in FIG. 1, the HMD 1 includes temples 21R and 21L, a bridge 22, nose pads 23R and 23L, transparent substrates 24R and 24L, a display unit 25 (LCD display unit 26, eyepiece optical system 27), a camera unit 28, and an earphone. 29R, 29L, a microphone 30, a control unit 31, and the like.

  The HMD 1 having such a configuration is such that the worker wears the HMD 1 on the head, works while observing the see-through image of the work object through the display unit 25, and the work object photographed by the camera unit 28 by the instructor. While observing the above image on a monitor such as a host computer provided at a remote place, the operator is provided with appropriate information by voice or video via a wireless LAN or the like to give a work instruction.

  The temples 21R and 21L are long members made of a flexible elastic material or the like. The temples 21R and 21L are hooked on the ears or the temporal region of the wearer and held on the head of the wearer of the HMD1. This is for adjusting the mounting position. The temples 21R and 21L are configured to be rotatable in the directions of arrows P and Q in the rotating portions 21Ra and 21La. When the HMD 1 is not used, the temples 21R and 21L are rotated in the directions of the arrows P and Q. It can be made compact by moving it along the transparent substrates 24R and 24L.

  The bridge 22 includes nose pads 23R and 23L that hold the HMD 1 on the face of the wearer, and is a short rod-like member that is stretched between predetermined positions of the transparent substrates 24R and 24L facing each other. The transparent substrate 24L is held in a relative positional relationship with a fixed gap.

  The transparent substrates 24R and 24L are substantially flat transparent bodies that form a U-shaped space 24Rs at a position corresponding to one eyeball. An eyepiece optical system 27 is fitted in a U-shaped space 24Rs formed by being surrounded by the transparent substrate 24R.

  The display unit 25 includes an LCD display unit 26, an eyepiece optical system 27, and the like. The display unit 25 is provided in a remote place via an image captured by the camera unit 28 or a communication unit 36 described later provided in the control unit 31. For example, an image provided from a host computer (not shown) is displayed. Further, the display unit 25 is configured to be able to see through the outside world (front subject) through the eyepiece optical system 27. Details of the display unit 25 will be described later.

  The camera unit 28 corresponds to an electronic camera according to the present invention, and includes a zoom lens 281 (not shown), a CCD (Charge Coupled Device) 282, an image processing unit 283, and the like. The camera unit 28 shoots the outside world (front subject) around the wearer, and the subject optical image formed by the zoom lens 281 is photoelectrically converted by the CCD 282 to form an image signal (hereinafter also referred to as a video signal). The image processing unit 283 or the like performs predetermined image processing on the image signal.

  As shown in FIG. 1, the camera unit 28 is attached to the display unit 25 so that its optical axis L can be rotated in the horizontal and vertical directions, that is, panning and tilting are possible. The orientation of the camera unit 28 is controlled by the control unit 31 via a pan motor 288 and a tilt motor 289 described later.

  Earphones 29R and 29L are inserted into the ears of the wearer and provide sound to the wearer.

  The microphone 30 converts the voice of the wearer into an audio signal.

  The control unit 31 includes a microcomputer and includes a power switch 351, an operation switch 352, and the like. The control unit 31 receives signals from these various operation switches and a control signal from a host computer (not shown) provided in a remote place. The display operation and the image signal processing operation performed by the HMD 1 are comprehensively controlled.

  Here, the configuration of the display unit 25 will be described with reference to FIG. FIG. 2 is a side sectional view from the left side surface of the display unit 25, and mainly shows the internal configuration of the display unit 25.

  As shown in FIG. 2, the display unit 25 includes a housing 261, an LED 262, a collimator lens 263, an LCD display unit 26 composed of an LCD (Liquid Crystal Display) 264, an eyepiece composed of a prism 271 and an HOE (Holographic Optical Element) 272. An optical system 27 and the like are included.

  With the LED 262, the collimator lens 263, and the LCD 264 built in the housing 261 of the LCD display unit 26, the housing 261 is diagonally upward and forward at the upper end of the prism 271 of the eyepiece optical system 27 (FIG. 2). Then, it is attached in such a manner as to protrude obliquely upward to the right.

  The LED 262 is a point light source composed of a white light emitting diode (LED) including a predetermined wavelength color.

  The collimator lens 263 converts the light from the LED 262 into substantially parallel light and projects it onto the LCD 264.

  The LCD 264 displays a video signal photographed and generated by the camera unit 28, a video provided from a host computer (not shown) provided at a remote location, for example, via a communication unit 36 described later provided in the control unit 31. For example, it is a transmissive liquid crystal display panel.

  The prism 271 is a substantially plate-shaped transparent member made of glass, transparent resin, or the like, and causes light from the LCD 264 to be reflected a plurality of times inside. The upper end portion of the prism 271 is formed in a wedge shape so that the front side (opposite side of the eyepiece surface) protrudes to be thicker upward so that most of the light from the LCD 264 can be collected inside. The thick part 271b thus formed is formed.

  An inclined surface 271a is formed at the lower end of the prism 271. The prism 271 is bonded (for example, bonded) to the inclined surface 24Ra formed on the transparent substrate 24R via the HOE 272. The front and back surfaces of the prism 271 are flush with the front and back surfaces of the transparent substrate 24R. Thereby, the prism 271 is integrated with the transparent substrate 24R in a single plate shape.

  The HOE 272 is a volume phase type holographic optical element having a positive power, which is constituted by a so-called free-form surface that is optically asymmetric with respect to the axis, and is supported at a lower end portion of the prism 271 with a predetermined inclination angle. The HOE 272 provides a hologram image to the eyeball E using the light interference phenomenon when irradiated with the light guided by the prism 271.

  In the display unit 25 having the above-described configuration, the light emitted from the LED 262 illuminates the LCD 264 through the collimator lens 263, and the image light generated by the LCD 264 by this illumination is totally reflected in the prism 271 a plurality of times. Then, the light is diffracted by the HOE 272 and guided to the wearer's eyeball E as a virtual image.

  Further, the prism 271 guides light incident from the front to the eyeball E of the wearer. As a result, the wearer can see through the outside world (front subject) and is provided in a remote place via an image captured and generated by the camera unit 28 or a communication unit 36 described later provided in the control unit 31. For example, an image provided from, for example, a host computer (not shown) is superimposed on the outside (a subject in front) and viewed.

  Next, a schematic configuration of the HMD 1 will be described with reference to FIG. FIG. 3 is a block diagram illustrating a schematic configuration of the HMD 1. In FIG. 3, the same members as those shown in FIGS. 1 and 2 are assigned the same numbers.

  The main part of the HMD 1 includes a display unit 25, a camera unit 28, a control unit 31, and the like.

  The display unit 25 includes an LCD display unit 26 and an eyepiece optical system 27. Since the operations performed in each part have been described above, the description thereof will be omitted.

  The camera unit 28 includes a zoom lens 281, a CCD (Charge Coupled Device) 282, an image processing unit 283, a pan motor 288, a tilt motor 289, and the like.

  The CCD 282 is a color area imaging sensor in which red (R), green (G), and blue (B) color transmission filters are arranged in a checkered pattern in pixel units (pixel units), and is imaged by the zoom lens 281. The subject light image is photoelectrically converted into image signals of R, G, and B color components (signals composed of a signal sequence of pixel signals received in units of pixels).

  The image processing unit 283 includes, for example, a black level correction unit, a pixel interpolation unit, a white balance control unit, a gamma correction unit, a matrix calculation unit, a shading correction unit, and an image compression unit (not shown), and is read from the CCD 282. The image signal is subjected to known image signal processing. Then, the image signal that has undergone predetermined processing at these parts is stored in an image memory 33 of the control unit 31 described later.

  The control unit 31 includes a control unit 32, an image memory 33, a VRAM (Video Random Access Memory) 34, an operation unit 35, a communication unit 36, a pan / tilt motor drive circuit 37, and the like.

  The control unit 32 reads a ROM (Read Only Memory) that stores each control program (not shown), a RAM (Random Access Memory) that temporarily stores data such as arithmetic processing and control processing, and a control program from the ROM. CPU (central processing unit) etc. to be executed. The control unit 32 receives a signal from the operation unit 35 having the power switch 351 and the operation switch 352 and a control signal from a host computer (not shown) provided at a remote place, and performs display operations and images performed in the HMD 1. The signal processing operation is comprehensively controlled. Further, the control unit 32 controls the direction (photographing direction) of the camera unit 28 based on the information of the image photographed by the camera unit 28. Details of the control of the orientation of the camera unit 28 by the control unit 32 will be described later.

  The image memory 33 is a temporary memory used as a work area for performing various processing on the image signal by the control unit 32.

  The VRAM 34 has a recording capacity of an image signal corresponding to the number of pixels of the LCD 264 in the LCD display unit 26, and is a buffer memory for an image signal that constitutes a video that is reproduced and displayed on the LCD 264.

  The communication unit 36 is constituted by, for example, a wireless LAN wireless communication device, and transmits and receives video, audio, control signals, and the like with a host computer (not shown) provided at a remote location via a communication network (not shown). Do. In addition, Bluetooth, wireless USB, or a low power consumption type device may be used as the wireless communication means.

  The pan / tilt motor drive circuit 371 generates drive signals for driving the pan motor 288 and the tilt motor 289 based on the control of the control unit 32.

  Here, details of the control unit 32 will be described. The control unit 32 includes a camera position / orientation calculation unit 321, a camera position / orientation difference calculation unit 322, a camera orientation control unit 323, and the like.

  The camera position / orientation calculation unit 321 calculates the position / orientation of the camera unit 28 based on the information of the image captured by the camera unit 28. As a calculation method, for example, a method disclosed in Japanese Patent Application Laid-Open No. 2000-41173 that detects the position / orientation of a camera using three landmarks on the same plane can be used.

The camera position / orientation difference calculation unit 322 includes the first position / orientation M ₀ of the camera unit 28 calculated by the camera position / orientation calculation unit 321 based on the information of the first image I ₀ captured by the camera unit 28. The second position / orientation M ₁ of the camera unit 28 calculated by the camera position / orientation calculation unit 321 based on the information of the second image I ₁ photographed by the camera unit 28 is used. Based on this, the difference X ₁ between the two positions and orientations is calculated using the following equation (2).

Here, the first image I ₀ is an image taken by the camera unit 28 when the predetermined index is located at a predetermined position (for example, the center) of the see-through screen that can be observed through the screen of the display unit 25. is there. The second image I ₁ is taken by the camera unit 28 when the index taken by the camera unit 28 and displayed on the screen of the display unit 25 is in the predetermined position (for example, the center). It is an image.

(2)
Where M ₀ ; matrix indicating the first position and orientation in the simultaneous coordinate system M ₁ ; matrix indicating the second position and orientation in the simultaneous coordinate system R ₁ ; 3 × 3 rotation matrix T ₁ ; 3 × 1 It is a translation component.

Further, the camera position / orientation difference calculation unit 322 calculates the rotation angle θ ₁ in the camera coordinate system from the 3 × 3 rotation matrix R ₁ of the calculated two position / orientation differences X ₁ . Note that the rotation angle θ ₁ of the 3 × 3 rotation matrix R ₁ can be calculated using the Rodrigues formula.

Based on the rotation angle θ ₁ calculated by the camera position / orientation difference calculation unit 322, the camera direction control unit 323 performs the pan motor 288 and tilt in a direction in which the rotation angle θ ₁ decreases through the pan / tilt motor drive circuit 37. The driving of the motor 289 is controlled to change the direction (photographing direction) of the camera unit 28. That is, the direction (shooting direction) of the camera unit 28 is changed so as to substantially match the line-of-sight direction.

  Here, the flow of the camera orientation control operation performed by the HMD 1 will be described with reference to FIG. FIG. 4 is a flowchart showing the flow of the camera orientation control operation.

First, the head is rotated so that the predetermined index is located at a predetermined position (for example, the center) of the see-through screen that can be observed through the screen of the display unit 25 (step S1). The camera unit 28 in this state to obtain the first image I ₀ (step S2). Then, the camera position / orientation calculation unit 321 calculates the first position / orientation M ₀ of the camera unit 28 based on the acquired information of the first image I ₀ (step S3).

  Here, an example of the index is shown in FIG. FIG. 5 is a schematic diagram illustrating a chart S provided with an index according to an example. As shown in FIG. 5, yellow, blue, green, and red ring-shaped indexes M <b> 1 to M <b> 4 are provided at four corners of the chart S, respectively. By using such a chart S, the position / orientation of the camera unit 28 can be calculated with high accuracy.

Returning to FIG. 4, next, the head is rotated so that the index is located at the predetermined position (for example, the center) described above, which is photographed by the camera unit 28 and displayed on the screen of the display unit 25 (step S4). (Step S5). In this state, a second image I ₁ is acquired by the camera unit 28 (step S6). Then, the camera position / orientation calculation unit 321 calculates the second position / orientation M ₁ of the camera unit 28 based on the acquired information of the second image I ₁ (step S7).

Next, the camera position / orientation difference calculation unit 322 includes two positions / positions of the first position / posture M ₀ and the second position / posture M ₁ of the camera unit 28 calculated by the camera position / posture calculation unit 321. Based on the posture, the difference X ₁ between the two positions and postures is calculated using the above-described equation (2). Then, the camera position and orientation difference calculating unit 322 calculates the rotation angle theta ₁ in the camera coordinate system from the rotation matrix R ₁ a 3 × 3 of the difference X ₁ in two position and orientation calculated (step S8) (step S9).

Next, the camera orientation control unit 323 compares the rotation angle θ ₁ calculated by the camera position and orientation difference calculation unit 322 with a reference value θ _k related to a preset rotation angle. When the comparison result is θ ₁ ≧ θ _k (step S10; Yes), the camera orientation control unit 323 then performs the camera orientation change count i that has been executed and a reference value i related to the preset camera orientation change count. Compare _k . If i ≦ i _k as a result of the comparison (step S11; Yes), the camera orientation control unit 323 controls the pan / tilt motor drive circuit 37 based on the rotation angle θ ₁ calculated by the camera position and orientation difference calculation unit 322. Accordingly, the driving of the pan motor 288 and the tilt motor 289 is controlled in the direction in which the rotation angle θ ₁ is decreased, and the direction (photographing direction) of the camera unit 28 is changed (step S12). That is, the direction (shooting direction) of the camera unit 28 is changed so as to substantially match the line-of-sight direction. By repeatedly executing the operations in steps S5 to S12, the direction of the camera unit 28 (shooting direction) can be substantially matched with the line-of-sight direction.

As described above, in the HMD 1 according to the present invention, the camera orientation control unit 323 has the difference X between the two positions / postures of the first position / posture M ₀ and the second position / posture M ₁ of the camera unit 28. The orientation of the camera unit 28 is controlled based on the calculation result of the camera position and orientation difference calculation unit 322 that calculates ₁ . Thus, the image (scene) I ₀ photographed when the line of sight is directed toward the index and the image (scene) I ₁ photographed when the direction of the camera unit 28 is directed toward the index are substantially the same image (scene). Thus, the orientation of the camera unit 28 can be controlled. That is, the shooting direction (optical axis) of the camera unit 28 can be made to substantially coincide with the line-of-sight direction regardless of the mounting state of the HMD 1 or the position of the index (work object). As a result, when the HMD 1 according to the present invention is used in, for example, the above-described remote work support system, the HMD 1 that the instructor observes in the see-through image of the work object that the worker wearing the HMD 1 observes. Since the image of the work object photographed by the camera unit 28 can be made substantially coincident, the instructor can provide appropriate information to the worker.

  The present invention has been described above with reference to the embodiments. However, the present invention should not be construed as being limited to the above-described embodiments, and can be changed or improved as appropriate. For example, in the above-described embodiment, the direction of the camera unit 28 is changed two-dimensionally in the pan / tilt direction, but the camera unit 28 may be further rotated around the optical axis. That is, an image photographed by the camera unit 28 may be rotated. Thereby, the camera image can be brought closer to the see-through image.

It is a perspective view showing appearance of HMD concerning one embodiment of the present invention. It is side sectional drawing of the display unit of HMD. It is a block diagram which shows schematic structure of HMD. It is a flowchart which shows the flow of camera direction control operation | movement. It is a schematic diagram which shows the chart provided with the parameter | index by an example.

Explanation of symbols

1 Head-mounted video display (HMD)
21R, 21L Temple 22 Bridge 23R, 23L Nose pad 24R, 24L Transparent substrate 25 Display unit 26 LCD display unit 261 Case 262 LED
263 Collimator lens 264 LCD
27 Eyepiece optical system 271 Prism 272 HOE
28 Camera Unit 281 Zoom Lens 282 CCD
283 Image processing unit 288 Pan motor 289 Tilt motor 29R, 29L Earphone 30 Microphone 31 Control unit 32 Control unit 321 Camera position / orientation calculation unit 322 Camera position / orientation difference calculation unit 323 Camera orientation control unit 33 Image memory 34 VRAM
35 Operation section 351 Power switch 352 Operation switch 36 Communication section 37 Pan / tilt motor drive circuit E Eye

Claims (3)

A display unit that displays images and can see through the outside world;
A head-mounted image display device comprising: an electronic camera that shoots the outside world and a shooting direction is variably attached;
A camera position / orientation calculation unit that calculates the position / orientation of the electronic camera based on information of an image captured by the electronic camera;
The electronic position calculated by the camera position / orientation calculation unit based on the information of the image captured by the electronic camera when the predetermined index is set at a predetermined position on the see-through screen that can be observed through the screen of the display unit The first position and orientation of the camera;
Calculated by the camera position / orientation calculation unit based on information of an image photographed by the electronic camera when the index photographed by the electronic camera and displayed on the screen of the display unit is located at the predetermined position. A camera position and orientation difference calculating unit that calculates a difference between the two positions and orientations based on the two positions and orientations of the electronic camera.
A head-mounted image display device comprising: a camera orientation control unit that controls an orientation of the electronic camera based on a calculation result of the camera position and orientation difference calculation unit. The camera orientation control unit compares the difference between the two positions / orientations calculated by the camera position / orientation difference calculation unit with a preset reference value related to the difference between the two positions / orientations, The head-mounted image display device according to claim 1, wherein the direction of the electronic camera is controlled based on a comparison result. The camera position / orientation difference calculation unit calculates X ₁ based on the following equation (1), where X ₁ is a difference between the two positions / orientations. Head-mounted video display device.

(1)
However, M ₀ is a matrix indicating the first position / orientation in the simultaneous coordinate system, and M ₁ is a matrix indicating the second position / orientation in the simultaneous coordinate system.

Images