JP2008113317A - Remote operation support system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a remote operation support system capable of reducing a burden to presentation of an instruction to an operator for an instructor, allowing him to provide an appropriate instruction. <P>SOLUTION: The remote operation support system has: a head wearing type video display device equipped with an electronic camera which photographs an object for operation and a display unit which displays a video image; and a remote controller which is provided at a remote place from the head wearing type video display device, is equipped with a monitor which displays the video image photographed by the electronic camera and controls operations of the head wearing type video display device, wherein the instructor gives an operation instruction to the operator wearing the head wearing type video display device by the remote controller, the electronic camera is equipped with a zoom lens, rotatably supported by the head wearing type video display device and the remote controller performs remote control of the direction of the electronic camera and a zoom position of the zoom lens. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a remote work support system, and more particularly to a remote work support system including a head-mounted video display device.

  Conventionally, an image (hereinafter also referred to as an image) that is detachably attached to the observer's head or face and obtained from an image display element such as a small CRT or a liquid crystal display element is displayed on the observer's eyeball by an eyepiece optical system. A head-mounted image display device, that is, an HMD (Head Mount Display) is known in which a virtual image can be observed as if the image is enlarged and projected in the air.

  The HMD is used as a display device for viewing content videos such as DVDs and videos, and for remote operation of industrial equipment or medical equipment, and has a wide variety of uses.

For remote operation, the operator wears an HMD equipped with an electronic camera, and the instructor observes the work target image captured by the electronic camera on a monitor such as a host computer provided at a remote location, Various remote work support systems have been proposed in which appropriate information by voice or video is provided to workers and work instructions are given (see, for example, Patent Document 1).
JP 2002-132487 A

  However, in an HMD equipped with an electronic camera as disclosed in Patent Document 1, the electronic camera is attached to the HMD so that its optical axis is normally oriented in substantially the same direction as the optical axis of the eyepiece optical system. It has been. That is, the electronic camera is attached to the HMD so as to photograph a scene in the direction substantially the same as the line of sight of the worker wearing the HMD.

  For this reason, the camera does not always capture the desired image of the instructor. To obtain the desired image, for example, “Please turn right” or “Slightly more right” It is necessary to give complicated instructions such as “,” which may cause a decrease in workability. In addition, even if an operator is instructed and his / her face is directed in that direction, it is not easy to accurately convey the intention of the instructor, and the image is shaken when the operator shakes his / her head. There is a problem that it is difficult to find an object to be observed.

  The present invention has been made in view of the above problems, and provides a remote operation support system that can reduce the burden on an instructioner to present an instruction to an operator and can smoothly and appropriately give an instruction. With the goal.

  The above object is achieved by the invention described in any one of the following items 1 to 3.

1. An electronic camera that captures the work object;
A head-mounted video display device comprising a display unit for displaying video;
A remote operation device that is provided at a remote location of the head-mounted video display device and includes a monitor that displays an image captured by the electronic camera, and controls the operation of the head-mounted video display device; Have
A remote work support system in which an operator gives a work instruction to the worker wearing the head-mounted image display device by the remote operation device,
The electronic camera includes a zoom lens and is rotatably supported by the head-mounted video display device.
The remote operation support system characterized in that the remote operation device remotely controls the direction of the electronic camera and the zoom position of the zoom lens.

  2. The remote operation support system according to 1, wherein the remote operation device displays a warning on the display unit based on an orientation of the electronic camera and a zoom position of the zoom lens.

3. The head-mounted video display device includes a speaker that emits sound,
3. The remote operation support system according to 1 or 2, wherein the remote control device causes the speaker to emit a warning based on an orientation of the electronic camera and a zoom position of the zoom lens.

  According to the present invention, the remote control device remotely controls the orientation of the electronic camera provided in the HMD and the zoom position of the zoom lens. That is, the instructor remotely controls the direction of the electronic camera and the zoom position of the zoom lens by using the remote control device, and thereby the direction of the head (line of sight) of the worker wearing the HMD, the worker and the work object. Regardless of the distance, an image having a desired direction and size can be obtained. Therefore, for example, when the electronic camera does not capture the desired image of the instructor, the desired image can be obtained without giving a complicated instruction for the operator to turn his head or approach the work object. You can get a picture. As a result, it is possible to reduce the burden of the instruction person on the presentation of the instruction to the worker, and to perform an appropriate instruction smoothly.

  Embodiments of a remote work support system according to the present invention will be described below with reference to the drawings. In addition, although this invention is demonstrated based on embodiment of illustration, this invention is not limited to this embodiment.

  First, the configuration of the remote work support system 1 will be described with reference to FIG. FIG. 1 is a schematic diagram showing an outline of the overall configuration of a remote operation support system 1 according to the present invention.

  As shown in FIG. 1, the remote work support system 1 includes an HMD 2, a host computer 5, a video / audio recording server 6, a content server 7, and the like, and the host computer 5, the video / audio recording server 6, and the content server 7. Are connected via the Ethernet EN, and the host computer 5 and the HMD 2 are connected via the Internet IN, mobile phone, PHS, public network PN such as a wireless LAN, or the like.

  In the remote work support system 1 having such a configuration, an image of a work target captured by the camera unit 28 is provided at a remote place by the operator B wearing the HMD 2 equipped with the camera unit 28 described later. While observing on a monitor 53 (to be described later) of the host computer 5, the host computer 5 provides the worker B with appropriate information by video and audio and gives work instructions.

  Next, the configuration of the HMD 2 corresponding to the head-mounted image display device according to the present invention will be described with reference to FIG. FIG. 2 is an overall configuration diagram of the HMD 2 according to the present invention.

  As shown in FIG. 2, the HMD 2 includes temples 21R and 21L, a bridge 22, nose pads 23R and 23L, transparent substrates 24R and 24L, a display unit 25 (LCD display unit 26, an eyepiece optical system 27), a camera unit 28, and an earphone. 29R, 29L, a microphone 30, a control unit 31, and the like.

  The HMD 2 displays an image captured by the camera unit 28 or an image transmitted from the host computer 5 on the LCD display unit 26 provided in the display unit 25, and the image obtained from the LCD display unit 26 is displayed. By directly projecting onto the eyeball of the wearer (hereinafter also referred to as worker B) by the eyepiece optical system 27, it is possible to observe a virtual image as if the image is enlarged and projected in the air.

  The temples 21R and 21L are long members made of a flexible elastic material or the like. The temples 21R and 21L are hooked on the wearer's ears or the temporal region and are held on the head of the wearer by the wearer. This is for adjusting the mounting position. The temples 21R and 21L are configured to be rotatable in the directions of arrows P and Q in the rotating portions 21Ra and 21La. When the HMD 2 is not used, the temples 21R and 21L are rotated in the directions of the arrows P and Q. It can be made compact by moving it along the transparent substrates 24R and 24L.

  The bridge 22 includes nose pads 23R and 23L that hold the HMD 2 on the face of the wearer, and is a short rod-like member that is stretched over a predetermined position on the transparent substrates 24R and 24L facing each other. The transparent substrate 24L is held in a relative positional relationship with a fixed gap.

  The transparent substrates 24R and 24L are substantially flat transparent bodies that form a U-shaped space 24Rs at a position corresponding to one eyeball. An eyepiece optical system 27 is fitted in a U-shaped space 24Rs formed by being surrounded by the transparent substrate 24R.

  The display unit 25 includes an LCD display unit 26, an eyepiece optical system 27, and the like. The display unit 25 is transmitted from the computer 5 via an image captured by the camera unit 28 and a network I / F 36 described later provided in the control unit 31. Displayed video.

  The camera unit 28 corresponds to an electronic camera according to the present invention, and includes a zoom lens 281 (not shown), a CCD (Charge Coupled Device) 282, an image processing unit 284, and the like, and captures an external scene around the wearer. The subject optical image formed by the zoom lens 281 is photoelectrically converted by the CCD 282 to generate an image signal (video signal), and the image signal is subjected to predetermined image processing by the image processing unit 284 or the like. Video). As shown in FIG. 2, the camera unit 28 is attached to the display unit 25 so that its optical axis L can rotate in the horizontal and vertical directions, that is, pan and tilt. The zoom position of the zoom lens 281 and the orientation of the camera unit 28 are remotely controlled by the computer 5 via a zoom motor 287, a pan motor 288, and a tilt motor 289, which will be described later.

  Earphones 29R and 29L are inserted into the ears of the wearer and provide sound to the wearer.

  The microphone 30 converts the voice of the wearer into an audio signal.

  The control unit 31 includes a microcomputer and includes a power switch 351, an operation switch 352, and the like. The control unit 31 receives a signal from these various operation switches, a control signal from the host computer 5, and the like. The image signal processing operation is comprehensively controlled.

  Here, the configuration of the display unit 25 will be described with reference to FIG. FIG. 3 is a side sectional view from the left side surface of the display unit 25 in the HMD 2 according to the present invention, and mainly shows the internal configuration of the display unit 25.

  As shown in FIG. 3, the display unit 25 includes a housing 261, an LED (Light Emitting Diode) 262, a collimator lens 263, an LCD (Liquid Crystal Display) 264 LCD display unit 26, a prism 271, and HOE (Holographic Optical). Element) 272 and the eyepiece optical system 27.

  In the state where the LED 262, the collimator lens 263, and the LCD 264 are built in the housing 261 of the LCD display unit 26, the housing 261 is diagonally upward and forward at the upper end portion of the prism 271 of the eyepiece optical system 27 (FIG. 3). Then, it is attached in such a manner as to protrude obliquely upward to the right.

  The LED 262 is a point light source composed of a white light emitting diode (LED) including a predetermined wavelength color.

  The collimator lens 263 converts the light from the LED 262 into substantially parallel light and projects it onto the LCD 264.

  The LCD 264 generates a video (image) based on a video signal photographed and generated by the camera unit 28 or a video signal transmitted from the computer 5 via a network I / F 36 described later provided in the control unit 31. For example, a transmissive liquid crystal display panel.

  The prism 271 is a substantially plate-shaped transparent member made of glass, transparent resin, or the like, and causes light from the LCD 264 to be reflected a plurality of times inside. The upper end portion of the prism 271 is formed in a wedge shape so that the front side (opposite side of the eyepiece surface) protrudes to be thicker upward so that most of the light from the LCD 264 can be collected inside. The thick part 271b thus formed is formed.

  An inclined surface 271a is formed at the lower end of the prism 271. The prism 271 is bonded (for example, bonded) to the inclined surface 24Ra formed on the transparent substrate 24R via the HOE 272. The front and back surfaces of the prism 271 are flush with the front and back surfaces of the transparent substrate 24R. Thereby, the prism 271 is integrated with the transparent substrate 24R in a single plate shape.

  The HOE 272 is a volume phase type holographic optical element having a positive power, which is constituted by a so-called free-form surface that is optically asymmetric with respect to the axis, and is supported at a lower end portion of the prism 271 with a predetermined inclination angle. The HOE 272 provides a hologram image to the eyeball E using the light interference phenomenon when irradiated with the light guided by the prism 271.

  In the display unit 25 having the above-described configuration, the light emitted from the LED 262 illuminates the LCD 264 through the collimator lens 263, and the image light generated by the LCD 264 by this illumination is a plurality of times in the prism 271. After reflection, the light is diffracted by the HOE 272 and guided to the wearer's eyeball E as a virtual image.

  Further, the prism 271 guides light incident from the front to the eyeball E of the wearer. As a result, the wearer can see through the outside world (the subject in front) and is transmitted from the computer 5 via the video image captured and generated by the camera unit 28 or the network I / F 36 described later provided in the control unit 31. The recorded video is viewed with being superimposed on the outside world (the subject in front).

  Next, the electric circuit configuration of the HMD 2 will be described with reference to FIG. FIG. 4 is a block diagram of the electric circuit of the HMD 2 according to the present invention. In FIG. 4, the same members as those shown in FIGS. 1 to 3 are given the same numbers.

  The main electric circuit block of the HMD 2 includes a display unit 25, a camera unit 28, a control unit 30, and the like.

  The display unit 25 includes an LCD display unit 26 and an eyepiece optical system 27. Since the operations performed in each part have been described above, the description thereof will be omitted.

  The camera unit 28 includes a zoom lens 281, a CCD (Charge Coupled Device) 282, an AFE (Analog Front End) 283, an image processing unit 284, a zoom motor 287, a pan motor 288, a tilt motor 289, and the like.

  The CCD 282 is a color area imaging sensor in which R (red) light, G (green) light, and B (blue) light transmission filters are arranged in a checkered pattern in pixel units (pixel units), and is connected by a zoom lens 281. The imaged subject light image is photoelectrically converted into image signals (signals consisting of a signal sequence of pixel signals received in units of pixels) of each color component of R (red) light, G (green) light, and B (blue) light. To convert.

  The AFE 283 controls the drive of the CCD 282 based on the reference clock transmitted from the control unit 31, performs a known analog signal processing on the video signal (image signal) read from the CCD 282, and converts it into a digital video signal To do.

  In this manner, the video signal read by the CCD 282 is subjected to predetermined processing by the AFE 283 and converted into a digital video signal. The digitized video signal is captured by the image processing unit 284 and subjected to predetermined processing. Hereinafter, the process for the digital video signal performed by the image processing unit 284 will be described.

  First, the digital video signal captured by the image processing unit 284 is written into an image memory 33 in the control unit 31 described later in synchronization with the readout of the video signal output from the CCD 282. That is, a digital video signal used for processing performed by the image processing unit 284 is once recorded in the image memory 33 and taken out from the image memory 33 and used for processing in each part in the image processing unit 284.

  The image processing unit 284 includes, for example, a black level correction unit, a pixel interpolation unit, a white balance control unit, a gamma correction unit, a matrix calculation unit, a shading correction unit, an image compression unit, and the like (not shown). The digital video signal is subjected to known image signal processing. Then, the digital video signal that has undergone predetermined processing at these parts is stored in the image memory 33 again.

  The control unit 31 includes a control unit 32, an image memory 33, a VRAM (Video Random Access Memory) 34, an operation unit 35, a network I / F 36, a pan / tilt motor driving circuit 371, a zoom motor driving circuit 372, and an audio output circuit 381. , A voice input circuit 382 and the like.

  The control unit 32 reads a ROM (Read Only Memory) that stores each control program (not shown), a RAM (Random Access Memory) that temporarily stores data such as arithmetic processing and control processing, and a control program from the ROM. CPU (central processing unit) etc. that execute the above, and in response to the signals from the aforementioned operation switches and the control signals from the host computer 5, the display operation and image signal processing operation performed by the HMD 2 are integrated. To control.

  The image memory 33 is a temporary memory used as a work area for performing various processing on the digital video signal by the control unit 32 in the control unit 31.

  The VRAM 34 has a video signal recording capacity corresponding to the number of pixels of the LCD 264 in the LCD display unit 26, and is a video signal buffer memory that constitutes an image reproduced and displayed on the LCD 264.

  The network I / F 36 connects the HMD 2 to the public line network PN, communicates with the host computer 5, and inputs and outputs video, audio, and various control signals.

  The pan / tilt motor drive circuit 371 generates drive signals for driving the pan motor 288 and the tilt motor 289 based on the control of the control unit 32.

  The zoom motor drive circuit 372 generates a drive signal for driving the zoom motor 287 based on the control of the control unit 32.

  Next, the configuration of the host computer 5 corresponding to the remote control device according to the present invention will be described with reference to FIG. FIG. 5 is an overall configuration diagram of the host computer 5 (host PC 5) according to the present invention.

  As shown in FIG. 5, the host PC 5 is, for example, a personal computer including a computer main body 51, a monitor 53, a mouse 55, a keyboard 57, and the like.

  The host PC 5 displays the work target photographed by the camera unit 28 of the HMD 2 attached to the worker B on the monitor 53. The instructor A operates the host PC 5 with the mouse 55, the keyboard 57, etc. while observing the work target displayed on the monitor 53, and provides appropriate information by video and audio to the worker B by the PC 5, Is to do.

  In the remote work system 1 having such a configuration, the present invention is directed to the head of the worker B wearing the HMD 2 by the instructor A remotely controlling the operation of the camera unit 28 provided in the HMD 2 by the host PC 5. This makes it possible to observe a desired image regardless of the direction (line-of-sight direction) and the distance between the worker B and the work object.

  Specifically, this will be described with reference to FIG. FIG. 6 is a schematic diagram showing how the operation of the camera unit 28 in the HMD 2 according to the present invention is remotely controlled.

  The worker B wears the HMD 2 including the camera unit 28 and works toward the work target X. On the other hand, the instructor A operates the host PC 5 while observing the video of the work object X photographed by the camera unit 28 displayed on the monitor 53 of the host PC 5 provided at a remote place, and the video and audio. The worker B is instructed to work.

  In such a situation, when the instructor A wants to observe the right or left image from the image captured by the camera unit 28, he / she operates a predetermined key corresponding to each of the keyboards 57. When a key operation is performed, the camera unit 28 slowly rotates (pan operation) in the right direction (arrow R direction) or the left direction (arrow L direction). When the camera unit 28 rotates and captures a desired image, the instructor A stops the key operation. In addition, when it is desired to observe the upper or lower image from the image captured by the camera unit 28, the camera unit 28 is similarly tilted. Further, when it is desired to observe the work object X a little larger, the zoom lens 281 of the camera unit 28 is similarly zoomed. In this way, the instructor A can observe a desired image by causing the camera unit 28 to perform pan, tilt, and zoom operations.

  Further, if the instructor does not yet obtain a desired image even if the camera unit 28 is remotely controlled to perform pan, tilt and zoom operations, the host computer 5 warns the display unit 25 of the HMD 2. The display is performed to prompt the worker B to turn his head or approach the work target X. For example, an arrow indicating the direction of head rotation or approaching direction is displayed as a warning.

  Specifically, this will be described with reference to FIG. FIG. 7 is a schematic diagram showing an example of a warning display displayed on the display unit 25 in the HMD 2 according to the present invention.

  FIG. 7 is a schematic view of the HMD 2 viewed from the eyepiece direction, and the camera unit 28 is rotated to the left limit by the pan operation. In such a case, the host computer 5 displays a warning such as “Turn to the Left” on the display image IM of the display unit 25. As a result, the operator B rotates his / her head in the left direction, and the instructor A can observe a desired image. The warning may be given to the worker B by voice through the earphones 29R and 29L provided in the HMD 2.

  As described above, in the remote operation support system 1 according to the present invention, the orientation of the camera unit 28 provided in the HMD 2 and the zoom position of the zoom lens 281 are remotely controlled. That is, the instructor A remotely controls the orientation of the camera unit 28 and the zoom position of the zoom lens 281 with the host PC 5, so that the head direction (line-of-sight direction) of the worker B wearing the HMD 2 and the operator B Regardless of the distance from the work object X, an image having a desired direction and size can be obtained. Therefore, for example, when the camera unit 28 does not capture the desired image of the instructor A, a complicated instruction for rotating the head of the operator B or approaching the work object X is performed. A desired image can be obtained without any problem. Thereby, it is possible to reduce the burden of the instruction A on the presentation of the instruction to the worker B, and to perform appropriate instructions smoothly.

It is a schematic diagram which shows the whole structure of the remote work assistance system which concerns on this invention. 1 is an overall configuration diagram of an HMD according to the present invention. It is a sectional side view of the display unit in HMD concerning the present invention. It is an electric circuit block block diagram of HMD which concerns on this invention. 1 is an overall configuration diagram of a host PC according to the present invention. FIG. It is a schematic diagram which shows a mode that operation | movement of the camera unit in HMD which concerns on this invention is controlled remotely. It is a schematic diagram which shows an example of the warning display displayed on the display unit in HMD which concerns on this invention.

Explanation of symbols

1 Remote work support system 2 Head-mounted video display device (HMD)
21R, 21L Temple 22 Bridge 23R, 23L Nose pad 24R, 24L Transparent substrate 25 Display unit 26 LCD display unit 261 Case 262 LED
263 Collimator lens 264 LCD
27 Eyepiece optical system 271 Prism 272 HOE
28 Camera Unit 281 Zoom Lens 282 CCD
283 AFE
284 Image processing unit 287 Zoom motor 288 Pan motor 289 Tilt motor 29R, 29L Earphone 30 Microphone 31 Control unit 32 Control unit 33 Image memory 34 VRAM
35 Operation Unit 351 Power Switch 352 Operation Switch 36 Network I / F
371 Pan / tilt motor drive circuit 372 Zoom motor drive circuit 381 Audio output circuit 382 Audio input circuit 5 Host computer (host PC)
51 Computer Main Body 53 Monitor 55 Mouse 57 Keyboard 6 Video / Audio Recording Server 7 Content Server A Instructor B Worker EN Ethernet E Eye IN Internet IM Display Image PN Public Line Network X Work Object

Claims (3)

An electronic camera that captures the work object;
A head-mounted video display device comprising a display unit for displaying video;
A remote operation device that is provided at a remote location of the head-mounted video display device and includes a monitor that displays an image captured by the electronic camera, and controls the operation of the head-mounted video display device; Have
A remote work support system in which an operator gives a work instruction to the worker wearing the head-mounted image display device by the remote operation device,
The electronic camera includes a zoom lens and is rotatably supported by the head-mounted video display device.
The remote operation support system characterized in that the remote operation device remotely controls the direction of the electronic camera and the zoom position of the zoom lens. The remote operation support system according to claim 1, wherein the remote operation device displays a warning on the display unit based on an orientation of the electronic camera and a zoom position of the zoom lens. The head-mounted video display device includes a speaker that emits sound,
The remote operation support system according to claim 1, wherein the remote control device causes the speaker to emit a warning based on an orientation of the electronic camera and a zoom position of the zoom lens.

Images