JP2019074637A - Remote support system, display system, head mounted display, operation terminal, display control program, and control method - Google Patents

Abstract

To facilitate photographing performed by a photographing unit mounted on a head mounted display.SOLUTION: A head mounted display 1 comprises: a photographing unit 12 that photographs a first video; and a transmission unit that transmits a first video photographed by the photographing unit 12 to an operation terminal 2 and a remote support terminal 6. The operation terminal 2 includes a first display unit 21 that displays the first video transmitted by the transmission unit to a first user 4 wearing the head mounted display 1. The remote support terminal 6 includes a second display unit 61 that displays the first video transmitted by the transmission unit to a second user 7 present at a position different from that of the first user 4.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a remote support system, a display system, a head mounted display, an operation terminal, a display control program, and a control method.

  In recent years, various wearable terminals worn and used on the user's body have been developed. One type of wearable terminal is the Head Mounted Display (HMD).

  The HMD is, for example, a glasses-type or goggle-type device worn on the head of the user. By mounting the HMD on the head of the user, the display unit provided in the HMD is positioned in front of the user's eyes, and the user can view the image displayed on the display unit.

  A Hand Held Device (HHD) may be connected to the HMD as a terminal for operating the HMD.

  By the way, there is a remote support system using an HMD. In the remote support system, remote support is implemented by a worker wearing the HMD performing on-site work and transmitting an image from a camera mounted on the HMD to a support person at a remote location.

JP, 2006-189476, A Japanese Patent Application Publication No. 2003-308117 JP, 2011-071884, A

  In the conventional remote support system, the worker at the site changes the direction of the camera mounted on the HMD according to the instruction from the supporter at the remote location. When shooting a video that the supporter wants to see, the worker can not check the video of the camera mounted on the HMD that he / she wears. As a result, in the conventional remote support system, there is a possibility that the worker can not properly capture the video that the supporter wants to see.

  In addition, when the image of the camera is displayed on the display mounted on the HMD, the image is displayed overlapping with the actual image viewed by the operator, and therefore, there is a possibility that the operation is interrupted due to the decrease in the visibility.

  In one aspect, it is an object of the present invention to facilitate photographing by a photographing unit mounted on a head mounted display.

  The remote support system is a remote support system having a head mounted display, an operation terminal, and a remote support terminal, wherein the head mounted display is a photographing unit for photographing a first image, and the photographing unit A transmission unit for transmitting the first image to be transmitted to the operation terminal and the remote support terminal, the operation terminal may be configured to transmit the first image transmitted by the transmission unit to the head mounted display A first display unit for displaying to a first user who wears the video, the remote support terminal presents the first image transmitted by the transmission unit at a position different from the first user And a second display unit for displaying the second user.

  In one aspect, imaging by the imaging unit mounted on the head mounted display can be facilitated.

It is a figure showing an example of composition of a remote support system as an example of an embodiment. It is a figure which shows typically the function structural example of HMD shown in FIG. It is a figure which shows typically the function structural example of HHD shown in FIG. It is a sequence diagram which shows the operation example of the remote control system as an example of embodiment.

  Hereinafter, one embodiment will be described with reference to the drawings. However, the embodiments described below are merely examples, and there is no intention to exclude the application of various modifications and techniques that are not specified in the embodiments. That is, the present embodiment can be variously modified and implemented without departing from the scope of the present invention.

  In addition, each drawing is not intended to include only the components illustrated in the drawings, but may include other functions and the like.

  Hereinafter, in the drawings, the same reference numerals indicate the same parts, so the description thereof will be omitted.

[A] Example of Embodiment [A-1] Example of System Configuration FIG. 1 is a view showing an example of the configuration of a remote support system 100 as an example of the embodiment.

  As illustrated in FIG. 1, the remote support system 100 includes a head mounted display (HMD) 1, a handheld device (HHD) 2, a remote support terminal 6, and a server 8. Note that "HHD" may be referred to as "operation terminal". HMD1 and HHD2 may be collectively referred to as a "display system". In the example shown in FIG. 1, the remote support terminal 6 is installed at a location distant from the HMD 1 and the HHD 2.

  The HMD 1 and the HHD 2 may be communicably connected to each other in a wired manner via the communication cable 3. The communication standard of the communication cable 3 may be, for example, High-Definition Multimedia Interface (HDMI (registered trademark)) or Universal Serial Bus (USB). Further, the HMD 1 and the HHD 2 may be communicably connected to each other wirelessly. The wireless communication standard in this case may be, for example, Wi-Fi Direct or Bluetooth (registered trademark).

  The HMD 1 and the remote support terminal 6 may be communicably connected to each other via the server 8, for example, wirelessly. The wireless communication standard in this case may be, for example, third generation mobile communication (3G), Long Term Evolution (LTE), or Wi-Fi. In addition, for communication between the HMD 1 and the remote support terminal 6 via the server 8, Session Initiation Protocol (SIP) may be used.

  In the example illustrated in FIG. 1, the HMD 1 is a glasses-type device, and includes a display unit 11 in a portion corresponding to a portion to which a lens is attached in actual glasses. The HMD 1 may be a goggle type device.

  As shown in FIG. 1, the HMD 1 is mounted on the head (in other words, the “face”) of the worker 4 (which may be referred to as “the first user”). At this time, the display unit 11 of the HMD 1 is mounted so as to cover both eyes of the worker 4. Thus, the worker 4 can visually recognize the image displayed on the display unit 11.

  The HMD 1 also includes an imaging unit 12 (which may be referred to as a "camera"). The imaging unit 12 captures an image of the front of the HMD 1. In the example shown in FIG. 1, the photographing unit 12 photographs the object 5. The HMD 1 transmits an image (which may be referred to as an “image”) captured by the imaging unit 12 to the HHD 2 and the remote support terminal 6. Note that the image captured by the imaging unit 12 may be a moving image or a still image. An area indicated by an alternate long and short dash line in FIG. 1 indicates an area photographed by the photographing unit 12.

  As shown in FIG. 1, the HHD 2 includes a display unit 21. The display unit 21 displays the video transmitted from the HMD 1. The image displayed on the display unit 21 can be viewed by the worker 4. In the example illustrated in FIG. 1, the display unit 21 displays the video of the object 5 transmitted from the HMD 1.

  The server 8 is a computer (may be called an “information processing apparatus”) having a server function. In the example shown in FIG. 1, the server 8 relays communication between the HMD 1 and the remote support terminal 6.

  The remote support terminal 6 includes a display unit 61. The remote support terminal 6 receives the video transmitted from the HMD 1 via the server 8, and displays the received video on the display unit 61. The image displayed on the display unit 61 can be viewed by the supporter 7 (may be referred to as a “second user”). In the example illustrated in FIG. 1, the display unit 61 displays the video of the object 5 transmitted from the HMD 1. The display unit 61 may be, for example, a liquid crystal display or an organic EL (Organic Electro-Luminescence) display.

  FIG. 2 is a view schematically showing an example of a functional configuration of the HMD 1 shown in FIG.

  The HMD 1 illustratively includes a Central Processing Unit (CPU) 10, a display control unit 110, a display unit 11, an imaging control unit 120, an imaging unit 12, an audio input / output unit 140, an earphone microphone 14, and a memory 15. The HMD 1 further includes a communication control unit 130, a 3G / LTE communication unit 131, a WiFi communication unit 132, a BlueTooth communication unit 133, an HDMI communication unit 134, and a USB communication unit 135.

  The display unit 11 functions as an example of a third display unit that displays, to the worker 4, a second video different from the HHD 2 and the first video displayed on the remote support terminal 6. The display unit 11 may display, for example, work instruction content from the supporter 7 to the worker 4 as a second video. The display unit 11 may have transparency so that the appearance of the surroundings can be viewed even when the worker 4 wears the HMD 1. The display unit 11 may be, for example, a liquid crystal display or an organic EL display.

  The display control unit 110 controls the display of an image by the display unit 11. The display control unit 110 may control the second image to be displayed on a part of the display unit 11 and the other part of the display unit 11 to maintain transparency.

  The photographing unit 12 photographs the front of the HMD 1 as a first image.

  The imaging control unit 120 controls the imaging unit 12. For example, the imaging control unit 120 activates or terminates the imaging unit 12 according to the input from the input unit 22 (described later with reference to FIG. 3) of the HHD 2 or the input unit (not shown) of the HMD 1 or the remote support terminal 6 The recording of the first video by the unit 12 may be started or ended.

  The earphone microphone 14 outputs a voice to the worker 4 and acquires the voice input from the worker 4.

  The voice input / output unit 140 controls input / output to the earphone microphone 14.

  The communication control unit 130 communicably connects the HMD 1 to the HHD 2 and the remote support terminal 6, and includes a communication cable 3 and a communication interface conforming to the communication standard between the HMD 1 and the server 8. The communication control unit 130 functions as an example of a transmission unit that transmits the first image captured by the imaging unit 12 to the HHD 2 and the remote support terminal 6. The “first video shot by the shooting unit 12” may include both a video being recorded by the shooting unit 12 and a video shot by the shooting unit 12 that is not being recorded.

  The 3G / LTE communication unit 131 is a communication interface conforming to the 3G standard and the LTE standard.

  The WiFi communication unit 132 is a communication interface conforming to the WiFi standard.

  The BlueTooth communication unit 133 is a communication interface conforming to the BlueTooth standard.

  The HDMI communication unit 134 is a communication interface conforming to the HDMI standard.

  The USB communication unit 135 is a communication interface conforming to the USB standard.

  The HMD 1 may not include a part of the 3G / LTE communication unit 131, the WiFi communication unit 132, the BlueTooth communication unit 133, the HDMI communication unit 134, and the USB communication unit 135.

  The memory 15 is, for example, a storage device including a read only memory (ROM) and a random access memory (RAM). In the ROM of the memory 15, a program such as Basic Input / Output System (BIOS) may be written. The software program of the memory 15 may be appropriately read and executed by the CPU 10. Further, the RAM of the memory 15 may be used as a primary recording memory or a working memory.

  The CPU 10 is, for example, a processing device that performs various controls and calculations, and realizes various functions by executing an operating system (OS) and a program stored in the memory 15.

  The program for realizing the function of the CPU 10 is, for example, a flexible disk, a CD (CD-ROM, CD-R, CD-RW, etc.), a DVD (DVD-ROM, DVD-RAM, DVD-R, DVD + R, DVD) -It may be provided in the form of being recorded on a computer readable recording medium such as RW, DVD + RW, HD DVD, etc., Blu-ray Disc, magnetic disc, optical disc, magneto-optical disc and the like. The computer (in this embodiment, the CPU 10) may read the program from the above-described recording medium via a reading device (not shown), transfer the program to an internal recording device or an external recording device, and store it. Also, the program may be recorded in a storage device (recording medium) such as a magnetic disk, an optical disk, a magneto-optical disk or the like, and may be provided from the storage device to the computer via a communication path.

  When the functions of the CPU 10 are realized, a program stored in an internal storage device (the memory 15 in the present embodiment) may be executed by a computer (the CPU 10 in the present embodiment). Further, the computer may read and execute the program recorded on the recording medium.

  The CPU 10 illustratively controls the overall operation of the HMD 1 and HHD 2. The device for controlling the operation of the entire HMD1 and HHD2 is not limited to the CPU 10. For example, a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD) , And one of Field Programmable Gate Array (FPGA). The device for controlling the overall operation of the HMD 1 and HHD 2 may be a combination of two or more of a CPU, an MPU, a DSP, an ASIC, a PLD, and an FPGA.

  Part or all of the functions of the display control unit 110, the imaging control unit 120, the communication control unit 130, and the voice input / output unit 140 may be incorporated in the CPU 10.

  FIG. 3 is a view schematically showing an example of a functional configuration of the HHD 2 shown in FIG.

  The HHD 2 exemplarily includes a control unit 20, a display unit 21, an input unit 22, 2G / LTE communication unit 23, a WiFi communication unit 24, a BlueTooth communication unit 25, an HDMI communication unit 26, and a USB communication unit 27.

  The control unit 20 controls the display unit 21, the input unit 22, the 3G / LTE communication unit 23, the WiFi communication unit 24, the BlueTooth communication unit 25, the HDMI communication unit 26, and the USB communication unit 27. The control unit 20 functions as an example of a receiving unit that receives the first image captured by the HMD 1 from the HMD 1.

  The display unit 21 functions as an example of a first display unit that displays the first image transmitted by the HMD 1 to the worker 4 wearing the HMD 1. That is, the display unit 21 has a function of displaying an image captured by the HMD 1 as a finder at the hand of the worker 4. The display unit 21 may be, for example, a liquid crystal display or an organic EL display.

  The input unit 22 receives an input from the worker 4. The input unit 22 is, for example, a physical button or a touch panel. The input unit 22 may activate or terminate the imaging unit 12 of the HMD 1 and receive an input for starting or ending the recording of the first video by the imaging unit 12.

  The 3G / LTE communication unit 23 is a communication interface conforming to the 3G standard and the LTE standard.

  The WiFi communication unit 24 is a communication interface conforming to the WiFi standard.

  The BlueTooth communication unit 25 is a communication interface conforming to the BlueTooth standard.

  The HDMI communication unit 26 is a communication interface conforming to the HDMI standard.

  The USB communication unit 27 is a communication interface conforming to the USB standard.

  The HHD 2 may not include a part of the 3G / LTE communication unit 23, the WiFi communication unit 24, the BlueTooth communication unit 25, the HDMI communication unit 26, and the USB communication unit 27.

  The display unit 61 of the remote support terminal 6 shown in FIG. 1 displays an example of the second display unit for displaying the first image transmitted by the HMD 1 to the supporter 7 present at a position different from the worker 4 Act as.

[A-2] Operation Example An operation example of the remote support system 100 as an example of the embodiment described above will be described according to the sequence diagram (steps S1 to S5) shown in FIG. 4.

  The communication control unit 130 of the HMD 1 establishes a communication connection with the remote support terminal 6 (step S1). SIP may be used to establish a communication connection.

  The imaging control unit 120 of the HMD 1 activates the imaging unit 12, and the imaging unit 12 starts imaging of an image (step S2).

  The communication control unit 130 of the HMD 1 transmits the image captured by the imaging unit 12 to the HHD 2 and causes the display unit 21 of the HHD 2 to display the image (step S3).

  The communication control unit 130 of the HMD 1 transmits the image captured by the imaging unit 12 to the remote support terminal 6 (step S4).

  The remote support terminal 6 displays the photographed image received from the HMD 1 on the display unit 61 (step S5).

  The processes of steps S2 to S5 are repeatedly executed until the photographing by the photographing unit 12 is completed.

[A-3] Effects According to the remote support system 100 in the example of the embodiment described above, for example, the following effects can be achieved.

  The imaging unit 12 of the HMD 1 captures a first video. The communication control unit 130 of the HMD 1 transmits the first image captured by the imaging unit 12 to the HHD 2 and the remote support terminal 6. The display unit 21 of the HHD 2 displays the first image transmitted by the HMD 1 to the worker 4 wearing the HMD 1. The display unit 61 of the remote support terminal 6 displays the first image transmitted by the HMD 1 to the supporter 7 present at a position different from that of the worker 4.

  Thereby, the photographing by the photographing unit 12 mounted on the HMD 1 can be facilitated. Specifically, the worker 4 can visually recognize the video taken by the photographing unit 12 of the HMD 1 by the display unit 21 of the HHD 2, and it becomes easy to shoot the video desired by the support person 7.

  The display unit 11 of the HMD 1 displays a second video different from the first video to the worker 4.

  Accordingly, the worker 4 can visually recognize information and the like regarding his / her own work which is different from the video shot by the shooting unit 12.

  The display unit 11 of the HMD 1 has transparency. A part of the display unit 11 displays the second image, and the other part of the display unit 11 is controlled to maintain transparency.

  Thereby, even when the second video is displayed on the display unit 11 of the HMD 1, the worker 4 can visually recognize the first video displayed on the display unit 21 of the HHD 2.

[B] Others The disclosed technology is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present embodiment. Each configuration and each process of the present embodiment can be selected as needed, or may be combined as appropriate.

  Although the HMD 1 includes the CPU 10 and the memory 15 and the HHD 2 does not include the CPU and the memory 15 in the remote support system 100 as an example of the embodiment described above, the present invention is not limited to this. HHD 2 instead of HMD 1 may have a CPU and memory. Also, both HMD1 and HHD2 may have a CPU and a memory.

[C] Additional Notes The following additional notes will be disclosed regarding the above-described embodiment and modifications.

(Supplementary Note 1)
A remote support system comprising a head mounted display, an operation terminal, and a remote support terminal,
The head mounted display is
A shooting unit for shooting a first image;
A transmitter configured to transmit the first image captured by the imaging unit to the operation terminal and the remote support terminal;
Equipped with
The operation terminal is
A first display unit configured to display the first image transmitted by the transmission unit to a first user who wears the head mounted display;
The remote support terminal is
And a second display unit configured to display the first image transmitted by the transmission unit to a second user located at a position different from the first user.
Remote support system.

(Supplementary Note 2)
The head mounted display includes a third display unit that displays a second image different from the first image to the first user.
The remote support system according to appendix 1.

(Supplementary Note 3)
The third display unit has transparency.
A part of the third display part displays the second image, and the other part of the third display part is controlled to maintain the transparency.
The remote support system according to Appendix 2.

(Supplementary Note 4)
A display system having a head mounted display and an operation terminal,
The head mounted display is
A shooting unit for shooting a first image;
A transmitter configured to transmit the first image captured by the imaging unit to the operation terminal;
Equipped with
The operation terminal is
And a display unit configured to display the first image transmitted by the transmission unit to a user who wears the head mounted display.
Display system.

(Supplementary Note 5)
The head mounted display includes a third display unit that displays a second image different from the first image to the first user.
The display system according to appendix 4.

(Supplementary Note 6)
The third display unit has transparency.
A part of the third display part displays the second image, and the other part of the third display part is controlled to maintain the transparency.
The display system according to appendix 5.

(Appendix 7)
A head mounted display connected to the operation terminal,
A shooting unit for shooting a first image;
A transmitter configured to transmit the first image captured by the imaging unit to the operation terminal in order to display the first image to a user wearing the head mount display;
, A head mounted display.

(Supplementary Note 8)
The head mounted display according to Additional Note 7, further comprising: a third display unit configured to display a second video different from the first video to the first user.

(Appendix 9)
The third display unit has transparency.
A part of the third display part displays the second image, and the other part of the third display part is controlled to maintain the transparency.
The head mounted display according to appendix 8.

(Supplementary Note 10)
An operation terminal connected to a head mounted display,
A receiver configured to receive, from the head mounted display, a first image captured by the head mounted display;
A display unit that displays the first image received by the receiving unit to a user who wears the head mounted display;
, An operation terminal.

(Supplementary Note 11)
A computer provided in a display system having a head mounted display and an operation terminal;
Taking a first image on the head mounted display;
The head mount display transmits the first video to be photographed to the operation terminal;
In the operation terminal, the transmitted first image is displayed to a user who wears the head mounted display.
Display control program to execute processing.

(Supplementary Note 12)
In the head mounted display, a second image different from the first image is displayed to the first user.
The display control program according to appendix 11, causing the computer to execute a process.

(Supplementary Note 13)
The display unit for displaying the second image has transparency.
A part of the display part displays the second image, and the other part of the display part is controlled to maintain the transparency.
The display control program according to appendix 12, causing the computer to execute a process.

(Supplementary Note 14)
A control method of a remote support system comprising a head mounted display, an operation terminal, and a remote support terminal,
Taking a first image on the head mounted display;
The head mounted display transmits the first video to be photographed to the operation terminal and the remote support terminal,
In the operation terminal, the transmitted first image is displayed to a first user who wears the head mounted display;
In the remote support terminal, the transmitted first video is displayed to a second user present at a different position from the first user.
Control method.

(Supplementary Note 15)
In the head mounted display, a second image different from the first image is displayed to the first user.
The control method according to appendix 14.

(Supplementary Note 16)
The display unit for displaying the second image has transparency.
A part of the display part displays the second image, and the other part of the display part is controlled to maintain the transparency.
The control method according to appendix 15.

100: Remote support system 1: HMD
10: CPU
110: display control unit 11: display unit 120: imaging control unit 12: imaging unit 130: communication control unit 131: 3G / LTE communication unit 132: WiFi communication unit 133: communication unit 134: HDMI communication unit 134: communication unit 135: USB communication unit 135: communication unit 140: voice input / output unit 14: earphone microphone 15: memory 2: HHD
20: control unit 21: display unit 22: input unit 23: LTE communication unit 24: WiFi communication unit 25: communication unit 26: HDMI communication unit 27: USB communication unit 3: communication cable 4: operator 5: object 6: remote Support terminal 7: Supporter 8: Server 61: Display unit

Claims (8)

A remote support system comprising a head mounted display, an operation terminal, and a remote support terminal,
The head mounted display is
A shooting unit for shooting a first image;
A transmitter configured to transmit the first image captured by the imaging unit to the operation terminal and the remote support terminal;
Equipped with
The operation terminal is
A first display unit configured to display the first image transmitted by the transmission unit to a first user who wears the head mounted display;
The remote support terminal is
And a second display unit configured to display the first image transmitted by the transmission unit to a second user located at a position different from the first user.
Remote support system. The head mounted display includes a third display unit that displays a second image different from the first image to the first user.
The remote support system according to claim 1. The third display unit has transparency.
A part of the third display part displays the second image, and the other part of the third display part is controlled to maintain the transparency.
The remote support system according to claim 2. A display system having a head mounted display and an operation terminal,
The head mounted display is
A shooting unit for shooting a first image;
A transmitter configured to transmit the first image captured by the imaging unit to the operation terminal;
Equipped with
The operation terminal is
And a display unit configured to display the first image transmitted by the transmission unit to a user who wears the head mounted display.
Display system. A head mounted display connected to the operation terminal,
A shooting unit for shooting a first image;
A transmitter configured to transmit the first image captured by the imaging unit to the operation terminal in order to display the first image to a user wearing the head mount display;
, A head mounted display. An operation terminal connected to a head mounted display,
A receiver configured to receive, from the head mounted display, a first image captured by the head mounted display;
A display unit that displays the first image received by the receiving unit to a user who wears the head mounted display;
, An operation terminal. A computer provided in a display system having a head mounted display and an operation terminal;
Taking a first image on the head mounted display;
The head mount display transmits the first video to be photographed to the operation terminal;
In the operation terminal, the transmitted first image is displayed to a user who wears the head mounted display.
Display control program to execute processing. A control method of a remote support system comprising a head mounted display, an operation terminal, and a remote support terminal,
Taking a first image on the head mounted display;
The head mounted display transmits the first video to be photographed to the operation terminal and the remote support terminal,
In the operation terminal, the transmitted first image is displayed to a first user who wears the head mounted display;
In the remote support terminal, the transmitted first video is displayed to a second user present at a different position from the first user.
Control method.

Images