JP6074894B2 - Information display device - Google Patents

Description

The present invention relates to information display equipment.

  There is known a video conference system capable of handwritten writing on a video to be distributed. In such a video conference system, an image output from a personal computer or an image captured by a camera is displayed on a display device, and writing is performed on the display device with an input pen or the like.

  Patent Document 1 discloses that two electronic cameras are connected to each other via a communication line and used for a video conference. The electronic camera on the sender side transmits the captured image to the electronic camera on the receiver side. The electronic camera on the sender side displays a memo image (line drawing) drawn by the sender on the touch tablet on the display unit of the electronic camera and also transmits it to the electronic camera on the receiver side. The electronic camera on the receiver side displays the memo image superimposed on the received image.

Japanese Patent No. 4349371

  However, in the technique of Patent Document 1, when the shooting direction of the camera is moved, the relationship between the video displayed on the display device and the superimposed line drawing is deviated, so it is not clear what the image points to. There was a problem of disappearing. As described above, the technique of Patent Document 1 has a problem that a communication error occurs.

The present invention was made in view of the above circumstances, and an object thereof is to provide an information display equipment that can reduce the mistakes of communication between terminals.

In order to achieve the above object, an information display device according to one embodiment of the present invention is obtained by using a display unit that displays an image, an input unit that detects a user operation, an imaging unit that captures a subject, and the imaging unit. When converting the obtained image to the first resolution or the second resolution higher than the first resolution, and displaying the image obtained by the imaging unit on the display unit by operating the input unit, An image synthesis unit that displays the image of the second resolution converted by the conversion unit on the display unit, and the conversion unit converts the image obtained by the imaging unit to the first resolution, and that converts an image into the second resolution.

  The information display apparatus according to the present embodiment pauses an image to make a still image, and then superimposes a line drawing or the like on the temporarily stopped image. As a result, according to the present embodiment, even if the shooting direction of the photographer changes, it is possible to accurately overlay the screen image and the line drawing, thereby reducing communication errors between terminals.

It is a perspective view of the head mounted display concerning this embodiment. It is the perspective view which looked at the head mounted display concerning this embodiment from the back side. It is a figure which shows the mounting | wearing form of the head mounted display which concerns on this embodiment. It is a horizontal sectional view of a display body concerning this embodiment. It is a perspective view which shows the remote control which concerns on this embodiment. It is a functional block diagram of the head mounted display concerning this embodiment. It is a functional block diagram of the process part which concerns on this embodiment. 1 is a system configuration diagram of a communication system including a head mounted display according to the present embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a figure which shows the example of a display of the avatar which concerns on this embodiment. It is a figure explaining an example of composition of an image displayed on LCD or a display part concerning this embodiment. It is an example of the image where the cursor concerning this embodiment was displayed. It is an example of the line drawing which concerns on this embodiment. It is a figure which shows an example of the grid which concerns on this embodiment. It is a figure which shows the other example of the grid which concerns on this embodiment.

In the information display apparatus according to the present embodiment, when communication based on a screen is performed on a video between a plurality of apparatuses, the video is temporarily stopped to be a still image, and then a line drawing or the like is superimposed on the image.
In addition, the information display apparatus distributes an image captured by a terminal that first requested a still image to all terminals.
Hereinafter, an example in which the information output apparatus is applied to a head mounted display will be described, but the present invention is not limited to this.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited thereto. In the following description, an XYZ orthogonal coordinate system is set as necessary, and the positional relationship of each part will be described with reference to this XYZ orthogonal coordinate system. A predetermined direction in the horizontal plane is defined as an X-axis direction, a direction orthogonal to the X-axis direction in the horizontal plane is defined as a Y-axis direction, and a direction orthogonal to each of the X-axis direction and the Y-axis direction (that is, a vertical direction) is defined as a Z-axis direction. Further, the rotation (inclination) directions around the X axis, Y axis, and Z axis are the θX, θY, and θZ directions, respectively.

  FIG. 1 is a perspective view of the head mounted display of the present embodiment. FIG. 2 is a perspective view of the head mounted display of the present embodiment as viewed from the back side. FIG. 3 is a diagram showing a mounting form of the head mounted display of the present embodiment.

The head mounted display (head mounted device) 1 is a monocular head mounted display including a display main body 20 and a headband 40 that is mounted on a user's head and supports the display main body 20. The head mounted display 1 of the present embodiment can be used with both eyes as shown in FIG. FIG. 3A shows a state where the user is viewing the display unit 60 with the right eye, and FIG. 3B shows a state where the user is viewing with the left eye. As shown in FIG. 2, the display main body 20 and the headband 40 are configured to be detachable via a connecting pin 41.
1 and 2, the longitudinal direction of the display main body 20 is defined as the Y-axis direction, and the direction in which the headband 40 sandwiches the user's head is defined as the X-axis direction.

Hereinafter, the configuration of each part of the head mounted display 1 will be described in detail.
The display main body 20 includes a device main body 21 that includes a main circuit and includes an operation unit and various interfaces, and a display unit 60 that is connected to the tip of the device main body 21.

  The apparatus main body 21 has a substantially plate-shaped housing 21A shown in FIG. In the present embodiment, an end portion (+ Y side end portion) on the side where the connection portion with the headband 40 is provided in the apparatus main body portion 21 is a base end portion, and an end portion on the opposite side of the base end portion ( The end on the -Y side) is the tip. Further, in a state where the apparatus main body 21 is attached to the headband 40, the headband 40 side (+ X side) of the apparatus main body 21 is the inside, and the opposite side (−X side) from the headband 40 is the outside.

As shown in FIG. 1, a main switch 28, a touch switch 34 (input unit), and a sound collecting microphone 24 are disposed on the outer surface of the apparatus main body 21 along the longitudinal direction of the housing 21A. .
The main switch 28 is a switch for turning on / off the power of the display body 20. The touch switch 34 is a touch panel that can perform various operations of the head mounted display 1 by touching the surface with a finger or the like. The sound collection microphone 24 is an external microphone that collects environmental sounds.

  As shown in FIG. 2, an ear speaker 23, an audio connector 26, and a headband hinge 32 having a connection hole 31 are provided on the base end side of the inner surface of the apparatus main body 21. A heart rate sensor 137 is provided at the center of the inner surface of the apparatus main body 21. A call microphone 37 is provided at the tip of the inner side surface of the apparatus main body 21.

  The ear speaker 23 is disposed in the vicinity of the user's ear. Audio information is transmitted from the ear speaker 23 to the user. The audio connector 26 is an audio input / output terminal to which an earphone (not shown) is connected, for example. The headband hinge 32 is a joint part with the headband 40. The voice of the user is input to the call microphone 37.

  The heart rate sensor 137 is a sensor that measures a user's heart rate by contacting the surface of the user's face. The heart rate sensor 137 includes a light emitting unit including a light emitting diode and the like, and a light receiving unit that detects light reflected inside the user's skin. The heart rate sensor 137 counts the heart rate by detecting a change in the amount of reflected light due to a change in blood flow. The heart rate sensor 137 is disposed near the user's eyes, but if the configuration is such that infrared light is emitted from the light emitting unit, the user will not feel glare.

A USB connector 25, an operation switch 30, and a video connector 27 are provided on the side end surface of the apparatus main body 21 on the base end side.
The USB connector 25 is a connection terminal of a USB (Universal Serial Bus) device. In the present embodiment, for example, a remote controller (remote controller) 140 shown in FIG. 5 is connected.
The operation switch 30 is a pointing device such as a trackball or a stick. The operation switch 30 is provided so as to face the screen displayed on the display unit 60. As a result, the left-right direction of the operation on the operation switch 30 matches the left-right direction of the screen, so that the user can intuitively operate the operation switch 30 while looking at the screen.
The video connector 27 is a video input / output terminal.

FIG. 4 is a horizontal sectional view of the display main body 20 of the present embodiment.
As shown in FIG. 4, the apparatus body 21 includes a plate-like circuit board 29 extending along the longitudinal direction of the housing 21 </ b> A and a battery 33. A control circuit, a power supply circuit, and the like (not shown) are mounted on the circuit board 29, and are electrically connected to each part of the display body 20 via wiring (not shown).

  A display panel 36 made of a liquid crystal panel and a backlight 35 are arranged inside the touch switch 34 exposed on the outer surface of the apparatus main body 21. In the present embodiment, the display image on the display panel 36 is displayed through the touch switch 34. The display panel 36 and the backlight 35 may be an organic EL panel or an electrophoresis panel.

  The headband hinge 32 is a ball joint composed of a concavely curved housing portion 32a provided in the housing 21A and a spherical portion 32b fitted to the housing portion 32a. The spherical portion 32b has a spherical side surface portion and two plane portions formed in parallel to each other so as to sandwich the side surface portion. A connecting hole 31 is formed so as to penetrate the two flat portions vertically. The connection hole 31 is formed in a hexagonal shape when viewed in the axial direction. The display main body 20 and the headband 40 are connected by inserting the connection pin 41 of the headband 40 into the connection hole 31.

  By providing the headband hinge 32, the display main body 20 can be rotated in the A direction (around the X axis centering on the headband hinge 32) shown in FIG. In the present embodiment, the rotatable range of the display body 20 is about 270 °. By the rotation operation around the X axis, a switching function between a mode in which an image is observed with the right eye shown in FIG. 3A and a mode in which an image is observed with the left eye shown in FIG.

  Further, since the headband hinge 32 is a ball joint, the display main body 20 can also be swung in the B direction (around the Z axis centering on the headband hinge 32) shown in FIG. By this swinging operation, the position of the display main body 20 relative to the user's eyes and ears can be adjusted.

  The heart rate sensor 137 in the vicinity of the headband hinge 32 is provided so as to protrude from the inner surface of the apparatus main body 21, and can contact the surface of the user's face when the head mounted display 1 is worn. The battery 33 may be either a primary battery or a secondary battery.

  As shown in FIGS. 1 and 2, the display unit 60 is connected to the distal end portion of the apparatus main body unit 21. In the present embodiment, the headband 40 side of the display unit 60 is the inside, and the side opposite to the headband 40 is the outside. The display unit 60 is an arm member that has a curved shape in a top view (Z-axis view), and has a shape that curves inward from the connecting portion with the apparatus main body 21 toward the distal end side. A finder opening 67 is provided on the inner surface of the display unit 60. A camera 64 (imaging unit) is provided on the outer surface of the display unit 60.

  As shown in FIG. 4, the display unit 60 is connected to the apparatus main body 21 via a display hinge 61. The display hinge 61 is a ball joint composed of a concave curved housing 61a formed in the housing 60A of the display 60 and a spherical portion 61b formed in the apparatus main body 21 and fitted in the housing 61a. .

  The spherical portion 61b of the apparatus main body 21 has an inclined surface extending obliquely with respect to the longitudinal direction of the casing 21A formed at the front end of the outer surface of the apparatus main body 21, and the normal direction of the inclined surface (Y in FIG. 4). It is provided so as to protrude in the “axial direction”.

  The display unit 60 can be freely rotated around the Y ′ axis with respect to the spherical portion 61 b of the display hinge 61. In the present embodiment, the display hinge 61 is provided on the inclined surface of the apparatus main body 21, and the display unit 60 has a shape curved inward.

Further, since the display hinge 61 is a ball joint, the display unit 60 can swing in a direction orthogonal to the rotation plane around the Y ′ axis (generally along the X ′ axis). This swinging operation facilitates position adjustment of the finder opening 67 by the user. If the interval between the display unit 60 and the apparatus main body unit 21 is increased, the swinging width of the display unit 60 can be increased.
That is, according to the present embodiment, the display hinge 61 serving as the holding unit can swing the display unit 60 around at least one axis.

  The spherical portion 61b of the display hinge 61 is formed with a through hole 61c that penetrates the spherical portion 61b in the height direction (Y′-axis direction). The inside of the display unit 60 and the inside of the apparatus main body unit 21 are communicated with each other through the through hole 61c. A cable (not shown) is inserted through the through hole 61c. The circuit board 29 and each part of the display unit 60 are electrically connected via the inserted cable.

  Inside the display unit 60, there are a backlight 62, a display panel 63 (display unit), a camera 64 (imaging unit), a prism 65, a reflection mirror 66, a front light 68, and a front speaker 70. An image lens 71 and an image sensor 72 are provided.

  The prism 65 has a configuration in which a substantially triangular first prism 65a and a second prism 65b are bonded to each other on a top view (Z-axis view). A display panel 63 made of a liquid crystal panel is provided at a position facing one of the other two surfaces of the first prism 65a other than the bonding surface. A backlight 62 that transmits and illuminates the display panel 63 is disposed on the back surface of the display panel 63. A reflection mirror 66 is disposed at a position facing the other surface of the first prism 65a. The reflection mirror 66 is located substantially in front of the finder opening 67.

  Of the two surfaces other than the bonding surface of the second prism 65 b, one surface is a viewfinder eyepiece disposed in the viewfinder opening 67. An imaging element 72 is disposed opposite to the other surface of the second prism 65b via the imaging lens 71.

  In the display unit 60, the image displayed on the display panel 63 is emitted from the finder opening 67 through the first prism 65a, the reflection mirror 66, and the second prism 65b, and is observed by the user. The eye of the user looking at the finder opening 67 is imaged on the image sensor 72 via the second prism 65 b and the imaging lens 71. The eye image acquired through the image sensor 72 is used for analyzing the direction of the user's line of sight, blinking, and facial expression. The backlight 62 is also used as a lighting device for photographing the eye by the image sensor 72.

The camera 64 includes, for example, an image sensor with 5 to 10 million pixels, and is configured to be capable of an autofocus operation.
In the case of this embodiment, the shooting direction of the camera 64 is set so as to coincide with the user's line-of-sight direction. The adjustment between the photographing direction and the line-of-sight direction may be performed mechanically or by image processing. For example, the camera 64 is configured to be able to acquire a wide-angle image, and a part of the electrically captured image is extracted as an image in the line-of-sight direction based on the information on the line-of-sight direction acquired from the user's eye image. 63 is displayed. As a result, it is possible to shoot and display an image in front of the user without providing an adjustment mechanism in the camera 64.

  The front light 68 is, for example, an LED light. The front light 68 may include red, green, and blue light emitting elements, and may be configured to emit any color at any timing. The front light 68 may be used as a device that displays information to the outside depending on the light emission color and the light emission timing, or may be used as an illumination device when photographing with the camera 64.

  Although not shown in FIG. 4, the display unit 60 according to the present embodiment is provided with a laser transmitter. The laser transmitter can be provided in the vicinity of the front light 68, for example. By irradiating, for example, red laser light emitted from a laser transmitter forward, pointing (display of an index) with the laser light is possible.

Next, the headband 40 will be described with reference to FIG.
As shown in FIG. 2, the headband 40 includes a pair of head pads (mounting members) 46 and 47 that sandwich the user's head, a first headband 43, a second headband 44, and a rotation mechanism 56. , 57.

  The first headband 43 is an elastic member having an arc shape as a whole. A joint portion 43 a for bending the first headband 43 is provided on the top of the first headband 43. At both ends of the first headband 43, bearing portions 43b and 43c constituting the rotation mechanisms 56 and 57 are provided, respectively. Bearing portions 43d and 43e to which the head pads 46 and 47 are connected are provided further on the front end side of the band than the bearing portions 43b and 43c.

  The second headband 44 is an elastic member having an arc shape as a whole. A joint portion 44 a for bending the second headband 44 is provided on the top of the second headband 44. At both ends of the second headband 44, shaft members 44b and 44c constituting rotation mechanisms 56 and 57 are provided.

  In the present embodiment, the second headband 44 has a configuration in which the surface of the spring member 48 made of a metal such as stainless steel is covered with a flexible material such as a resin. The spring member 48 of the second headband 44 generates a spring force that holds the user's head. The second headband 44 is formed wider than the first headband 43. The formation part of the joint part 44a is a mark display part 49 formed wider than the other band parts. A product tag or the like is attached to the mark display section 49 by sticker or printing. The entire second headband 44 may be made of metal.

  The head pad 46 includes a plate-like support plate 46a and an elastic member 46b having an arched cross section provided on one surface side of the support plate 46a. On the surface of the support plate 46a opposite to the elastic member 46b, a substantially hexagonal column-shaped connecting pin 41 is erected in a posture perpendicular to the surface. The connecting pin 41 is pivotally supported by a bearing portion 43 d provided at one end of the first headband 43. As a result, the head pad 46 can rotate around the connecting pin 41.

  The head pad 47 includes a plate-like support plate 47a and an elastic member 47b having an arcuate cross section provided on one surface of the support plate 47a. A shaft member 47c is provided on the surface of the support plate 47a opposite to the elastic member 47b. The shaft member 47 c is pivotally supported by a bearing portion 43 e provided at the tip of the first headband 43. Thereby, the head pad 47 can rotate around the shaft member 47c.

  With this configuration, the second headband 44 is urged to rotate in a direction away from the first headband 43 by the rotation mechanism 56, while being within a predetermined angle with respect to the first headband 43 by the rotation mechanism 57. The rotation is restricted to the range. As a result, the second headband 44 is held at a predetermined angle with respect to the first headband 43. Therefore, the rotation mechanisms 56 and 57 function as an angle holding mechanism in the headband 40.

FIG. 5 is a perspective view showing the remote controller of the present embodiment.
The remote controller 140 includes a housing 140A, a touch pad 141, an operation switch 147, a USB connector 142, indicators 143a, 143b, and 143c, a mounting clip 144, and an indicator switch 150.

  The remote controller 140 is electrically connected to the apparatus main body 21 by connecting one terminal of the USB cable 148 to the USB connector 142 and connecting the other terminal to the USB connector 25 of the apparatus main body 21. A commercially available USB cable can be used as the USB cable 148. In the present embodiment, a plurality of clips 149 are provided in the middle of the USB cable 148. The plurality of clips 149 can be used to fix the USB cable 148 along the second headband 44, for example.

  The remote control 140 incorporates a communication circuit for performing wireless communication with the display main body 20 and other electronic devices. As a communication method of the communication circuit, for example, Bluetooth (registered trademark), wireless LAN, infrared communication, UWB (Ultra Wide Band), Transfer Jet, or the like can be used. With such a communication circuit, communication with the display main body 20 is possible even when the USB cable 148 is not connected.

  The remote control 140 may incorporate a G sensor, an acceleration sensor, and a direction sensor. Information detected by these sensors can also be communicated to the display body 20 via the communication circuit described above.

  The touch pad 141 is provided on one main surface of the housing 140A. The touch pad 141 is an operation unit for operating the display main body 20 when the user touches. A display panel and a backlight may be provided in the housing 140 </ b> A on the back side of the touch pad 141, similarly to the touch switch 34 of the apparatus main body 21.

  The operation switch 147 is a switch for manually operating the remote controller 140 and the display main body 20. The operation switch 147 may be provided as necessary. The indicator switch 150 provided on the side surface of the housing 140A is a switch for performing a predetermined operation using the indicators 143a to 143c.

The USB connector 142 is used not only for connection with the apparatus main body 21 but also for connection with a charging terminal of the remote control 140 or a PC when the remote control 140 has a built-in rechargeable battery.
When the remote controller 140 has a built-in battery, it can also be used as a power supply terminal for the apparatus main body 21. By making the battery of the remote control 140 usable as an auxiliary power source, the battery capacity of the device main body 21 can be reduced, and the device main body 21 can be reduced in size and weight.

  Indicators 143a, 143b, and 143c are arranged around the touch pad 141. In the present embodiment, an indicator 143a is disposed at a position along one side of the touch pad 141, and an indicator 143b and an indicator 143c are disposed on the opposite side of the touch pad 141 with respect to the indicator 143a.

  The indicators 143a to 143c include light emitting elements such as LEDs. In the present embodiment, the indicator 143a includes a green light emitting element, the indicator 143b includes an orange light emitting element, and the indicator 143c includes a blue light emitting element. These light emitting elements are configured to be capable of pattern light emission at controlled timing. The pattern light emission of the indicators 143a to 143c can be turned on and off by operating the indicator switch 150 on the side surface of the housing 140A.

  The pattern light emission of the indicators 143a to 143c indicates, for example, the state of the remote controller 140 (for example, the up / down / left / right tilt state of the remote controller 140). It is possible to transmit the state of the remote controller 140 to the apparatus main body 21 by photographing such pattern emission with the camera 64 of the apparatus main body 21 and analyzing the acquired image. The information transmission method using the indicators 143a to 143c can suppress power consumption as compared with the case where wireless communication such as Bluetooth (registered trademark) is used.

  The attachment clip 144 is provided on the surface of the housing 140A opposite to the touch pad 141. In the case of this embodiment, the attachment clip 144 is an S-shaped spring plate member having one end fixed to the housing 140A. With the attachment clip 144, the remote controller 140 can be attached to a user's clothes, belt, or the like.

FIG. 6 is a functional block diagram of the head mounted display (hereinafter referred to as HMD) 1 of the present embodiment.
The HMD 1 includes various electric circuits centering on the processing unit 123. The processing unit 123 is a CPU (Central Processing Unit) and is connected to various circuits of the HMD 1 and comprehensively controls the HMD 1.

The battery 33 supplies power to the power supply circuit 120.
The power supply circuit 120 supplies the power supplied from the battery 33 to each unit of the HMD 1 under the control of the processing unit 123.

  In the present embodiment, the processing unit 123 includes an encoder 129 and a decoder 121, a power supply circuit 120, a BL driver 126, an operation switch 30, a flash memory 122, a BT communication circuit 130, a WiFi communication circuit 131, an acceleration sensor 132, and a geomagnetic sensor. 133, front light 68, 3G / LTE communication circuit 138, laser transmitter 73, angular velocity sensor 134, GPS sensor 135, temperature / humidity sensor 136, heart rate sensor 137, memory 127, main switch 28, and touch switch 34 are connected. Yes.

  The encoder 129 encodes (encodes) the audio signal and the video signal into audio data and video data of a predetermined method. Connected to the encoder 129 are a camera 64, an image sensor 72, a sound collection microphone 24, a call microphone 37, an audio connector 26, and a video connector 27.

  The encoder 129 includes an audio signal input from the sound collection microphone 24 and the call microphone 37, a video signal input from the camera 64, an audio signal input from the audio connector 26, a video signal input from the video connector 27, and The video signal of the image sensor 72 that captures the user's eyes is input. The audio signal and video signal input to the encoder 129 are encoded into audio data and video data, and then input to the processing unit 123. The input audio data and video data are used for a reproduction operation by the processing unit 123 or recorded in the flash memory 122.

  The decoder 121 decodes (decodes) audio data and video data into audio signals and video signals. An LCD driver 125, a speaker amplifier 162, an audio connector 26, and a video connector 27 are connected to the decoder 121. The LCD driver 125 is a drive control device for a liquid crystal panel, and is connected to the display panel 36 and the display panel 63. The speaker amplifier 162 is a device that amplifies an audio signal and outputs it to the speaker, and is connected to the ear speaker 23 and the front speaker 70.

  When reproducing the video data, the video data recorded in the flash memory 122 or the video data input from the encoder 129 is input to the decoder 121 via the processing unit 123. The video data input to the decoder 121 is decoded into a video signal and then supplied to the display panels 36 and 63 via the LCD driver 125. Then, an image based on the image data is displayed on the display panel 36 or the display panel 63 to which the image signal is input. The video signal output from the decoder 121 to the video connector 27 is output to an external device via the video connector 27.

  When displaying the video, the processing unit 123 turns on the backlight 35 for the display panel 36 and the backlight 62 for the display panel 63 as necessary. The BL driver 126 is a drive control device for the backlight, and is connected to the backlight 35 and the backlight 62. The processing unit 123 transmits a drive signal to the BL driver 126, and the BL driver 126 lights the backlights 35 and 62 individually.

  When reproducing the audio data, the audio data recorded in the flash memory 122 or the audio data input from the encoder 129 is input to the decoder 121 via the processing unit 123. The audio data input to the decoder 121 is decoded into an audio signal and then output to one or both of the ear speaker 23 and the front speaker 70 via the speaker amplifier 162. Then, sound is output from the ear speaker 23 or the front speaker 70 to which the sound signal is input. The audio signal output from the decoder 121 to the audio connector 26 is output to a stereo earphone (not shown) via the audio connector 26.

In the case of this embodiment, the ear speaker 23 and the front speaker 70 are assumed to use monaural sound, and the ear speaker 23 and the front speaker 70 emit a sound in which left and right audio signals are synthesized.
On the other hand, when an audio signal is output to a stereo earphone (not shown), left and right channel sounds are output to the left and right speakers, respectively.

The memory 127 stores a control program executed by the processing unit 123.
When the user operates the main switch 28 for turning on / off the entire power source, the operation switch 30 for performing a pointing operation in the screen, or the touch switch 34 for performing various operations by a touch operation, the operation is performed from these switches. Is output to the processing unit 123. The processing unit 123 detects an operation based on the control signal, and executes an operation defined in the control program.

  The BT communication circuit 130 is a communication circuit for performing Bluetooth (registered trademark) communication with other devices. The WiFi communication circuit 131 is a communication circuit for performing wireless LAN communication (IEEE 802.11) with other devices. The 3G / LTE communication circuit 138 is a communication circuit for performing mobile communication with other devices.

  The acceleration sensor 132 is used for detecting the inclination of the HMD 1. The geomagnetic sensor 133 is used for detecting the direction of the HMD 1. An angular velocity sensor (gyro sensor) 134 is used to detect rotation of the HMD 1. The GPS sensor 135 is used for positioning detection using GPS (Global Positioning System). The temperature / humidity sensor 136 is used to detect the temperature and humidity of the environment. The heart rate sensor 137 contacts the user's cheek and detects the user's heart rate.

FIG. 7 is a functional block diagram of the processing unit 123 of this embodiment.
As illustrated in FIG. 7, the processing unit 123 includes an input detection unit 301 (input unit), an image switching unit 302, a transmission / reception unit 303, a line drawing generation unit 304, a cursor generation unit 305, an avatar selection unit 306, an avatar storage unit 307, And an image composition unit 308. In FIG. 7, the WiFi communication circuit 131 and the 3G / LTE communication circuit are collectively referred to as communication interfaces (131, 138).

The input detection unit 301 determines the type of operation performed on the touch switch 34, and sends control signals based on the determined result to the image switching unit 302, the transmission / reception unit 303, the avatar selection unit 306, and the image composition unit 308. Output. The types of operations performed on the touch switch 34 are a transmission instruction, a reception instruction, a cursor display instruction, a line drawing display instruction, and the like. When a predetermined area on the touch switch 34 is touched, the input detection unit 301 determines the type of these operations based on the touched area. Further, the input detection unit 301 detects the movement of the finger on the touch switch 34 at the time of cursor display or line drawing display.
In the case of cursor display, the input detection unit 301 outputs information indicating a position corresponding to the finger movement to the cursor generation unit 305. In the case of line drawing display, the input detection unit 301 outputs information indicating the position according to the finger movement to the line drawing generation unit 304.

  The image switching unit 302 outputs the image data read from the flash memory 122 or the image data output from the encoder 129 to the image composition unit 308 based on the control signal output from the input detection unit 301. The image switching unit 302 extracts an image switching signal from the reception signal output from the transmission / reception unit 303. The image switching unit 302 outputs the image data read from the flash memory 122 or the image data output by the encoder 129 to the image composition unit 308 based on the extracted image switching signal. The image switching unit 302 converts the resolution of the image data output from the encoder 129 or the image data read from the flash memory 122 based on the control signal output from the input detection unit 301, and converts the converted image data into an image. The data is output to the combining unit 308.

  Based on the control signal output from the input detection unit 301, the transmission / reception unit 303 transmits the image data output from the image composition unit 308 via the communication interfaces (131, 138). The transmission / reception unit 303 outputs reception signals received via the communication interfaces (131, 138) to the image switching unit 302, the line drawing generation unit 304, and the cursor generation unit 305.

  The line drawing generation unit 304 generates line drawing data with a locus corresponding to the finger movement based on the information indicating the position corresponding to the finger movement output from the input detection unit 301. The line drawing generation unit 304 outputs the generated line drawing data to the image composition unit 308. The line drawing generation unit 304 extracts line drawing data from the reception signal output from the transmission / reception unit 303, and outputs the extracted line drawing data to the image composition unit 308.

  The cursor generation unit 305 generates cursor image data without a locus corresponding to the finger movement based on the information indicating the position corresponding to the finger movement output from the input detection unit 301. The cursor generation unit 305 outputs the generated cursor image data to the image composition unit 308. The cursor generation unit 305 extracts cursor image data accompanied by position information from the reception signal output from the transmission / reception unit 303 and outputs the extracted cursor image data to the image composition unit 308.

The avatar selection unit 306 reads avatar image data set in advance for the user of the HMD 1 from the avatar storage unit 307 based on the control signal output by the input detection unit 301. The avatar selection unit 306 extracts information indicating an avatar of another user from the reception signal output from the transmission / reception unit 303. The avatar selection unit 306 reads out avatar image data corresponding to the user from the avatar storage unit 307 based on the extracted information indicating the avatar of the other user.
The avatar is an illustration of the face of the user of HMD1 and the user who is transmitting / receiving to / from HMD1. The avatar selection unit 306 outputs image data indicating the read avatar to the image composition unit 308. The avatar selection unit 306 is based on the output of various sensors (acceleration sensor 132, geomagnetic sensor 133, angular velocity sensor 134, GPS sensor 135, temperature / humidity sensor 136, heart rate sensor 137, etc.) and information on the environment in which HMD1 is used. Is output to the image composition unit 308.
The avatar selection unit 306 generates each piece of image information indicating that each terminal is being called (hereinafter also referred to as image information being called) based on the received signal output from the transmission / reception unit 303, and the generated call The image information inside is output to the image composition unit 308.

  In the avatar storage unit 307, for example, a plurality of avatar image data is stored in advance.

The image composition unit 308 outputs image data output from the image switching unit 302, line drawing data output from the line drawing generation unit 304, cursor image data with position information output from the cursor generation unit 305, and output from the avatar selection unit 306. The image is synthesized based on the image data indicating the avatar.
When the control signal output from the input detection unit 301 is a line drawing display instruction (second mode), the image composition unit 308 pauses the video displayed on the display panel 63 and generates a still image. The image switching unit 302 outputs the generated still image data and the environment information input from the avatar selection unit 306 to the transmission / reception unit 303, the video connector 27, and the LCD driver 125.
Note that the video displayed on the display panel 63 is image data read from the flash memory 122, image data output from the encoder 129, or video data from another terminal included in the reception signal output from the transmission / reception unit 303. .
The resolution of the image generated by pausing is higher than the resolution at the time of transmitting the video. The resolution of a still image is, for example, an image having 2500 pixels in the horizontal direction and 1875 pixels in the vertical direction. Note that the image composition unit 308 may perform frame advance over a period during which the user draws a line drawing or a period during which the line drawing is used for explanation, instead of pausing.
When the control signal output from the input detection unit 301 is a cursor display instruction (first mode), the image composition unit 308 does not pause the video displayed on the display panel, and selects the synthesized image data and avatar. The environment information input from the unit 306 is output to the transmission / reception unit 303, the video connector 27, and the LCD driver 125.

Next, a procedure when the HMD 1 communicates with the head mounted display 500 (hereinafter referred to as HMD 500), the PC 210, and the PC 220 used by other users will be described.
FIG. 8 is a system configuration diagram of a communication system including the head mounted display 1 according to the present embodiment. 9 to 19 are flowcharts illustrating an example of a processing procedure according to the present embodiment.

  As illustrated in FIG. 8, the communication system includes an HMD 1, an HMD 500, a PC 210, a PC 220, and a server 200. The HMD 1, HMD 500, PC 210, PC 220, and server 200 are connected to the network 230. The network 230 is a network such as the Internet connected via a wireless LAN (Local Area Network), a 3G line, an LTE line, or the like.

8, the HMD 1 has the configuration described with reference to FIG. 6, but shows only functional units necessary for explaining the processing procedure. The display panel 63 of the HMD 1 is hereinafter referred to as an LCD 63.
As shown in FIG. 8, the HMD 500 mainly includes a flash memory 522, a processing unit 523, communication interfaces (531, 538), a geomagnetic sensor 533, a touch switch 534, a display panel (display unit) 563 (hereinafter referred to as an LCD 563), A camera 564 is provided. Each function part of HMD500 is equivalent to each function part of HMD1.

Next, the configuration of the server 200 will be described.
As illustrated in FIG. 8, the server 200 includes a processing unit 201, a storage unit 202, and a communication interface 203.

  The processing unit 201 of the server 200 includes image data (moving image, still image, line drawing, etc.) received from each terminal (HMD1, HMD500, PC210, and PC220) via the communication interface 203, audio data, environment information of each terminal, In addition, a communication record such as a request from each terminal is recorded in the storage unit 202. The processing unit 201 reads image data recorded in the storage unit 202 in response to a request from each terminal, and transmits the read image data to each terminal.

The storage unit 202 stores communication records such as image data (moving images and still images) received from each terminal, audio data, environment information of each terminal, and requests from each terminal.
The communication interface 203 includes, for example, a WiFi communication circuit, a 3G communication circuit, an LTE communication circuit, a wired LAN circuit, and the like.

Next, the configuration of the PC 210 and the PC 220 will be described. The PC 210 and the PC 220 are, for example, personal computers.
The PC 210 includes a processing unit 211, a storage unit 212, a communication interface 213, an input unit 214, a display unit 215, and a camera 216.
The PC 220 includes a processing unit 221, a storage unit 222, a communication interface 223, an input unit 224, a display unit 225, and a camera 226. Each functional unit of the PC 220 is equivalent to each functional unit of the PC 210.

The processing unit 211 of the PC 210 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like. The processing unit 211 controls the communication interface 213, the input unit 214, the display unit 215, and the camera 216. The processing unit 211 switches the image data stored in the storage unit 212, the image data acquired through the communication interface 213, and the image data captured by the camera 216 through the communication interface 213 as will be described later. Switch based on signal. The processing unit 211 outputs the switched image data to the communication interface 213 and the network 230. The processing unit 211 is based on the avatar image data stored in association with the PC 210 stored in the storage unit 212, the avatar image data of other devices acquired through the communication interface 223, and the image switching signal. Synthesize the switched image data. The processing unit 211 outputs the combined image data to the display unit 215. The processing unit 211 controls to display a cursor image or a line drawing image on the display unit 215 based on the operation of the input unit 214. The processing unit 211 controls to display a cursor image or a line drawing image on the display unit 215 based on the reception information acquired via the communication interface 213.
Hereinafter, an avatar corresponding to each device is referred to as an avatar of the PC 210, for example.

  The storage unit 212 is, for example, a memory or an HDD (hard disk). The storage unit 212 stores an OS (operation system) of the PC 210, an application for communicating with the HMD1, the HMD 500, and the PC 220, avatar image data set in advance for the user of the PC 210, and the like.

  The communication interface 213 is an interface that performs communication such as a wireless LAN, a wired LAN, a 3G line, an LTE line, or the like. The communication interface 213 outputs information received via the network 230 to the processing unit 211. The communication interface 213 transmits information output from the processing unit 211 to the HMD 1, the HMD 500, the PC 220, and the server 200 via the network 230.

  The input unit 214 is, for example, a keyboard, a mouse, a tablet, and an operation pen. The input unit 214 outputs the input information to the processing unit 211. Note that when the display unit 215 has a touch panel function, the display unit 215 may also function as the input unit 214.

The display unit 215 displays an image based on the image data output from the processing unit 211.
As an example, the image captured by the HMD 1, the avatar of the HMD 1, the avatar of the HMD 500, the avatar of the PC 220, and the line drawing image corresponding to the operation of the input unit 214 of the PC 210 are combined and displayed on the display unit 215.

  The camera 216 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) having an image sensor with 5 to 10 million pixels, or a CCD (Charge Coupled Device).

  Next, a communication procedure performed between the HMD1, the HMD500, the PC 210, and the PC 220 will be described. 9 to 19, each terminal (HMD1, HMD500, PC210, PC220, and server 200) has a communication interface (131, 138, 531, 538, 213, 223, and 203) and a network. Communication is performed via 230.

First, a procedure for making a call from the HMD 1 to another terminal will be described.
(Step S1) As shown in FIG. 9, the user of HMD1 operates the touch switch 34 of HMD1 in order to communicate with HMD500, PC210, and PC220. Based on the control signal output from the touch switch 34, the processing unit 123 of the HMD 1 performs a call operation indicating that communication with the HMD 500, the PC 210, and the PC 220 is started.

(Step S2) The processing unit 123 of the HMD 1 adds the avatar additional information to the avatar of the HMD 1, the avatar of the HMD 500, the avatar of the PC 210, and the avatar of the PC 220, and displays them on the LCD 63.
In the image displayed on the LCD 63 of the HMD 1, the received video and the avatar (including avatar additional information) are displayed on different layers.

Specifically, the avatar selection unit 306 of the processing unit 123 reads the avatar image data of HMD1, the avatar image data of HMD500, the avatar image data of PC210, and the avatar image data of PC220 from the avatar storage unit 307. Next, the avatar selection unit 306 generates image information being called for each terminal of the HMD 500, the PC 210, and the PC 220 based on the reception signal output from the transmission / reception unit 303. Next, the avatar selection unit 306 associates each read avatar image data with the generated image information being called and outputs the image information to the image composition unit 308. Next, the image composition unit 308 displays the image data output from the avatar selection unit 306 on the LCD 63 via the LCD driver 125. That is, the HMD 1 avatar, the HMD 500 avatar and calling image, the PC 210 avatar and calling image, and the PC 220 avatar and calling image are displayed on the LCD 63 of the HMD 1.
The displayed avatar is set in advance by the user of the HMD 1 or set in advance by default.
Also in the following process, each part of the process part 123 performs display of an avatar and avatar additional information.

  Next, the processing unit 523 of the HMD 500 extracts call information from the received reception signal. Next, the processing unit 523 reads the avatar image data of the HMD 1 from the storage unit of the processing unit 523 based on the extracted call information. Next, the processing unit 221 generates image information being called based on the extracted call information. Next, the processing unit 523 displays an image based on the read avatar image data of the HMD 1 and the generated image information being called on the LCD 563.

Next, the processing unit 211 of the PC 210 extracts call information from the received reception signal. Next, the processing unit 211 reads the avatar image data of the HMD 1 from the storage unit 212 based on the extracted call information. Next, the processing unit 211 generates image information indicating that a call is in progress based on the extracted call information. Next, the processing unit 211 displays on the display unit 215 an image based on the read avatar image data of the HMD 1 and the generated image information indicating that the call is in progress.
Next, the processing unit 221 of the PC 220 performs processing similar to that of the PC 210 and displays on the display unit 225 an image based on the avatar image data of the HMD 1 read from the storage unit 222 and the generated image information indicating that the call is in progress. To do.

  Next, the processing unit 201 of the server 200 extracts call information from the received reception signal. Next, based on the extracted call information, the processing unit 201 records call information indicating that there is a call from the HMD 1 in the storage unit 202 in association with, for example, time information.

(Step S <b> 3) The processing unit 123 of the HMD 1 controls to display the video (moving image) captured by the camera 64 on the LCD 63.

(Step S <b> 4) The user of the PC 210 confirms the image indicating that the calling is displayed on the display unit 215 and operates the input unit 214 to participate in the call. Based on the information input from the input unit 214, the processing unit 211 of the PC 210 transmits information indicating participation in the call (hereinafter referred to as call participation information) to the HMD1, the HMD 500, the PC 220, and the server 200. Next, the processing unit 211 displays the avatar of the HMD 1 and the avatar of the PC 210 on the display unit 215.

(Step S <b> 5) Based on the call participation information received from the PC 210, the processing unit 123 of the HMD 1 deletes from the LCD 63 image information indicating that the PC 210 is being called.
Next, the processing unit 201 of the server 200 records information indicating that the PC 210 is participating in a call in the storage unit 202 based on the call participation information received from the PC 210.

(Step S <b> 6) The user of the HMD 500 confirms the image indicating that the calling is displayed on the LCD 563 and operates the touch switch 534 to participate in the call. In this state, the avatar of HMD1 and the avatar of HMD500 are displayed on the LCD 563. Next, the processing unit 523 of the HMD 500 transmits call participation information to the HMD 1, the PC 210, and the server 200 based on the control signal output from the touch switch 534.

(Step S7) The processing unit 523 of the HMD 500 adds and displays the avatar of the PC 210 on the LCD 563.
Next, based on the call participation information received from the HMD 500, the processing unit 123 of the HMD 1 erases from the LCD 63 image information indicating that the HMD 500 is being called.
Next, the processing unit 211 of the PC 210 displays the avatar of the HMD 500 on the display unit 215 based on the call participation information received from the HMD 500.
Next, the processing unit 201 of the server 200 records information indicating that the HMD 500 is participating in a call in the storage unit 202 based on the call participation information received from the HMD 500.

Here, when two or more avatars are displayed on the LCD (63, 563) and the display unit (215, 225) of each terminal (HMD1, HMD500, PC210, and PC220), the processing unit (123, 523, 211, and 221) calculate the longitude of the avatar display from the environment information. And you may make it the processing part of each terminal display horizontally in order from the right side from the avatar of the terminal located most east.
FIG. 20 is a diagram illustrating a display example of an avatar according to the present embodiment. The example shown in FIG. 20 is an example in which three parties are participating in a call. As shown in FIG. 20, for example, an image 400 is displayed on the LCD 63 of the HMD 1. The image 400 is an example in which an avatar 401 of the PC 220, an avatar 402 of the PC 210, and an avatar 403 of the HMD 500 are displayed below the image 400 in order from the left. In the example of the image 400 shown in FIG. 20, since the position information of the HMD 500 is the east, the avatar 403 of the HMD 500 is displayed on the rightmost side.

  In addition, when three parties have a conversation, each processing unit of each terminal may perform control so that the voice of the speaker corresponding to the right avatar can be heard from the right. That is, the right ear speaker can hear a louder sound than the left speaker. Further, each processing unit of each terminal may reproduce a louder sound from the left ear speaker so that the left avatar's voice can be heard from the left side. This makes it possible to grasp the position of the other party who is talking both visually and audibly, thereby providing an environment that facilitates communication.

In this embodiment, an example in which avatars are displayed side by side has been described, but avatars may be displayed side by side. When displaying an avatar up and down, each processing unit of each terminal may display vertically in order from the top of the avatar of the terminal located in the north according to the latitude.
In this case, each processing unit of each terminal may be controlled so that the voice from the avatar arranged above can be heard from the right side and the voice from the lower avatar can be heard from the left side. Alternatively, each processing unit of each terminal has an acoustic characteristic so that the voice of the speaker corresponding to the avatar arranged above can be heard from above, and the voice of the speaker corresponding to the avatar arranged below can be heard from below. May be changed.

If each terminal does not have a means for acquiring position information, each processing unit of each terminal may substitute position information corresponding to the IP address of the counterpart terminal or the telephone station number. When the position information corresponding to the IP address of the counterpart terminal or the telephone station number cannot be substituted, each processing unit of each terminal may set the position of each avatar in the order of time of participation in the call.
Furthermore, each processing unit of each terminal may control the mouth of the corresponding avatar to move when the user of each terminal speaks.
As a result, the speaker can be recognized visually, so that the user of each terminal can easily understand the information and does not recognize a speaker having a similar voice quality.

Next, a procedure for transmitting video captured by the HMD 1 to another terminal will be described.
(Step S <b> 8) Returning to FIG. 8, the user of the HMD 1 touches a predetermined area of the touch switch 34 in order to transmit the video captured by the camera 64 to another terminal. The transmitted video is preset to a low resolution in the initial state. The low resolution is, for example, 320 pixels horizontally, 240 pixels vertically, and 30 frames per second. The reason for transmitting low-resolution video is to some extent, even in a network environment where the communication infrastructure is limited and the transfer speed is slow, even if there is blurring of the video due to the movement of the neck of the HMD1 user. This is because an image to be transmitted is transmitted.

(Step S <b> 9) Moving to FIG. 10, the image switching unit 302 of the processing unit 123 of the HMD 1 converts the image captured by the camera 64 to a low resolution based on the control signal input from the touch switch 34. The processing unit 123 transmits the converted low-resolution video signal and HMD1 environment information to the HMD 500, the PC 210, and the server 200 participating in the call.

(Step S10) Each processing unit (523 and 211) of each terminal (HMD500 and PC210) calculates a direction (azimuth) in which the camera 64 of the HMD1 is facing based on the environment information received from the HMD1. Each processing unit displays the calculated azimuth and the video signal received from the HMD 1 on the LCD 563 and the display unit 215.
The processing unit 201 of the server 200 calculates a direction (azimuth) in which the camera 64 of the HMD 1 is facing based on the environment information received from the HMD 1. The processing unit 201 records the calculated orientation and the video data received from the HMD 1 in the storage unit 202.

(Step S11) The processing unit 123 of the terminal HMD1 starts a three-party call with the PC 210 and the HMD 500 based on the call participation information in steps S4 and S6.
Next, the processing unit 523 of the HMD 500 starts a three-party call with the HMD 1 and the PC 210 based on Steps S1 and S4.
Next, the processing unit 211 of the PC 210 starts a three-party call with the HMD 1 and the HMD 500 based on Steps S1 and S6.
Next, the processing unit 201 of the server 200 starts recording the contents (voice data, video data) of the three-party call based on steps S1, S4, and S6.

(Step S <b> 12) The user of the HMD 1 performs a predetermined finger operation on the touch switch 34 in order to indicate a specific part of the image captured by the camera 64 to the user participating in the call. Instead of operating the touch switch 34, the user of the HMD 1 may blink twice, for example. This is detected by the image sensor 72 of the HMD 1 and processed as being equivalent to an operation on the touch switch 34.

(Step S13) The image switching unit 302 of the processing unit 123 of the HMD 1 converts the video signal captured by the camera 64 into a medium resolution based on the control signal input from the touch switch 34.
Next, the processing unit 123 transmits the converted medium resolution video data and environment information to the HMD 500, the PC 210, and the server 200 participating in the call. The medium-resolution video is, for example, 800 pixels in the horizontal direction and 600 pixels in the vertical direction. When transmitting a medium-resolution video, the user of the HMD 1 aims to transmit the video to the user of another terminal, and therefore, the movement of the neck of the user wearing the HMD is small. There is little blur of the camera 64. For this reason, the processing unit 123 switches to the medium resolution video for the purpose of increasing the resolution from the low resolution set in step S9 to the medium resolution and transmitting a video that is easy for other users to view.

If the communication infrastructure environment is poor and it is difficult to secure a communication speed sufficient to transmit at a resolution of 800 pixels in the horizontal direction, 600 pixels in the vertical direction, and 30 frames / second, the processing unit 123 determines that there is little camera shake. In this situation, the frame rate may be decreased, for example, 8 frames / second.
When the angular velocity sensor 134 of the HMD 1 determines that the user's neck rotation is large, the processing unit 123 holds the frame rate at 30 frames / second in order to follow the flow of the image due to the neck rotation. The resolution may be transmitted with 320 pixels in the horizontal direction and 240 pixels in the vertical direction.

(Step S14) Based on the control signal input from the touch switch 34, the processing unit 123 of the HMD 1 displays an image of a light blue cursor 410 as shown in FIG. To do. The images displayed on the LCD (63, 563) and the display unit 215 of each terminal (HMD1, HMD500, and PC210) are the received video and the cursor image displayed on different layers, and To be recorded.

Specifically, the cursor generation unit 305 of the processing unit 123 generates cursor image data and cursor position information based on the control signal output from the input detection unit 301. The cursor generation unit 305 outputs the generated cursor image data and cursor position information to the image composition unit 308. Based on the input cursor image data and the cursor position information, the image composition unit 308 converts the cursor into an image captured by the camera 64 or an image received from another terminal via the communication interface (131, 138). The superimposed images are displayed on the LCD 63.
Also in the following processing, each part of the processing unit 123 displays a cursor image.

  FIG. 21 is a diagram illustrating an example of a configuration of an image displayed on the LCD or the display unit according to the present embodiment. In the example shown in FIG. 21, an image (video or still image) being received or transmitted is displayed on layer 1 (800A). An avatar is displayed on layer 2 (800B). A cursor image by the HMD 1 is displayed on the layer 3 (800C), a line drawing by the HMD 500 is displayed on the layer 4 (800D), and a line drawing by the PC 210 is displayed on the layer 5 (800E). In addition, the images having the layer configuration shown in FIG. 21 are also recorded in the flash memories (122, 522) and the storage units (212, 221 and 202) of each terminal (including server). Note that the example of the layer illustrated in FIG. 21 is an example, and the order of the layers may not be in this order. For example, the layers may be displayed in the order in which they are operated.

FIG. 22 is an example of an image on which a cursor according to the present embodiment is displayed. As shown in FIG. 22, for example, the LCD (63, 563) and the display unit (215, 225) of each terminal (HMD1, HMD500, PC210, and PC220) have a shape indicated by a finger in the image 400a. A cursor 410 is displayed. Note that the shape of the cursor is an example, and other shapes may be used. In the example illustrated in FIG. 22, the user of the HMD 1 is in a state of pointing the price tag 411 with the cursor 410 in the image 400 a.
In addition, since the image transmitted from HMD1 is a moving image, if the user of HMD1 moves the neck, the image | video currently transmitted will also shift. For this reason, the user of the HMD 1 may change the position of the cursor in order to change the position indicated by the cursor. For this reason, a cursor having no trace is displayed on the LCD and the display unit of each device.
Note that the color of the cursor may be determined in advance for each terminal. In this embodiment, the HMD1 cursor is light blue, the PC210 cursor is green, and the HMD500 cursor is blue. In this way, the cursor is color-coded according to each terminal, so that mistakes in communication are less likely to occur.

Next, the processing unit 123 transmits position information for displaying the cursor and cursor color information to the HMD 500, the PC 210, and the server 200.
Next, the processing unit 523 of the HMD 500 displays a light blue cursor image on the LCD 563 based on the received position information for displaying the cursor and the cursor color information. Note that the processing unit 523 may determine the display color of the cursor for each terminal in advance. In this case, the processing unit 523 may acquire only the position information for displaying the cursor and display a light blue cursor because the transmission source is the HMD1.
Next, the processing unit 211 of the PC 210 displays an image of a light blue cursor on the display unit 225 based on the received position information for displaying the cursor and cursor color information.
Next, the processing unit 201 of the server 200 records the position information of the cursor of the HMD 1 in the storage unit 202 based on the received position information for displaying the cursor.

Next, a procedure in which the PC 210 pauses the received video and draws a line drawing on the paused image will be described.
(Step S15) Moving to FIG. 11, the PC 210 user operates the mouse of the input unit 214 so that the cursor image of the PC 210 is displayed on the display unit 215. Next, the user of the PC 210 clicks the mouse to view a still image.

(Step S16) The processing unit 211 of the PC 210 temporarily stops the received video based on the information input from the input unit 214. Next, the processing unit 211 records the paused still image in the storage unit 212 and transmits the still image to the HMD 1, the HMD 500, and the server 200.
The reason for temporarily stopping the video is that it is assumed that the video distributed from the HMD 1 is displayed with a delay on a specific terminal depending on the communication environment. For this reason, in order to superimpose still images, the same still image needs to be displayed on all terminals. By transmitting the paused image, it is possible to observe the same still image on all terminals. In addition, even if each terminal superimposes a line image on a still image, the relationship between the line image and the still image is the same for all terminals, so it is possible to reduce communication errors.
In an environment where it is known that the communication speed is sufficiently high, each terminal may temporarily display the video as a still image in response to the operation of the PC 210 in step S15.
In addition, the pause operation is illustrated as being performed by the PC 210, but it can be performed from the HMD1 that is the transmission source of the video, or from the HMD 500 that has the same components as the HMD1 and is receiving. . For example, when a still image is distributed from the HMD 1, a frame of video displayed on the LCD 63 of the HMD 1 is distributed.

(Step S <b> 17) The processing unit 123 of the HMD 1 receives the still image data transmitted from the PC 210 and records the received still image data in the flash memory 122 of the HMD 1.
Next, the processing unit 523 of the HMD 500 receives the still image data transmitted from the PC 210 and records the received still image data in the flash memory 522 of the HMD 500.
Next, the processing unit 201 of the server 200 receives the still image data transmitted from the PC 210 and records the received still image data in the storage unit 202.

(Step S18) In order to display a line drawing on a still image, for example, the user of the PC 210 draws a line drawing by moving the cursor within the display unit 225 while clicking the mouse. At this time, for example, a cursor shaped like a pencil may be displayed on the display unit 225.
Next, the processing unit 211 of the PC 210 displays a line drawing with a green locus on the display unit 225 based on the information input from the input unit 214. Next, the processing unit 211 sequentially records cursor position information in the storage unit 212. Next, the processing unit 211 sequentially transmits the cursor position information to the HMD 1, the HMD 500, and the server 200.
Note that the color of the line drawing by the PC 210 may be determined in advance for each terminal, or color information may be transmitted from the PC 210 to another terminal.

Specifically, the line drawing generation unit 304 of the processing unit 123 generates line drawing data based on the control signal output from the input detection unit 301. The line drawing generation unit 304 outputs the generated line drawing data to the image composition unit 308. The image composition unit 308 superimposes the input line drawing data on the image data captured by the camera 64 or the image data received from another terminal via the communication interface (131, 138), and superimposes the superimposed image data. The based image is displayed on the LCD 63.
Also in the following processing, each unit of the processing unit 123 displays a line drawing.

(Step S19) The processing unit 123 of the HMD 1 displays a green line drawing on the LCD 63 based on the received cursor position information. Next, the processing unit 123 generates line drawing data based on the cursor position information, and records the generated line drawing data in the flash memory 122.
Next, the processing unit 523 of the HMD 500 displays a green line drawing on the LCD 563 based on the received position information of the cursor. Next, the processing unit 523 generates line drawing data based on the cursor position information, and records the generated line drawing data in the flash memory 522.
Next, the processing unit 201 of the server 200 generates line drawing data based on the received cursor position information, and records the generated line drawing data in the storage unit 202.

FIG. 23 is an example of a line drawing according to the present embodiment. In the example shown in FIG. 23, an arrow 421a and the characters “OK!” 421b are displayed on the image 400b. The arrow 421a and the character 421b are line drawings drawn by the user of the PC 210 using the mouse. 23, for example, in FIG. 22, the user of HMD1 points the price tag 411 to the user of another terminal with the cursor 410, but the user of the PC 210 wants to talk about the destination indicated by the arrow 421a. The price tag 422 is shown.
As described above, when a line image is drawn by being superimposed on a camera image, the camera image is preferably a still image. That is, if the user of the HMD 1 moves his / her head against the intention of the user of the PC 210, the image moves and the position of the subject pointed to by the line drawing is relatively shifted on the screen. However, if a still image is displayed, the image and the line drawing do not relatively shift even if the line drawing is superimposed on the image. For this reason, it is not necessary to be stationary so that the user of the HMD 1 does not move the neck, and an easy-to-use system can be provided.
Note that the line image and the still image are recorded in different layers, and it is possible to delete only the line image at any time.
Further, not only the PC 210 but all terminals are capable of moving a still image at the same time as drawing a line drawing.

Next, a procedure for changing the grid superimposed on the image will be described.
(Step S20) Returning to FIG. 11, the user of the PC 210 uses the input unit 214 to select the type of grid. Based on the signal input from the input unit 214, the processing unit 211 of the PC 210 transmits information indicating the type of the selected grid to the HMD1, the HMD5, and the server 200. The reason for displaying the grid is, for example, that the user of the PC 210 explains a specific area in the image.
FIG. 24 is a diagram illustrating an example of a grid according to the present embodiment. FIG. 25 is a diagram illustrating another example of the grid according to the present embodiment.
The example illustrated in FIG. 24 is an example in which a grid-like grid 431 including vertical lines and horizontal lines is displayed in the image 400c. The example illustrated in FIG. 25 is an example in which a grid 441 including vertical lines, horizontal lines, and concentric circles is displayed in the image 400d. 24 and 25, the grid is indicated by a solid line, but the grid may be a dotted line, a broken line, or the like.
As described above, when the user of the PC 210 changes the grid, the reason for transmitting the changed grid information to another terminal is to prevent a communication error if the grid is different for each user terminal.
Note that the grids shown in FIGS. 24 and 25 are only examples, and other grids may be displayed. In addition, the grid displayed on the LCD or display unit of each terminal is displayed on a layer different from that of the moving image or the still image.

(Step S21) The processing unit 211 of the PC 210 changes the display to the selected grid.
Next, the processing unit 123 of the HMD 1 changes the display to the selected grid based on the received signal.
Next, the processing unit 523 of the HMD 5 changes the display to the selected grid based on the received signal.
Next, the processing unit 201 of the server 200 records the grid change based on the received signal.

Next, a processing procedure when the PC 220 participates in a call from the middle will be described.
(Step S <b> 22) Moving to FIG. 12, the user of the PC 220 confirms the image indicating that the calling is displayed on the display unit 225 and operates the input unit 224 to participate in the call. The processing unit 221 of the PC 220 transmits call participation information to the server 200 based on the information input from the input unit 224.
Next, the processing unit 221 controls the display unit 225 to display the PC 220 avatar.

(Step S23) Based on the call participation information received from the PC 220, the processing unit 201 of the server 200 performs a call participation process of the PC 220 on the HMD1, the HMD 500, the PC 210, and the PC 220. Specifically, the processing unit 201 transmits the same image data as the images (including line drawings, cursors, and grids) displayed on the other LCD and the display unit to the PC 220. The processing unit 221 transmits information indicating that the PC 220 has joined the call and an instruction to display the avatar of the PC 220 to the HMD 1, the HMD 500, and the PC 210. The processing unit 201 transmits an instruction to display the avatars of the HMD 1, the HMD 500, and the PC 210 to the PC 220.
Next, the processing unit 201 records information indicating that the PC 220 has joined the call in the storage unit 202.

(Step S <b> 24) The processing unit 221 of the PC 220 displays an image based on the image data received from the server 200 on the display unit 225.

(Step S <b> 25) The processing unit 123 of the HMD 1 additionally displays the avatar of the PC 220 on the LCD 63 based on the instruction to display the avatar of the PC 220 received from the server 200, and an image indicating that the PC 220 is being called Erase information.
Next, the processing unit 523 of the HMD 500 adds and displays the PC 220 avatar on the LCD 563 based on the instruction to display the PC 220 avatar received from the server 200.
Next, the processing unit 211 of the PC 210 adds and displays the avatar of the PC 220 on the display unit 215 based on the instruction to display the avatar of the PC 220 received from the server 200.
Next, the processing unit 221 of the PC 220 adds and displays the avatars of HMD1, HMD500, and PC 210 on the display unit 225 based on the instruction to display the avatars of HMD1, HMD500, and PC 210 received from the server 200. .
The display of the avatar displayed on the LCD and the display unit of each terminal here is again from the right from the terminal located east according to the east longitude from the right according to the location of each terminal with the participation of the PC 220. It may be displayed again.

Next, a procedure in which different terminals display a cursor and a line drawing on a common image will be described.
(Step S26) The user of the HMD 500 performs a predetermined finger operation on the touch switch 534 to draw a line drawing. The processing unit 523 of the HMD 500 transmits an instruction to display a line drawing by the HMD 500 to the HMD 1, the PC 210, the PC 220, and the server 200 based on the control signal input from the touch switch 534.

(Step S27) The processing units (123, 211, and 221) of each terminal (HMD1, PC210, and PC220) are red on the LCD 63 and the display units (215, 225) based on the received signal received from the HMD500. Display the line drawing. The processing unit 523 of the HMD 500 displays a red line drawing on the LCD 563 based on the control signal input from the touch switch 534. That is, it is displayed in a color different from the line drawing (green) of the PC 210.
Next, the processing unit of each terminal records the position information of the cursor in the flash memory 122 and the storage unit (212, 222, 202) sequentially based on the received signal received from the HMD 500. Based on the control signal input from the touch switch 534, the processing unit 523 of the HDM 500 sequentially records the cursor position information in the flash memory 522 and the storage units (212, 222, 202).
Alternatively, each processing unit (123, 523, 211, 221 and 201) records the image data of the line drawing drawn by the user of the HMD 500 in the flash memory (122, 522) and the storage unit (212, 222, 202). .
In addition, on the LCD (63, 563) and the display unit (215, 225) of each terminal, the user of the HMD 500 depicted the image temporarily stopped by the PC 210 by moving the finger with the touch switch 534 in step S16. The line drawing is displayed superimposed.
The line drawing by the HMD 500 is stored in each terminal and server as layer information different from the transmitted image and the line drawing drawn by other terminals.

(Step S28) The user of the HMD 1 performs a predetermined finger operation on the touch switch 34 in order to draw a line drawing. The processing unit 123 of the HMD 1 transmits an instruction to display a line drawing by the HMD 1 to the HMD 500, the PC 210, the PC 220, and the server 200.

(Step S29) The processing units (523, 211, and 221) of each terminal display a blue line drawing on the LCD 563 and the display units (215, 225) based on the reception signal received from the HMD1. The processing unit 123 of the HMD 1 displays a blue line drawing on the LCD 63 based on the control signal input from the touch switch 34.
Next, the processing units (523, 211, 221 and 201) of each terminal sequentially store the cursor position information on the flash memory 522 and the storage units (212, 222, 202) based on the received signals received from the HMD1. ). The processing unit 123 of the HMD 1 sequentially records cursor position information in the flash memory 122 based on the control signal input from the touch switch 34. Alternatively, the processing units (123, 523, 211, 221 and 201) record the line drawing image data drawn by the user of the HMD 1 in the flash memory (122, 522) and the storage unit (212, 222, 202).
Note that the LCD (63, 563) and the display unit (215, 225) of each terminal have a green line drawing drawn by the PC 210 in Step S18 on the image paused by the PC 210 in Step S16, and in Step S26. The red line drawing drawn by the HMD 500 is superimposed and displayed.
The reason why the line drawing by HMD1 is blue (R: 0, G: 0, B: 255) is that the cursor is displayed in light blue (R: 100, G: 100, B: 255). This is to make it easier to visually understand which user is investigating.

Next, a description will be given of a processing procedure in a case where a video of another terminal is requested according to a request of a user of another terminal, and the video is captured as a still image and image information is shared.
(Step S30) Moving to FIG. 13, for example, a user of the HMD 500 requests transmission of an image captured by the camera 64 of the HMD 1 via a call microphone of the HMD 500.
Next, in response to a request from the user of the HMD 500, the user of the HMD 1 touches a predetermined area of the touch switch 34 in order to transmit the video captured by the camera 64 to another terminal at a low resolution. In the initial setting, the resolution at the time of video transmission is set to a low resolution. However, the resolution can be changed by operating the touch switch 34.

(Step S <b> 31) The processing unit 123 of the HMD 1 deletes the image that has been displayed up to Step S <b> 29 and displays an image captured by the camera 64 on the LCD 63.

(Step S32) Based on the control signal output from the touch switch 34, the processing unit 123 of the HMD 1 converts the video captured by the camera 64 into a low resolution. The processing unit 123 transmits the converted low-resolution video data and the environment information of the HMD 1 to the HMD 500, the PC 210, the PC 220, and the server 200 participating in the call.

(Step S33) The processing units (523, 211, and 221) of each terminal (HMD500, PC210, and PC220) are displayed on the LCD 563 and the display units (215, 225) based on the received signal received from the HMD1. Erase the displayed image and line drawing.

(Step S34) The processing units (523, 211, and 221) of each terminal (HMD500, PC210, and PC220) receive from the HMD1 on the LCD 563 and the display units (215, 225) based on the reception signal received from the HMD1. The video based on the low-resolution video data is displayed.
Next, the processing unit 201 of the server 200 records low-resolution video data received from the HMD 1 in the storage unit 202 based on the reception signal received from the HMD 1.

(Step S35) The user of the HMD 500 touches a predetermined area of the touch switch 534 to make a request for transmitting a high-resolution still image to the HMD 1 to view a detailed image. Next, the processing unit 523 of the HMD 500 transmits a request for transmitting a high-resolution still image to the HMD 1 based on the control signal input from the touch switch 534.

(Step S <b> 36) The processing unit 123 of the HMD 1 displays information indicating a request for transmitting a high-resolution still image received from the HMD 500 on the LCD 63. Note that a high-resolution still image is, for example, an image with 2500 pixels in the horizontal direction and 1875 pixels in the vertical direction.

(Step S37) Based on the received signal from the HMD 500, for example, when the front of the HMD 1 is dark, the processing unit 123 of the HMD 1 turns on the front light 68 for illumination.

(Step S38) Moving to FIG. 15, the processing unit 123 of the HMD 1 combines high-resolution still image data automatically captured by the camera 64 based on the received signal from the HMD 500 with the environment information, the HMD 500, the PC 210, The data is transmitted to the PC 220 and the server 200.

(Step S39) The processing units (523, 211, and 221) of each terminal (HMD500, PC210, and PC220) receive from the HMD1 on the LCD 563 and the display units (215, 225) based on the received signal received from the HMD1. The high-resolution still image and information based on the environment information are displayed.
Next, the processing unit 201 of the server 200 records the high-resolution still image received from the HMD 1 in the storage unit 202 based on the reception signal received from the HMD 1.
In this way, high-resolution video that is difficult to distribute as a moving image is photographed with high definition in response to an operation from another terminal, and is distributed to each terminal as a still image. This makes it possible to analyze the environment in which the HMD 1 is used more easily than others.

(Step S40) The user of the PC 210 operates the input unit 214 to enlarge and display the still image received from the HMD 1 on the display unit 215. Based on the control signal input from the input unit 214, the processing unit 211 of the PC 210 transmits an instruction to enlarge the display image to the HMD1, the HMD 500, the PC 220, and the server 200.

(Step S41) The processing units (123, 523, and 221) of each terminal (HMD1, HMD500, and PC220), based on the received signal received from the PC210, the images on the LCD (63,563) and the display unit 225 Is magnified.
Next, the processing unit 201 of the server 200 records the enlarged image data and information indicating that the image has been enlarged in the storage unit 202 based on the received signal received from the PC 210.
As described above, since high-resolution still image data is sent from the HMD 1, an image that does not deteriorate in image quality even when enlarged is displayed on each terminal. The high-resolution still image can be automatically enlarged by simply operating a single terminal, and detailed communication can be performed on this image.

Next, processing for transmitting a medium-resolution video will be described.
(Step S <b> 42) The user of the HMD 500 touches a predetermined area of the touch switch 534 in order to request the HMD 1 to transmit a medium resolution video. Next, the processing unit 523 of the HMD 500 transmits a request for transmitting a medium resolution video to the HMD 1 based on the control signal input from the touch switch 534.

In this case, the LCD 63 of the HMD 1 displays that a medium resolution image is desired. Alternatively, the request may be transmitted by voice call. When responding to this request, the user of the HMD 1 needs to prevent the image from blurring. That is, it can be seen that the user of the HMD 1 is required to stably point the camera at the subject so as not to move the neck much.
In this way, the user of each terminal can know the level of video request from each terminal (still image or moving image). For this reason, since the user of each terminal can know how to point the camera 64 toward the subject, the system is easy to use.

As described above, the medium-resolution video can be transmitted only in a state where the user of the HMD 1 can point the camera 64 toward the subject relatively stably. For this reason, when it is determined that transmission of a medium-resolution video is not preferable, the user of the HMD 1 may refuse this by voice or the like. In this way, by obtaining the consent of the HMD1 user, it is possible to prevent the image of the environment where the HMD1 user is present from being transmitted as it is. That is, the privacy of the user of HMD1 can be protected by obtaining the consent of the user of HMD1.
Even if the HMD1 user approves the transmission of the medium-resolution video, the camera 64 may be severely blurred while the user is actually moving. You may make it set the resolution of the image | video transmitted by the angular velocity sensor 134. FIG. For example, when the output of the angular velocity sensor 134 is larger than a predetermined value, the processing unit 123 of the HMD 1 may perform transmission at a low resolution and a high frame rate even when a request for medium resolution is received.

(Step S <b> 43) The processing unit 123 of the HMD 1 displays information indicating a request for transmitting the medium resolution video received from the HMD 500 on the LCD 63. Next, the user of the HMD 1 confirms the display on the LCD 63 and touches a predetermined area of the touch switch 34 in order to accept the request from the HMD 500.

(Step S44) The processing unit 123 of the HMD 1 transmits the medium resolution video data captured by the camera 64 to the HMD 500, the PC 210, the PC 220, and the server 200 together with the environment information.

Next, a procedure in which one terminal adds information to an image from another terminal and distributes the image to another terminal in a completed state will be described.
(Step S45) The processing units (523, 211, and 221) of each terminal (HMD500, PC210, and PC220) receive the LCD 563 and the display units (215, 225) from the HMD1 based on the received signal received from the HMD1. Displays the medium resolution video.
Next, the processing unit 201 of the server 200 records the medium resolution video data received from the HMD 1 in the storage unit 202 based on the reception signal received from the HMD 1.

(Step S46) Moving to FIG. 15, the user of the HMD 500 touches a predetermined area of the touch switch 34 in order to record a still image of the video received from the HMD 1.

(Step S <b> 47) The processing unit 523 of the HMD 500 records the image data of the still image received from the HMD 1 in the flash memory 522 based on the control signal input from the touch switch 534.

(Step S48) The user of the HMD 500 touches a predetermined area of the touch switch 534 in order to superimpose and display a line image on a still image. Next, the user of the HMD 500 moves a finger on the touch switch 534 to draw a line drawing. The processing unit 523 superimposes the drawn line image on the still image and displays it on the LCD 563. The processing unit 523 records image data in which a line image is superimposed on a still image in the flash memory 522.

(Step S49) The user of the HMD 500 touches a predetermined area of the touch switch 534 in order to transmit the image data in which the line image is superimposed on the still image recorded in the flash memory 522 to the HMD 1, the PC 210, the PC 220, and the server 200. To do.

(Step S50) Based on the control signal input from the touch switch 534, the processing unit 523 of the HMD 500 converts the image data obtained by superimposing the line image on the still image recorded in the flash memory 522 to the HMD1, the PC 210, the PC 220, and the server. 200.

(Step S51) Based on the received signal received from the received HMD 500, the processing units (123, 211, and 221) of each terminal (HMD1, PC 210, and PC 220) send the HMD 500 to the LCD 63 and the display units (215, 225). An image based on the image data in which the line drawing is superimposed on the still image received from is displayed.
Next, the processing unit 201 of the server 200 records, in the storage unit 202, image data in which a line drawing is superimposed on a still image received from the HMD 500 based on the received signal received from the received HMD 500.

Next, processing for switching a video from another terminal, and processing for transmitting a low-resolution video by default in that case will be described.
(Step S <b> 52) The user of the HMD 1 touches a predetermined area of the touch switch 34 in order to view the video captured by the camera 564 of the HMD 500. Next, the processing unit 123 of the HMD 1 transmits a video browsing request to the HMD 500.

(Step S <b> 53) The processing unit 523 of the HMD 500 displays information indicating a request to transmit the video received from the HMD 1 on the LCD 563. Next, the user of the HMD 500 confirms the display on the LCD 563 and touches a predetermined area of the touch switch 534 in order to accept the request from the HMD 1.

(Step S54) Moving to FIG. 16, the processing unit 523 of the HMD 500 converts the video captured by the camera 564 into a low resolution. The processing unit 523 transmits the converted low-resolution video signal and the environment information of the HMD 500 to the HMD 1, the PC 210, the PC 220, and the server 200. That is, when there is a transmission request for an image captured by a camera from another terminal, the processing unit of each terminal first transmits a low-resolution video. Thereafter, the resolution is switched according to the request of the user of each terminal.

(Step S55) Based on the received signals from the received HMD 500, the processing units (123, 211, and 221) of the terminals (HMD1, PC 210, and PC 220) transfer the HMD 500 to the LCD 63 and the display units (215, 225). The video based on the low resolution video data received from is displayed. Next, the processing units (123, 211, and 221) of each terminal (HMD1, PC210, and PC220) calculate the imaged direction based on the environment information received from the HMD500, based on the received signal from the HMD500. The calculated imaging direction is displayed on the LCD 63 and the display unit (215, 225).
Next, the processing unit 201 of the server 200 records low-resolution video data and environment information in the still image received from the HMD 500 in the storage unit 202 based on the received signal from the HMD 500.

Next, a procedure for terminating a call by a specific terminal will be described.
(Step S56) The user of the PC 210 operates the input unit 214 to end the connection. Next, based on the control signal input from the input unit 214, the processing unit 211 of the PC 210 transmits information indicating the end of connection to the HMD1, the HMD 500, the PC 220, and the server 200.

(Step S57) The processing units (123, 523, and 221) of each terminal (HMD1, HMD500, and PC220) are displayed on the LCD (63, 563) and the display unit 225 based on the received signal received from the PC210. Delete the avatar of the PC 210 that was being used.
Next, the processing unit 201 of the server 200 records the connection end of the PC 210 in the storage unit 202 based on the received signal received from the PC 210.

(Step S58) The user of the HMD 500 operates the touch switch 534 to end the connection. Next, the processing unit 523 of the HMD 500 transmits information indicating the end of connection to the HMD 1, the HMD 500, the PC 220, and the server 200 based on the control signal input from the input unit 214.

(Step S59) The processing units (123 and 523) of each terminal (HMD1 and HMD500) erase the avatar of the HMD 500 displayed on the LCD 63 and the display unit 225 based on the received signal received from the HMD 500.
Next, the processing unit 201 of the server 200 records the connection end of the HMD 500 in the storage unit 202 based on the received signal received from the HMD 500.

Next, a description will be given of a procedure of processing in which the user of each terminal calls and uses an image (video, still image) recorded on the server 200.
(Step S60) In order to select an image recorded in the server 200, the user of the PC 220 operates the input unit 224, for example, presses a predetermined key on the keyboard.

(Step S61) Referring to FIG. 17, it is assumed that a list of thumbnails, which are reduced images of images recorded in the server 200, is displayed on the display unit 225 of the PC 220. The user of the PC 220 operates the input unit 224, for example, presses a predetermined key on the keyboard in order to select an image recorded in the server 200. Note that the thumbnail list may be read from the server 200 and displayed by the user of the PC 220 by operating predetermined keys on the keyboard and the mouse.

(Step S <b> 62) The processing unit 221 of the PC 220 transmits information indicating the selected image to the server 200 based on the control signal input from the input unit 224.

(Step S <b> 63) Based on the received signal received from the PC 220, the processing unit 201 of the server 200 reads out the selected video and the environment information recorded in the storage unit 202 in association with the video. The processing unit 201 calculates an orientation based on the read environment information. The processing unit 201 transmits the read video data and the calculated azimuth information to the HMD 1 and the PC 220.

(Step S64) The processing unit 201 of the server 200 records in the storage unit 202 that the video data has been transmitted to the HMD 1 and the PC 220.
Next, the processing units (123 and 221) of the HMD 1 and the PC 220 display images based on the video data and the orientation information received from the server 200 on the LCD 63 and the display unit 225, respectively.

(Step S <b> 65) The user of the HMD 1 touches a predetermined area of the touch switch 34 (first operation) in order to draw a line drawing on the video transmitted from the server 200. Next, the processing unit 123 of the HMD 1 transmits a request for temporarily stopping the transmitted video to the server 200 based on the control signal input from the touch switch 34. That is, even when a line drawing is drawn on the video read from the server 200, the video is paused to obtain a still image. And each terminal can superimpose a line drawing on this still picture.

(Step S66) The processing unit 201 of the server 200 temporarily stops the video data being transmitted based on the received signal received from the HMD1. Next, the processing unit 201 transmits information indicating that the video data being transmitted has been temporarily stopped to the HMD 1 and the PC 220.

(Step S67) The processing units (123 and 221) of the HMD 1 and the PC 220 record the paused still images in the flash memory 122 and the storage unit 222, respectively, based on the received signal received from the server 200. . Note that the processing unit 123 of the HMD 1 may record the paused still image in the flash memory 122 based on the user touching the touch switch 34 in step S65.

(Step S <b> 68) Moving to FIG. 18, the user of the HMD 1 touches a predetermined area of the touch switch 34 in order to display a line image superimposed on a still image. Next, the user of the HMD 1 moves a finger on the touch switch 34 to draw a line drawing. Next, the processing unit 123 of the HMD 1 sequentially transmits the cursor position to the PC 200 and the server 200 in accordance with the movement of the finger of the touch switch 34.

(Step S69) Each processing unit (123 and 221) of the HMD 1 and the PC 220 superimposes the drawn blue line image on the still image and displays it on the LCD 63 and the display unit 225, respectively. Next, each processing unit (123 and 221) records image data in which a blue line image is superimposed on a still image in the flash memory 122 and the storage unit 222, respectively.

(Step S <b> 70) The user of the HMD 1 touches a predetermined area of the touch switch 34 in order to cancel the temporary stop and resume the transmission of the video. Next, the processing unit 123 of the HMD 1 cancels the temporary stop and transmits a request to resume transmission of video to the server 200.

(Step S <b> 71) The processing unit 201 of the server 200 cancels the pause based on the received signal received from the HMD 1 and resumes transmission of video data. Next, the processing unit 201 transmits to the HMD 1 and the PC 220 an instruction to delete the line drawing displayed in step S69.

(Step S <b> 72) Each processing unit (123 and 221) of the HMD 1 and the PC 220 deletes the blue line drawing displayed on the LCD 63 and the display unit 225 based on the received signal received from the server 200. Next, the respective processing units (123 and 221) of the HMD 1 and the PC 220 are based on the video data and the orientation information whose transmission is resumed to the LCD 63 and the display unit 225, respectively, based on the received signal received from the server 200. Display an image.

Next, a procedure for superimposing a line drawing on the video captured by the HMD 1 itself according to an instruction from the user of the HMD 1 will be described.
(Step S73) The user of the HMD 1 moves his / her finger at a predetermined position on the touch switch 34 in order to draw and transmit a line drawing. In this case, in order to transmit the line drawing, the processing unit 123 of the HMD 1 switches the video captured by the camera 64 from low resolution to medium resolution.

(Step S <b> 74) Moving to FIG. 19, the processing unit 123 of the HMD 1 converts the moving image captured by the camera 64 into a medium-resolution video, and transmits the converted medium-resolution video data to the PC 220 and the server 200.

(Step S75) The user of the HMD 1 touches a predetermined position of the touch switch 34 in order to pause the video being transmitted. Next, the user of the HMD 1 moves his / her finger on the touch switch 34 in order to draw a line drawing on the paused still image.
Next, based on the touch switch 34 being touched, the processing unit 123 of the HMD 1 transmits an instruction to pause the video being transmitted to the PC 220 and the server 200.
Next, the processing unit 123 sequentially transmits information indicating the position of the cursor to the PC 220 and the server 200 according to the movement of the finger on the touch switch 34.

(Step S76) The processing unit 123 of the HMD 1 superimposes and displays a blue line image on the still image according to the movement of the finger on the touch switch 34.
Next, the processing unit 221 of the PC 220 temporarily stops the video based on the received signal received from the HMD 1. Next, based on the received signal received from the HMD 1, the processing unit 221 superimposes a blue line drawing on the paused still image and displays it on the display unit 225.
Next, the processing unit 201 of the server 200 temporarily stops the video data based on the received signal received from the HMD 1 and records the still image when the video data is stopped in the storage unit 202. Next, the processing unit 201 records a blue line drawing in the storage unit 202 based on the reception signal received from the HMD 1.

Next, a process for ending calls with all call partners using the HMD 1 that started the call in step S1 will be described.
(Step S77) The user of the HMD 1 touches a predetermined position of the touch switch 34 in order to complete a call with all terminals. Next, based on the control signal input from the touch switch 34, the processing unit 123 of the HMD 1 transmits an instruction to complete the call to the PC 220 and the server 200 participating in the call in step S77. Next, the processing unit 123 ends the call with the PC 220 and the server 200.

(Step S78) The processing unit 221 of the PC 220 ends the call processing from the HMD 1 based on the received signal received from the HMD 1.
Next, the processing unit 201 of the server 200 records information indicating that the connection between the HMD 1 and the PC 220 is completed in the storage unit 202 based on the received signal received from the HMD 1.

  In the examples shown in FIGS. 9 to 19, the order of transmission of image data from each terminal, the drawing of line drawings, and the like is an example, and is performed in the order according to the instruction of the user of each terminal.

As described above, when you want to perform screen-based communication for video between multiple terminals, each processing unit of each terminal pauses the video to create a still image, and then superimposes a line drawing or the like on that image. To do. As a result, according to the present embodiment, it is possible to accurately superimpose the screen image and the line drawing even if the photographer changes the shooting direction.
In addition, according to the present embodiment, since the image captured by the terminal that first requested the still image is distributed to all the terminals, the line image and the still image are shifted from each other even if the line image is superimposed thereafter. Since there is nothing to do, communication errors can be reduced.

  Further, in this embodiment, each processing unit of each terminal pauses the video to make a still image, and then superimposes a line drawing or the like on the image, for example, an object or person moving in the video Still images can also be used. For this reason, there is also an effect of preventing displacement between the object in the image to be described and the line drawing.

  In the present embodiment, the first mode for sending an image with a high frame rate even when the screen resolution is reduced, and the second mode with a low frame rate even if the image resolution is high, in case the communication environment is severe. Prepare a mode. And according to this embodiment, when superimposing line drawings, by sending an image in the second mode, it is possible to transmit an image that is easy to see for the observer, and to convey the situation more accurately. It has become.

  In the present embodiment, an example in which information indicating a plurality of avatars is stored in advance in the avatar storage unit 307 is described, but the present invention is not limited thereto. For example, the processing unit 123 of the HMD 1 may receive information indicating the avatar of the user who is calling with the HMD 1 via the communication interface (131, 138).

  In the present embodiment, an example in which the color of the cursor and the color of the line drawing are made similar to each other has been described. However, the color of the avatar may also be made similar. For example, when the cursor by HDM1 displayed on the LCD 563 of the HMD 500 is light blue and the line drawing is blue, the clothes color of the avatar of HMD1 and the color of the hair may be blue. This makes it easier to visually recognize which user is drawing a cursor operation or line drawing.

  In this embodiment, the example in which the cursor display color and the line drawing display color are changed for each user has been described. However, the present invention is not limited to this. For example, the shape of the cursor, the type of line used for the line drawing (solid line, dotted line, broken line), and the thickness of the line used for the line drawing may be changed for each user.

  In the present embodiment, an example in which the users of the HMD 1 and the HMD 500 perform various operations by operating the touch switches (34, 534) has been described, but the present invention is not limited thereto. For example, the operation may be performed via the remote controller 140. Alternatively, in operations such as switching and selection, the processing units (122 and 522) of each terminal (HMD1 and HMD500) detect the number of blinks of the user, and based on the detected number of blinks, for example, You may control so that a cursor display, the drawing of a line drawing, etc. may be switched.

  In this embodiment, an example in which each terminal draws a line drawing on an image has been shown, but the present invention is not limited to this. For example, a comment input from the keyboard of the input unit 214 of the PC 210 may be superimposed on the received image. Even in this case, the comment is displayed and stored in a different layer from the received image in each terminal and server. For this reason, only the comment can be displayed and deleted.

In the present embodiment, an example in which video (moving image) is transmitted at low resolution in the initial state and switched to medium resolution or the like in response to a request from the user has been described. However, the present invention is not limited to this. There may be more types of resolution. Further, the user may set the resolution and the frame rate by using the touch switch 34. Specifically, the first setting of the resolution and the first frame rate at the low resolution, the second setting of the resolution and the second frame rate at the low resolution, the third setting of the resolution and the third frame rate at the medium resolution, and the like. .
However, when the user sets the resolution, it is necessary to exchange the call at the same resolution on the user terminal of the other terminal, so that information indicating the resolution to be used is transmitted at the start of the call. It may be.
The image to be transmitted may be monochrome or color depending on the application to be used. In the case of color, the image may be 8-bit color, 16-bit color, or the like.

In the example of FIGS. 9 to 19 of the present embodiment, an example in which a call is made by four parties including the user of the HMD 1 and the same image is shared among all users has been described. However, the present invention is not limited to this. For example, the terminal that requested the image may transmit the image information only to the requested terminal. Alternatively, the transmission source may limit the transmission destination or transmit the image information only to grouped members.
Further, not only for images but also for audio signals, the requested terminal may transmit the audio signal only to the requested terminal. Alternatively, the audio signal may be transmitted only to members who are limited by the transmission source or grouped.

  In addition, the display resolution of the terminal used by the user and the shooting resolution of the camera may be different. In this case, the display resolution of each terminal and the shooting resolution of the camera are transmitted to the other terminals and the server 200 at the start of the call. Based on these pieces of information, for example, the server 200 may set a common resolution used for a call and transmit the set common resolution to each terminal. Even when another terminal joins the call after setting the common resolution, for example, the server 200 resets the common resolution based on information from the terminal that joined the call later, The reset common resolution may be transmitted to each terminal.

  When a call is made with a plurality of call partners, when the LCD or display unit of a terminal (HMD) used by one user is in monochrome display and the camera 64 can only perform monochrome imaging, the processing of other terminals The unit may transmit a monochrome image in accordance with the terminal. In this case, there is an effect that data to be transmitted can be reduced. Alternatively, an image captured in color may be transmitted and converted into black and white on the terminal side and displayed.

  In the present embodiment, the example in which the user of the PC 210 operates the input unit 214 to enlarge and display the still image received from the HMD 1 on the display unit 215 (FIG. 14, step S40) has been described. May be performed on the HMD 1 side. For example, when the camera 64 included in the HMD 1 has a zoom function, the user of the PC 210 transmits an enlargement display request to the HMD 1. Then, in response to the received enlargement request, the processing unit 123 of the HMD 1 may transmit an image captured using the zoom function to the terminal that is in a call.

In this embodiment, an example in which an image is transmitted has been described. However, an audio signal may be included in a moving image, or only an image may be transmitted when there is a lot of ambient noise depending on the environment. Good.
Further, at the time of image transmission, the processing unit 123 adds or embeds information such as shooting date / time, terminal model information, shooting conditions (shutter speed, sensitivity, aperture value, lens focal length), etc., for transmission. It may be.
Further, the format of the image to be transmitted may be determined based on information registered in the server 200 in advance by registering a format reproducible by the terminal in the server 200. As a result, since it is not necessary to perform format conversion on the terminal side, users who are making a call can observe the same image at almost the same time, which has the effect of reducing communication errors.

  Note that the processing unit 123 may encrypt and transmit the image to be transmitted using, for example, a known technique in order to prevent the image to be transmitted from being exploited by a person who does not participate in the call. Further, the processing unit 123 may transmit the image to be transmitted by embedding “watermark”, for example, using a known technique.

  In the present embodiment, an example in which an avatar is used as information indicating the user of each terminal has been described. However, for example, a face photograph of the user may be used. In this case, for example, face photographs taken by the cameras (64, 564, 216, and 226) of each terminal may be recorded in the storage unit 202 of the server 200 in advance. Then, when making a call, the server 200 may transmit the face photograph of the user participating in the call to each terminal. Alternatively, a face photograph may be transmitted from each terminal to another terminal. When the entire face can be imaged during a call like the camera 216 of the PC 210, an image obtained by capturing the user's face instead of a face photograph may be transmitted to another terminal. In this case, the transmitted face image may have a resolution that can identify the user.

Note that the LCD 63 of the HMD 1 may be a display unit capable of three-dimensional display with the naked eye.
Further, the camera 64 may be provided with a camera capable of three-dimensional imaging. If the camera 64 is capable of three-dimensional imaging, the image information transmitted to the other terminal is a three-dimensional image if it is known in advance that the other terminal includes a display unit that can display three-dimensionally. Information may be transmitted.

  In this embodiment, the example in which the display main body 20 is mounted on the headband 40 has been described, but the present invention is not limited to this. The user may use the display body 20 alone without attaching it to the headband 40.

  In the present embodiment, the information display device includes an input / output detection unit (input unit) and an image composition unit, but the present invention is not limited to this. For example, a system having an apparatus including each unit may be used.

  The technical scope of the present invention is not limited to the above embodiment. At least one of the requirements described in the above embodiments may be omitted. The requirements described in the above embodiments can be combined as appropriate.

  The above-described HMD 1 (including 500) has a computer system (for example, the processing units 123 and 523) therein. The operation process of each functional unit is stored in a computer-readable recording medium in the form of a program, and the above processing is performed by the computer system reading and executing the program. Here, the computer system includes a CPU, various memories, an OS, and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, it also includes those that hold a program for a certain period of time, such as volatile memory inside a computer system that serves as a server or client in that case. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

DESCRIPTION OF SYMBOLS 1 ... Head mounted display, 20 ... Display main body, 34 ... Touch switch, 60 ... Display part, 63 ... Display panel (display part), 64 ... Camera, 72 ... Image sensor, 123 ... Processing part, 130 ... BT communication circuit, DESCRIPTION OF SYMBOLS 131 ... WiFi communication circuit, 132 ... Acceleration sensor, 133 ... Geomagnetic sensor, 137 ... Heart rate sensor, 138 ... 3G / LTE communication circuit, 140 ... Remote control, 301 ... Input detection part, 302 ... Image switching part, 303 ... Transmission / reception part , 304 ... Line drawing generation unit, 305 ... Cursor generation unit, 306 ... Avatar selection unit, 307 ... Avatar storage unit, 308 ... Image composition unit

Claims (7)

A display unit for displaying an image;
An input unit for detecting a user operation;
An imaging unit for imaging a subject;
A conversion unit for converting an image obtained by the imaging unit into a first resolution or a second resolution higher than the first resolution;
An image composition unit for displaying the second resolution image converted by the conversion unit on the display unit when the image obtained by the imaging unit is superimposed on the display unit by an operation to the input unit; Prepared ,
The information conversion device that converts the image to the second resolution after the conversion unit converts the image obtained by the imaging unit to the first resolution . The said image synthetic | combination part is provided with the following. When displaying the image acquired in the said imaging part on the said display part by superimposing by the operation to the said input part, the image currently displayed on the said display part is suspended. Information display device. The information display device according to claim 2, further comprising: a transmission / reception unit that transmits the temporarily stopped image displayed on the display unit to another terminal. The image composition unit
The information display device according to claim 3, wherein one of a resolution and a frame rate of an image transmitted to the other terminal is switched by the transmission / reception unit, and the image after the switching is transmitted to the other terminal. The image composition unit
When a still image is superimposed on the display unit by an operation on the input unit, the image displayed on the display unit is temporarily stopped, and a moving image is superimposed on the display unit by an operation on the input unit. 5. The information display device according to claim 1, wherein the image displayed on the display unit is not temporarily stopped. The image composition unit
When a line drawing in which a locus remains due to an operation on the input unit is to be superimposed on the display unit, the image displayed on the display unit is paused, and a cursor that does not leave a locus due to an operation on the input unit is displayed. The information display device according to any one of claims 1 to 5, wherein the image displayed on the display unit is not temporarily stopped when the display is superimposed on the unit. The information display device according to claim 6, wherein the image composition unit sets a color for displaying a line drawing that leaves a trace and a color for displaying a cursor that does not leave a trace to a similar color.