JP2013175929A - Information output device and information output method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information output device and an information output method, capable of easily identifying a speech target even when communicating with a plurality of communication targets.SOLUTION: The information output device includes: from a terminal, a position acquisition unit for acquiring position information of the terminal; from the terminal, a voice acquisition unit for acquiring voice information; based on the position information acquired by the position acquisition unit, a voice adjustment unit for distributing a voice signal acquired by the voice acquisition unit to right and left voice signals.

Description

  The present invention relates to an information output device and an information output method.

  A system technology for displaying an avatar of a communication partner in a TV (television) telephone, a TV conference, or the like is disclosed (for example, see Patent Document 1). In addition, an avatar is an image showing a user's alternation on a chat or Web (web), for example. In the technique described in Patent Document 1, for example, the arrangement of the avatar corresponding to the other party is first arranged based on the real person arrangement of the caller. And the avatar corresponding to a receiver has been arrange | positioned based on the positional information on the terminal which the receiver is using. For this reason, for example, when there are a plurality of speakers at the source, avatars corresponding to the plurality of speakers at the source are displayed on the receiver's terminal. For this reason, there is a problem in that it is difficult to identify which speaker is speaking from the sender even when the receiver looks at the avatar displayed on the terminal.

  For this reason, the technique described in Patent Document 2 discloses that the facial expression of the other party is photographed with a camera and the facial expression of the avatar is changed based on the photographed image.

Japanese Patent Laid-Open No. 10-55257 JP 2005-293335 A

  By the way, in the technique of patent document 2, since the one-to-one communication was the main use, the avatar for the speaker was always specified. However, in the one-to-many communication as in the technique of Patent Document 1, there is a problem that it is impossible to specify which avatar corresponds to the speaker.

  The present invention has been made in view of the above circumstances, and provides an information output device and an information output method capable of easily specifying an utterance partner even when communicating with a plurality of call partners. For the purpose.

  In order to achieve the above object, an information output apparatus according to an aspect of the present invention includes a position acquisition unit that acquires position information of the terminal from a terminal, a sound acquisition unit that acquires sound information from the terminal, and the position acquisition. And an audio adjustment unit that distributes the audio signal acquired by the audio acquisition unit into left and right audio signals based on the position information acquired by the unit.

  In order to achieve the above object, an information output method according to an aspect of the present invention is an information output method in an information output device, wherein a position acquisition unit acquires position information of the terminal from a terminal; The sound acquisition unit acquires sound information from the terminal, and the sound adjustment unit acquires the sound signal acquired by the sound acquisition procedure based on the position information acquired by the position acquisition procedure. And a sound adjustment procedure for distributing the signals.

  Since the balance between the left and right voices is switched according to the position based on the position information of the communication partner's terminal, it is easy to identify the speaking partner even when communicating with multiple call partners. it can.

It is a perspective view of the head mounted display concerning this embodiment. It is the perspective view which looked at the head mounted display concerning this embodiment from the back side. It is a figure which shows the mounting form of the head mounted display which concerns on this embodiment. It is a horizontal sectional view of a display body concerning this embodiment. It is a perspective view which shows the remote control which concerns on this embodiment. It is a perspective view which shows the stereo earphone attached to the head mounted display of this embodiment. It is a functional block diagram of the head mounted display 1 concerning this embodiment. It is a functional block diagram of the process part which concerns on this embodiment. 1 is a system configuration diagram of a communication system including an HMD 1 according to the present embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a flowchart which shows an example of the process sequence which concerns on this embodiment. It is a figure explaining an example of composition of an image displayed on LCD or a display part concerning this embodiment. It is an example of the image where the cursor concerning this embodiment was displayed. It is an example of the line drawing which concerns on this embodiment. It is a figure which shows an example of the grid which concerns on this embodiment. It is a figure which shows the other example of the grid which concerns on this embodiment. It is a flowchart of the display procedure of the avatar which concerns on this embodiment. It is a figure explaining an example of the image displayed while calling the other party of a call concerning this embodiment. It is a figure of an example of a display when the other party of a call concerning this embodiment participates in a call. It is a figure of an example of the display after rearranging the avatar which concerns on this embodiment. It is a figure of the example which displays the mouth and eyes | visual_axis of the avatar which concerns on this embodiment with an animation. It is a figure explaining the example of a display when one telephone call partner concerning an embodiment has left. It is a flowchart of adjustment of the left-right volume balance which the audio | voice adjustment part which concerns on this embodiment performs. It is a flowchart of the display procedure in the other avatar display which concerns on this embodiment. It is a figure explaining an example of the display image which has arranged the avatar concerning this embodiment two-dimensionally. In the display of FIG. 34, it is a figure explaining the example of an avatar display when the display part 60 of HMD1 rotates to the left about 90 degree | times.

The information output device of this embodiment arranges and displays a plurality of avatars, which are the alternates of the other party on the call, on the display unit based on the location of the place where the other party is making a call. The information output device changes the left / right balance of the voice of the other party based on the positional relationship of the avatars arranged in this way.
Hereinafter, an example in which the information output apparatus is applied to a head mounted display will be described, but the present invention is not limited to this.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
In the following description, an XYZ orthogonal coordinate system is set as necessary, and the positional relationship of each part will be described with reference to this XYZ orthogonal coordinate system. A predetermined direction in the horizontal plane is defined as an X-axis direction, a direction orthogonal to the X-axis direction in the horizontal plane is defined as a Y-axis direction, and a direction orthogonal to each of the X-axis direction and the Y-axis direction (that is, a vertical direction) is defined as a Z-axis direction. Further, the rotation (inclination) directions around the X axis, Y axis, and Z axis are the θX, θY, and θZ directions, respectively.

  FIG. 1 is a perspective view of the head mounted display of the present embodiment. FIG. 2 is a perspective view of the head-mounted display (head-mounted device, hereinafter also referred to as HMD) according to the present embodiment as viewed from the back side. FIG. 3 is a diagram showing a mounting form of the head mounted display of the present embodiment.

The HMD 1 is a monocular head mounted display that includes a display main body 20 and a headband 40 that is attached to the user's head and supports the display main body 20. The HMD 1 of the present embodiment can be used with either eye as shown in FIG. FIG. 3A shows a state where the user is viewing the display unit 60 with the right eye, and FIG. 3B shows a state where the user is viewing with the left eye. As shown in FIG. 2, the display main body 20 and the headband 40 are configured to be detachable via a connecting pin 41.
1 and 2, the longitudinal direction of the display main body 20 is defined as the Y-axis direction, and the direction in which the headband 40 sandwiches the user's head is defined as the X-axis direction.

Hereinafter, the configuration of each part of the HMD 1 will be described in detail.
The display main body 20 includes a device main body 21 that includes a main circuit and includes an operation unit and various interfaces, and a display unit 60 that is connected to the tip of the device main body 21.

  The apparatus main body 21 has a substantially plate-shaped housing 21A shown in FIG. In the present embodiment, an end portion (+ Y side end portion) on the side where the connection portion with the headband 40 is provided in the apparatus main body portion 21 is a base end portion, and an end portion on the opposite side of the base end portion ( The end on the -Y side) is the tip. Further, in a state where the apparatus main body 21 is attached to the headband 40, the headband 40 side (+ X side) of the apparatus main body 21 is the inside, and the opposite side (−X side) from the headband 40 is the outside.

As shown in FIG. 1, a main switch 28, a touch switch 34, and a sound collecting microphone 24 are arranged on the outer surface of the apparatus main body 21 along the longitudinal direction of the housing 21A.
The main switch 28 is a switch for turning on / off the power of the display body 20. The touch switch 34 is a touch panel that can perform various operations of the HMD 1 by touching the surface with a finger or the like. The sound collection microphone 24 is an external microphone that collects environmental sounds.

  As shown in FIG. 2, an ear speaker 23, an audio connector 26, and a headband hinge 32 having a connection hole 31 are provided on the base end side of the inner surface of the apparatus main body 21. A heart rate sensor 137 is provided at the center of the inner surface of the apparatus main body 21. A call microphone 37 is provided at the tip of the inner side surface of the apparatus main body 21.

  The ear speaker 23 is disposed in the vicinity of the user's ear. Audio information is transmitted from the ear speaker 23 to the user. The audio connector 26 is an audio input / output terminal to which, for example, the earphone shown in FIG. 6 is connected. The headband hinge 32 is a joint part with the headband 40. The voice of the user is input to the call microphone 37.

  The heart rate sensor 137 is a sensor that measures a user's heart rate by contacting the surface of the user's face. The heart rate sensor 137 includes a light emitting unit including a light emitting diode and the like, and a light receiving unit that detects light reflected inside the user's skin. The heart rate sensor 137 counts the heart rate by detecting a change in the amount of reflected light due to a change in blood flow. The heart rate sensor 137 is disposed near the user's eyes, but if the configuration is such that infrared light is emitted from the light emitting unit, the user will not feel glare.

A USB connector 25, an operation switch 30, and a video connector 27 are provided on the side end surface of the apparatus main body 21 on the base end side.
The USB connector 25 is a connection terminal of a USB (Universal Serial Bus) device. In the present embodiment, for example, a remote controller (remote controller) 140 shown in FIG. 5 is connected.
The operation switch 30 is a pointing device such as a trackball or a stick. The operation switch 30 is provided so as to face the screen displayed on the display unit 60. As a result, the left-right direction of the operation on the operation switch 30 matches the left-right direction of the screen, so that the user can intuitively operate the operation switch 30 while viewing the screen.
The video connector 27 is a video input / output terminal.

FIG. 4 is a horizontal sectional view of the display main body 20 of the present embodiment.
As shown in FIG. 4, the apparatus body 21 includes a plate-like circuit board 29 extending along the longitudinal direction of the housing 21 </ b> A and a battery 33. A control circuit, a power supply circuit, and the like (not shown) are mounted on the circuit board 29, and are electrically connected to each part of the display body 20 via wiring (not shown).

  A display panel 36 made of a liquid crystal panel and a backlight 35 are arranged inside the touch switch 34 exposed on the outer surface of the apparatus main body 21. In the present embodiment, the display image on the display panel 36 is displayed through the touch switch 34. The display panel 36 and the backlight 35 may be an organic EL panel or an electrophoresis panel.

  The headband hinge 32 is a ball joint composed of a concavely curved housing portion 32a provided in the housing 21A and a spherical portion 32b fitted to the housing portion 32a. The spherical portion 32b has a spherical side surface portion and two plane portions formed in parallel to each other so as to sandwich the side surface portion. A connecting hole 31 is formed so as to penetrate the two flat portions vertically. The connection hole 31 is formed in a hexagonal shape when viewed in the axial direction. The display main body 20 and the headband 40 are connected by inserting the connection pin 41 of the headband 40 into the connection hole 31.

  By providing the headband hinge 32, the display main body 20 can be rotated in the A direction (around the X axis centering on the headband hinge 32) shown in FIG. In the present embodiment, the rotatable range of the display body 20 is about 270 °. By the rotation operation around the X axis, a switching function between a mode in which an image is observed with the right eye shown in FIG. 3A and a mode in which an image is observed with the left eye shown in FIG. 3B is realized.

  Further, since the headband hinge 32 is a ball joint, the display main body 20 can also be swung in the B direction (around the Z axis centering on the headband hinge 32) shown in FIG. By this swinging operation, the position of the display main body 20 relative to the user's eyes and ears can be adjusted.

  The heart rate sensor 137 in the vicinity of the headband hinge 32 is provided so as to protrude from the inner surface of the apparatus main body 21, and can be brought into contact with the surface of the user's face when the HMD 1 is worn. The battery 33 may be either a primary battery or a secondary battery.

  As shown in FIGS. 1 and 2, the display unit 60 is connected to the distal end portion of the apparatus main body unit 21. In the present embodiment, the headband 40 side of the display unit 60 is the inside, and the side opposite to the headband 40 is the outside. The display unit 60 is an arm member that has a curved shape in a top view (Z-axis view), and has a shape that curves inward from the connecting portion with the apparatus main body 21 toward the distal end side. A finder opening 67 is provided on the inner surface of the display unit 60. A camera 64 is provided on the outer surface of the display unit 60.

  As shown in FIG. 4, the display unit 60 is connected to the apparatus main body 21 via a display hinge 61. The display hinge 61 is a ball joint composed of a concave curved housing 61a formed in the housing 60A of the display 60 and a spherical portion 61b formed in the apparatus main body 21 and fitted in the housing 61a. .

  The spherical portion 61b of the apparatus main body 21 has an inclined surface extending obliquely with respect to the longitudinal direction of the casing 21A formed at the front end of the outer surface of the apparatus main body 21, and the normal direction of the inclined surface (Y in FIG. 4). It is provided so as to protrude in the “axial direction”.

  The display unit 60 can be freely rotated around the Y ′ axis with respect to the spherical portion 61 b of the display hinge 61. In the present embodiment, the display hinge 61 is provided on the inclined surface of the apparatus main body 21, and the display unit 60 has a shape curved inward.

Further, since the display hinge 61 is a ball joint, the display unit 60 can swing in a direction orthogonal to the rotation plane around the Y ′ axis (generally along the X ′ axis). This swinging operation facilitates position adjustment of the finder opening 67 by the user. If the interval between the display unit 60 and the apparatus main body unit 21 is increased, the swinging width of the display unit 60 can be increased.
That is, according to the present embodiment, the display hinge 61 serving as the holding unit can swing the display unit 60 around at least one axis.

  The spherical portion 61b of the display hinge 61 is formed with a through hole 61c that penetrates the spherical portion 61b in the height direction (Y′-axis direction). The inside of the display unit 60 and the inside of the apparatus main body unit 21 are communicated with each other through the through hole 61c. A cable (not shown) is inserted through the through hole 61c. The circuit board 29 and each part of the display unit 60 are electrically connected via the inserted cable.

  Inside the display unit 60, there are a backlight 62, a display panel 63, a camera 64, a prism 65, a reflecting mirror 66, a front light 68, a front speaker 70, an imaging lens 71, and an image sensor 72. And are provided.

  The prism 65 has a configuration in which a substantially triangular first prism 65a and a second prism 65b are bonded to each other on a top view (Z-axis view). A display panel 63 made of a liquid crystal panel is provided at a position facing one of the other two surfaces of the first prism 65a other than the bonding surface. A backlight 62 that transmits and illuminates the display panel 63 is disposed on the back surface of the display panel 63. A reflection mirror 66 is disposed at a position facing the other surface of the first prism 65a. The reflection mirror 66 is located substantially in front of the finder opening 67.

  Of the two surfaces other than the bonding surface of the second prism 65 b, one surface is a viewfinder eyepiece disposed in the viewfinder opening 67. An imaging element 72 is disposed opposite to the other surface of the second prism 65b via the imaging lens 71.

  In the display unit 60, the image displayed on the display panel 63 is emitted from the finder opening 67 through the first prism 65a, the reflection mirror 66, and the second prism 65b, and is observed by the user. The eye of the user looking at the finder opening 67 is imaged on the image sensor 72 via the second prism 65 b and the imaging lens 71. The eye image acquired through the image sensor 72 is used for analyzing the direction of the user's line of sight, blinking, and facial expression. The backlight 62 is also used as a lighting device for photographing the eye by the image sensor 72.

The camera 64 includes, for example, an image sensor with 5 to 10 million pixels, and is configured to be able to perform an autofocus operation.
In the case of this embodiment, the shooting direction of the camera 64 is set so as to coincide with the user's line-of-sight direction. The adjustment between the photographing direction and the line-of-sight direction may be performed mechanically or by image processing. For example, the camera 64 is configured to be able to acquire a wide-angle image, and a part of the electrically captured image is extracted as an image in the line-of-sight direction based on the information on the line-of-sight direction acquired from the user's eye image. 63 is displayed. As a result, it is possible to shoot and display an image in front of the user without providing an adjustment mechanism in the camera 64.

  The front light 68 is, for example, an LED light. The front light 68 may include red, green, and blue light emitting elements, and may be configured to emit any color at any timing. The front light 68 may be used as a device that displays information to the outside depending on the light emission color and the light emission timing, or may be used as an illumination device when photographing with the camera 64.

  Although not shown in FIG. 4, the display unit 60 according to the present embodiment is provided with a laser transmitter. The laser transmitter can be provided in the vicinity of the front light 68, for example. By irradiating, for example, red laser light emitted from a laser transmitter forward, pointing (display of an index) with the laser light is possible.

Next, the headband 40 will be described with reference to FIG.
As shown in FIG. 2, the headband 40 includes a pair of head pads (mounting members) 46 and 47 that sandwich the user's head, a first headband 43, a second headband 44, and a rotation mechanism 56. , 57.

  The first headband 43 is an elastic member having an arc shape as a whole. A joint portion 43 a for bending the first headband 43 is provided on the top of the first headband 43. At both ends of the first headband 43, bearing portions 43b and 43c constituting the rotation mechanisms 56 and 57 are provided, respectively. Bearing portions 43d and 43e to which the head pads 46 and 47 are connected are provided further on the front end side of the band than the bearing portions 43b and 43c.

  The second headband 44 is an elastic member having an arc shape as a whole. A joint portion 44 a for bending the second headband 44 is provided on the top of the second headband 44. At both ends of the second headband 44, shaft members 44b and 44c constituting rotation mechanisms 56 and 57 are provided.

  In the present embodiment, the second headband 44 has a configuration in which the surface of the spring member 48 made of a metal such as stainless steel is covered with a flexible material such as a resin. The spring member 48 of the second headband 44 generates a spring force that holds the user's head. The second headband 44 is formed wider than the first headband 43. The formation part of the joint part 44a is a mark display part 49 formed wider than the other band parts. A product tag or the like is attached to the mark display section 49 by sticker or printing. The entire second headband 44 may be made of metal.

  The head pad 46 includes a plate-like support plate 46a and an elastic member 46b having an arched cross section provided on one surface side of the support plate 46a. On the surface of the support plate 46a opposite to the elastic member 46b, a substantially hexagonal column-shaped connecting pin 41 is erected in a posture perpendicular to the surface. The connecting pin 41 is pivotally supported by a bearing portion 43 d provided at one end of the first headband 43. As a result, the head pad 46 can rotate around the connecting pin 41.

  The head pad 47 includes a plate-like support plate 47a and an elastic member 47b having an arcuate cross section provided on one surface of the support plate 47a. A shaft member 47c is provided on the surface of the support plate 47a opposite to the elastic member 47b. The shaft member 47 c is pivotally supported by a bearing portion 43 e provided at the tip of the first headband 43. Thereby, the head pad 47 can rotate around the shaft member 47c.

  With this configuration, the second headband 44 is urged to rotate in a direction away from the first headband 43 by the rotation mechanism 56, while being within a predetermined angle with respect to the first headband 43 by the rotation mechanism 57. The rotation is restricted to the range. As a result, the second headband 44 is held at a predetermined angle with respect to the first headband 43. Therefore, the rotation mechanisms 56 and 57 function as an angle holding mechanism in the headband 40.

FIG. 5 is a perspective view showing the remote controller of the present embodiment.
The remote controller 140 includes a housing 140A, a touch pad 141, an operation switch 147, a USB connector 142, indicators 143a, 143b, and 143c, a mounting clip 144, and an indicator switch 150.

  The remote controller 140 is electrically connected to the apparatus main body 21 by connecting one terminal of the USB cable 148 to the USB connector 142 and connecting the other terminal to the USB connector 25 of the apparatus main body 21. A commercially available USB cable can be used as the USB cable 148. In the present embodiment, a plurality of clips 149 are provided in the middle of the USB cable 148. The plurality of clips 149 can be used to fix the USB cable 148 along the second headband 44, for example.

  The remote control 140 incorporates a communication circuit for performing wireless communication with the display main body 20 and other electronic devices. As a communication method of the communication circuit, for example, Bluetooth (registered trademark), wireless LAN, infrared communication, UWB (Ultra Wide Band), Transfer Jet, or the like can be used. With such a communication circuit, communication with the display main body 20 is possible even when the USB cable 148 is not connected.

  The remote control 140 may incorporate a G sensor, an acceleration sensor, and a direction sensor. Information detected by these sensors can also be communicated to the display body 20 via the communication circuit described above.

  The touch pad 141 is provided on one main surface of the housing 140A. The touch pad 141 is an operation unit for operating the display main body 20 when the user touches. A display panel and a backlight may be provided in the housing 140 </ b> A on the back side of the touch pad 141, similarly to the touch switch 34 of the apparatus main body 21.

  The operation switch 147 is a switch for manually operating the remote controller 140 and the display main body 20. The operation switch 147 may be provided as necessary. The indicator switch 150 provided on the side surface of the housing 140A is a switch for performing a predetermined operation using the indicators 143a to 143c.

The USB connector 142 is used not only for connection with the apparatus main body 21 but also for connection with a charging terminal of the remote control 140 or a PC when the remote control 140 has a built-in rechargeable battery.
When the remote controller 140 has a built-in battery, it can also be used as a power supply terminal for the apparatus main body 21. By making the battery of the remote control 140 usable as an auxiliary power source, the battery capacity of the device main body 21 can be reduced, and the device main body 21 can be reduced in size and weight.

  Indicators 143a, 143b, and 143c are arranged around the touch pad 141. In the present embodiment, an indicator 143a is disposed at a position along one side of the touch pad 141, and an indicator 143b and an indicator 143c are disposed on the opposite side of the touch pad 141 with respect to the indicator 143a.

  The indicators 143a to 143c include light emitting elements such as LEDs. In the present embodiment, the indicator 143a includes a green light emitting element, the indicator 143b includes an orange light emitting element, and the indicator 143c includes a blue light emitting element. These light emitting elements are configured to be capable of pattern light emission at controlled timing. The pattern light emission of the indicators 143a to 143c can be turned on and off by operating the indicator switch 150 on the side surface of the housing 140A.

  The pattern light emission of the indicators 143a to 143c indicates, for example, the state of the remote controller 140 (for example, the up / down / left / right tilt state of the remote controller 140). It is possible to transmit the state of the remote controller 140 to the apparatus main body 21 by photographing such pattern emission with the camera 64 of the apparatus main body 21 and analyzing the acquired image. The information transmission method using the indicators 143a to 143c can suppress power consumption as compared with the case where wireless communication such as Bluetooth (registered trademark) is used.

  The attachment clip 144 is provided on the surface of the housing 140A opposite to the touch pad 141. In the case of this embodiment, the attachment clip 144 is an S-shaped spring plate member having one end fixed to the housing 140A. With the attachment clip 144, the remote controller 140 can be attached to a user's clothes, belt, or the like.

FIG. 6 is a perspective view showing a stereo earphone attached to the head mounted display of the present embodiment.
Stereo earphone 100 includes a connector 101, a cable 102, a first speaker 103, a second speaker 104, a sound collecting microphone 105, and a plurality of clips 106.

  The connector 101 is provided at one end of the cable 102. The connector 101 is a general 4-pole φ3.5 mm mini plug. The breakdown of the four poles is the sound collection microphone 105, the first speaker 103, the second speaker 104, and the ground (GND). The cable 102 is bifurcated in the vicinity of the connector 101, and a first speaker 103 is provided at the end of the branched cable. A second speaker 104 and a sound collecting microphone 105 are provided at the other end of the cable 102. The plurality of clips 106 are arranged on the cable 102 at a predetermined interval.

  The stereo earphone 100 is used by connecting the connector 101 to the audio connector 26 of the display body 20. When the stereo earphone 100 is connected, the ear speaker 23 of the display body 20 and the call microphone 37 of the display unit 60 are invalidated. Further, the front speaker 70 of the display unit 60 is also invalidated as necessary. Then, the first speaker 103, the second speaker 104, and the sound collecting microphone 105 of the stereo earphone 100 are activated.

  The first speaker 103 of the stereo earphone 100 is attached to the ear of the user on the side where the display main body 20 is arranged, and the second speaker 104 is attached to the ear on the opposite side to the first speaker 103. At this time, the cable 102 can be fixed to the second headband 44 by the clip 106.

  Further, stereo recording can be performed by the sound collecting microphone 105 of the stereo earphone 100 and the sound collecting microphone 24 provided on the outer surface of the housing 21 </ b> A in the display main body 20. For example, if the display main body 20 is arranged on the right eye side as shown in FIG. 3A, the sound collecting microphone 24 of the display main body 20 collects sounds on the right side of the user and is attached to the left ear. The sound collecting microphone 105 collects the sound on the left side of the user. Note that the right channel sound is output from the first speaker 103 of the stereo earphone 100, and the left channel sound is output from the second speaker 104.

  On the other hand, when the display main body 20 is arranged on the left eye side as shown in FIG. 3B, the sound collecting microphone 24 of the display main body 20 collects the sound of the left side of the user and is attached to the right ear. The sound collecting microphone 105 collects the sound on the right side of the user. The left channel sound is output from the first speaker 103 of the stereo earphone 100, and the right channel sound is output from the second speaker 104.

  In the case of the present embodiment, since the sound collection microphones 24 and 105 are both disposed in the vicinity of the user's ear, recording with a sense of reality such as binaural recording is possible. Also, by using microphones having the same characteristics as the sound collecting microphone 24 and the sound collecting microphone 105, the difference in the characteristics of the left and right sounds can be reduced.

FIG. 7 is a functional block diagram of the head mounted display 1 of the present embodiment.
The HMD 1 includes various electric circuits centering on the processing unit 123. The processing unit 123 is a CPU (Central Processing Unit), and is connected to various circuits of the HMD 1 and comprehensively controls the HMD 1.

The battery 33 supplies power to the power supply circuit 120.
The power supply circuit 120 supplies the power supplied from the battery 33 to each unit of the head mounted display 1 under the control of the processing unit 123.

  In the present embodiment, the processing unit 123 includes an encoder 129 and a decoder 121, a power supply circuit 120, a BL driver 126, an operation switch 30, a flash memory 122, a BT communication circuit 130, a WiFi communication circuit 131, an acceleration sensor 132, and a geomagnetic sensor. 133, front light 68, 3G / LTE communication circuit 138, laser transmitter 73, angular velocity sensor 134, GPS sensor 135, temperature / humidity sensor 136, heart rate sensor 137, memory 127, main switch 28, and touch switch 34 are connected. Yes.

  The encoder 129 encodes (encodes) the audio signal and the video signal into audio data and video data of a predetermined method. Connected to the encoder 129 are a camera 64, an image sensor 72, a sound collection microphone 24, a call microphone 37, an audio connector 26, and a video connector 27.

  The encoder 129 includes an audio signal input from the sound collection microphone 24 and the call microphone 37, a video signal input from the camera 64, an audio signal input from the audio connector 26, a video signal input from the video connector 27, and The video signal of the image sensor 72 that captures the user's eyes is input. The audio signal and video signal input to the encoder 129 are encoded into audio data and video data, and then input to the processing unit 123. The input audio data and video data are used for a reproduction operation by the processing unit 123 or recorded in the flash memory 122.

  The decoder 121 decodes (decodes) audio data and video data into audio signals and video signals. An LCD driver 125, a speaker amplifier 162, an audio connector 26, and a video connector 27 are connected to the decoder 121. The LCD driver 125 is a drive control device for a liquid crystal panel, and is connected to the display panel 36 and the display panel 63. The speaker amplifier 162 is a device that amplifies an audio signal and outputs it to the speaker, and is connected to the ear speaker 23 and the front speaker 70.

  When reproducing the video data, the video data recorded in the flash memory 122 or the video data input from the encoder 129 is input to the decoder 121 via the processing unit 123. The video data input to the decoder 121 is decoded into a video signal and then supplied to the display panels 36 and 63 via the LCD driver 125. Then, an image based on the image data is displayed on the display panel 36 or the display panel 63 to which the image signal is input. The video signal output from the decoder 121 to the video connector 27 is output to an external device via the video connector 27.

  When displaying the video, the processing unit 123 turns on the backlight 35 for the display panel 36 and the backlight 62 for the display panel 63 as necessary. The BL driver 126 is a drive control device for the backlight, and is connected to the backlight 35 and the backlight 62. The processing unit 123 transmits a drive signal to the BL driver 126, and the BL driver 126 lights the backlights 35 and 62 individually.

  When reproducing the audio data, the audio data recorded in the flash memory 122 or the audio data input from the encoder 129 is input to the decoder 121 via the processing unit 123. The audio data input to the decoder 121 is decoded into an audio signal and then output to one or both of the ear speaker 23 and the front speaker 70 via the speaker amplifier 162. Then, sound is output from the ear speaker 23 or the front speaker 70 to which the sound signal is input. The audio signal output from the decoder 121 to the audio connector 26 is output to the stereo earphone 100 via the audio connector 26.

  In the case of this embodiment, the ear speaker 23 and the front speaker 70 are assumed to use monaural sound, and the ear speaker 23 and the front speaker 70 emit a sound in which left and right audio signals are synthesized.

  On the other hand, when an audio signal is output to the stereo earphone 100, the sound of the left channel or the right channel of the first speaker 103 and the second speaker 104 is output. Here, since the HMD 1 of this embodiment is for both left and right, the channel of the audio signal to the stereo earphone 100 is switched according to the mounting position. That is, when the display body 20 is attached to the right eye, the right channel sound is output to the first speaker 103 and the left channel sound is output to the second speaker 104. When the display body 20 is attached to the left eye, the left channel sound is output to the first speaker 103 and the right channel sound is output to the second speaker 104.

The memory 127 stores a control program executed by the processing unit 123.
When the user operates the main switch 28 for turning on / off the entire power source, the operation switch 30 for performing a pointing operation in the screen, or the touch switch 34 for performing various operations by a touch operation, the operation is performed from these switches. Is output to the processing unit 123. The processing unit 123 detects an operation based on the control signal, and executes an operation defined in the control program.

  The BT communication circuit 130 is a communication circuit for performing Bluetooth (registered trademark) communication with other devices. The WiFi communication circuit 131 is a communication circuit for performing wireless LAN communication (IEEE 802.11) with other devices. The 3G / LTE communication circuit 138 is a communication circuit for performing mobile communication with other devices.

  The acceleration sensor 132 is used for detecting the inclination of the HMD 1. The geomagnetic sensor 133 is used for detecting the direction of the HMD 1. An angular velocity sensor (gyro sensor) 134 is used to detect rotation of the HMD 1. The GPS sensor 135 is used for positioning detection using GPS (Global Positioning System). The temperature / humidity sensor 136 is used to detect the temperature and humidity of the environment. The heart rate sensor 137 contacts the user's cheek and detects the user's heart rate.

FIG. 8 is a functional block diagram of the processing unit 123 of the present embodiment.
As illustrated in FIG. 8, the processing unit 123 includes an input detection unit 301, an image switching unit 302, a transmission / reception unit 303, a line drawing generation unit 304, a cursor generation unit 305, an avatar generation unit 306, an avatar storage unit 307, and an image composition unit 308. , A position calculation unit 309 (position acquisition unit), and an audio adjustment unit 310 (audio acquisition unit, audio adjustment unit). In FIG. 8, the WiFi communication circuit 131 and the 3G / LTE communication circuit are collectively referred to as communication interfaces (131, 138).

The input detection unit 301 determines the type of operation performed on the touch switch 34, and outputs a control signal based on the determined result to the image switching unit 302, the transmission / reception unit 303, and the avatar generation unit 306. The types of operations performed on the touch switch 34 are a transmission instruction, a reception instruction, a cursor display instruction, a line drawing display instruction, and the like. When a predetermined area on the touch switch 34 is touched, the input detection unit 301 determines the type of these operations based on the touched area. Further, the input detection unit 301 detects the movement of the finger on the touch switch 34 at the time of cursor display or line drawing display.
In the case of cursor display, the input detection unit 301 outputs information indicating a position corresponding to the finger movement to the cursor generation unit 305. In the case of line drawing display, the input detection unit 301 outputs information indicating the position according to the finger movement to the line drawing generation unit 304.

  The image switching unit 302 outputs the image data read from the flash memory 122 or the image data output from the encoder 129 to the image composition unit 308 based on the control signal output from the input detection unit 301. The image switching unit 302 extracts an image switching signal from the reception signal output from the transmission / reception unit 303. When the image switching signal can be extracted, the image switching unit 302 outputs the image data read from the flash memory 122 or the image data output by the encoder 129 to the image composition unit 308 based on the extracted image switching signal. The image switching unit 302 converts the resolution of the image data output from the encoder 129 or the image data read from the flash memory 122 based on the control signal output from the input detection unit 301, and converts the converted image data into an image. The data is output to the combining unit 308.

  Based on the control signal output from the input detection unit 301, the transmission / reception unit 303 transmits the image data output from the image composition unit 308 via the communication interfaces (131, 138). The transmission / reception unit 303 outputs reception signals received via the communication interfaces (131, 138) to the image switching unit 302, the line drawing generation unit 304, and the cursor generation unit 305.

  The line drawing generation unit 304 generates line drawing data with a locus corresponding to the finger movement based on the information indicating the position corresponding to the finger movement output from the input detection unit 301. The line drawing generation unit 304 outputs the generated line drawing data to the image composition unit 308. The line drawing generation unit 304 extracts line drawing data from the reception signal output from the transmission / reception unit 303, and outputs the extracted line drawing data to the image composition unit 308.

  The cursor generation unit 305 generates cursor image data without a locus corresponding to the finger movement based on the information indicating the position corresponding to the finger movement output from the input detection unit 301. The cursor generation unit 305 outputs the generated cursor image data to the image composition unit 308. The cursor generation unit 305 extracts cursor image data accompanied by position information from the reception signal output from the transmission / reception unit 303 and outputs the extracted cursor image data to the image composition unit 308.

The avatar generation unit 306 reads avatar image data set in advance for the user of the HMD 1 from the avatar storage unit 307 based on the control signal output from the input detection unit 301. The avatar generation unit 306 outputs the read avatar image data to the transmission / reception unit 303.
The avatar production | generation part 306 extracts the information which shows another user's avatar from the received signal which the transmission / reception part 303 outputs. The avatar generation unit 306 reads the avatar image data corresponding to the user from the avatar storage unit 307 based on the extracted information indicating the other user's avatar.
The avatar is an illustration of the face of the user of the HMD 1 and the user who is transmitting and receiving with the HMD 1. The avatar image data may be configured by parts such as the appearance of the face, hair, eyes, eyebrows, nose, mouth, ears, and the like.

The avatar generation unit 306 generates a facial expression of an avatar (hereinafter referred to as HMD1 avatar) for the HMD1 user based on the image captured by the image sensor 72 and the read avatar image data. Note that the image captured by the image sensor 72 is an image captured including the eyes and mouth of the user of the HMD 1. For example, the avatar generation unit 306 extracts an image captured by the image sensor 72 using a well-known image recognition technique, such as whether the mouth is moving or whether the line of sight is facing left or right. The avatar generation unit 306 changes the parts of the eyes and mouth from the parts in the initial state based on the extracted information, for example, from the avatar image data. In addition, the parts in the initial state are, for example, that the eyes are facing the front and the mouth is closed. The parts to be changed are those whose eyes are facing left, those whose eyes are facing right, those whose mouth is open, and the like.
In addition, you may make it the avatar image data with respect to the other terminal contained in the received signal output from the transmission / reception part 303 contain the expression information of the avatar produced | generated by the other terminal. Or the avatar production | generation part 306 extracts the image which imaged the user's face (including the eyes and mouth) of the user of another terminal from the received signal output from the transmission / reception part 303, and another terminal based on the extracted image Information indicating the expression of the avatar for the user may be generated.
The avatar generation unit 306 outputs the generated HMD1 avatar image data (including information indicating the expression of the avatar) to the transmission / reception unit 303. That is, the avatar image data of the HMD 1 is transmitted to each terminal of a user who makes a call via the transmission / reception unit 303 and the communication interfaces (131, 138).

The avatar generation unit 306 calculates the arrangement of the HMD 1 and avatars of other terminals based on the position information output from the position calculation unit 309. For example, the avatar generation unit 306 extracts information related to longitude from the position information, and based on the extracted information related to longitude, the avatar corresponding to the user who is east relative to the HMD 1 Place to the right. In this way, the avatar generation unit 306 sequentially compares the longitudes and arranges them on the display panel 63 in order from the right. The avatar production | generation part 306 outputs the positional information on each avatar arrange | positioned in this way, and the avatar image data of another terminal to the image synthetic | combination part 308. FIG.
In addition, the avatar generation unit 306 outputs the generated position information of each avatar to the audio adjustment unit 310.

Based on the position information output from the position calculation unit 309, the avatar generation unit 306 generates an additional information icon indicating an environment where the user of the HMD 1 is present, as will be described later. In addition, the avatar generation unit 306 generates an additional information icon indicating the environment in which the user of each terminal exists, as will be described later, based on the position information output from the transmission / reception unit 303. The environment where the user of each terminal is present is, for example, a home, a company, the sea, a train, or the like. The avatar generation unit 306 outputs image data indicating the additional information icon for the generated HMD 1 to the transmission / reception unit 303. That is, the processing unit 123 transmits the image data indicating the additional information icon corresponding to the HMD1 to the terminal of the user who makes a call via the transmission / reception unit 303 and the communication interfaces (131, 138).
The avatar generation unit 306 outputs image data indicating the additional information icon corresponding to the user of each terminal to the image composition unit 308.
In addition, the avatar generation unit 306 determines from the received signal output from the transmission / reception unit 303 whether another terminal is being called or another terminal is participating in a call. If the result of the determination is that another terminal is being called, the avatar generation unit 306 generates image information indicating that the terminal is being called, and outputs the generated image information indicating that the call is being performed to the image composition unit 308.

  In the avatar storage unit 307, for example, a plurality of avatar image data is stored in advance.

The image composition unit 308 receives the image data output from the image switching unit 302 and the line drawing data output from the line drawing generation unit 304. Further, the image composition unit 308 receives the cursor image information with the position information output from the cursor generation unit 305, the avatar image data output from the avatar generation unit 306, and the avatar additional information output from the avatar generation unit 306. The
The image composition unit 308 synthesizes images based on these input image data. The image synthesis unit 308 outputs the synthesized image to the transmission / reception unit 303, the video connector 27, and the LCD driver 125.

The position calculation unit 309 relates to an environment in which the HMD 1 is used based on outputs from various sensors (acceleration sensor 132, geomagnetic sensor 133, angular velocity sensor 134, GPS sensor 135, temperature / humidity sensor 136, heart rate sensor 137, etc.). Environment information that is information is generated. The position calculation unit 309 generates position information where the HMD 1 is used based on the generated environment information. The position calculation unit 309 outputs the generated position information of the HMD 1 to the transmission / reception unit 303 and the avatar generation unit 306.
The position calculation unit 309 extracts environment information of another terminal from the reception signal output from the transmission / reception unit 303, and generates position information where the other terminal is used based on the extracted environment information.
The position calculation unit 309 outputs position information where the generated other terminal is used to the avatar generation unit 306.

The audio adjustment unit 310 outputs audio data output from the encoder 129 (HMD1 user audio data, hereinafter also referred to as HMD1 audio data), a reception signal output from the transmission / reception unit 303, and an avatar generation unit 306 Avatar position information is input. The voice adjustment unit 310 extracts the voice data of the user of the other terminal (hereinafter also referred to as voice data of the other terminal) from the reception signal output from the transmission / reception unit 303. Based on the position information of each avatar output from the avatar generation unit 306, the audio adjustment unit 310 outputs the audio based on the audio data of the HMD1 and the audio data of other terminals to the left and right speakers, Adjust each volume. For example, when two people are talking, when one user (HMD1 user) is located in the east, the audio adjustment unit 310 reproduces the HMD1 audio from the right speaker. Adjust the audio signal. When the other user (user of another terminal) is located in the west, the audio adjustment unit 310 adjusts the audio signal so as to reproduce the audio of the user of the other terminal from the left speaker. That is, in this embodiment, not only the display position of the avatar but also the reproduction of the sound is adjusted based on the positional relationship of the user.
Further, the audio adjustment unit 310 may adjust the volume according to the positional relationship between the terminals. For example, when three people are talking and only one person is far away, the sound adjustment unit 310 reproduces the sound signal of the user who is far away at a volume smaller than the sound signal of the user who is near. As a result, it becomes possible for the user who is listening to the sound with the stereo earphone 100 to perceive the positional relationship of each user who is making a call even with the sound.

Next, a procedure when the HMD 1 communicates with the head mounted display 500 (hereinafter referred to as the HMD 500), the PC 210, and the PC 220 used by other users will be described.
FIG. 9 is a system configuration diagram of a communication system including the HMD 1 according to the present embodiment. 10 to 20 are flowcharts illustrating an example of a processing procedure according to the present embodiment.

  As illustrated in FIG. 9, the communication system includes an HMD 1, an HMD 500, a PC 210, a PC 220, and a server 200. The HMD 1, HMD 500, PC 210, PC 220, and server 200 are connected to the network 230. The network 230 is a network such as the Internet connected via a wireless LAN (Local Area Network), a 3G line, an LTE line, or the like.

In FIG. 9, the HMD 1 has the configuration described in FIG. 7, but shows only functional units necessary for explaining the processing procedure. The display panel 63 of the HMD 1 is hereinafter referred to as an LCD 63.
As shown in FIG. 9, the HMD 500 mainly includes a flash memory 522, a processing unit 523, communication interfaces (531, 538), a geomagnetic sensor 533, a touch switch 534, a display panel (display unit) 563 (hereinafter referred to as an LCD 563), A camera 564 is provided. Each function part of HMD500 is equivalent to each function part of HMD1.

Next, the configuration of the server 200 will be described.
As illustrated in FIG. 9, the server 200 includes a processing unit 201, a storage unit 202, and a communication interface 203.

  The processing unit 201 of the server 200 includes image data (moving image, still image, line drawing, etc.) received from each terminal (HMD1, HMD500, PC210, and PC220) via the communication interface 203, audio data, environment information of each terminal, In addition, communication records such as requests from each terminal are recorded in the storage unit 202. The processing unit 201 reads image data recorded in the storage unit 202 in response to a request from each terminal, and transmits the read image data to each terminal.

The storage unit 202 stores communication records such as image data (moving images and still images) received from each terminal, audio data, environment information of each terminal, and requests from each terminal.
The communication interface 203 includes, for example, a WiFi communication circuit, a 3G communication circuit, an LTE communication circuit, a wired LAN circuit, and the like.

Next, the configuration of the PC 210 and the PC 220 will be described. The PC 210 and the PC 220 are, for example, personal computers.
The PC 210 includes a processing unit 211, a storage unit 212, a communication interface 213, an input unit 214, a display unit 215, and a camera 216.
The PC 220 includes a processing unit 221, a storage unit 222, a communication interface 223, an input unit 224, a display unit 225, and a camera 226. Each functional unit of the PC 220 is equivalent to each functional unit of the PC 210.

The processing unit 211 of the PC 210 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like. The processing unit 211 controls the communication interface 213, the input unit 214, the display unit 215, and the camera 216. The processing unit 211 switches the image data stored in the storage unit 212, the image data acquired through the communication interface 213, and the image data captured by the camera 216 through the communication interface 213 as will be described later. Switch based on signal. The processing unit 211 outputs the switched image data to the communication interface 213 and the network 230. The processing unit 211 is based on the avatar image data stored in association with the PC 210 stored in the storage unit 212, the avatar image data of other devices acquired through the communication interface 223, and the image switching signal. Synthesize the switched image data. The processing unit 211 outputs the combined image data to the display unit 215. The processing unit 211 controls to display a cursor image or a line drawing image on the display unit 215 based on the operation of the input unit 214. The processing unit 211 controls to display a cursor image or a line drawing image on the display unit 215 based on the reception information acquired via the communication interface 213.
Hereinafter, an avatar corresponding to each device is referred to as an avatar of the PC 210, for example.

  The storage unit 212 is, for example, a memory or an HDD (hard disk). The storage unit 212 stores an OS (operation system) of the PC 210, an application for making a call with the HMD1, the HMD 500, and the PC 220, avatar image data set in advance for the user of the PC 210, and the like.

  The communication interface 213 is an interface that performs communication such as a wireless LAN, a wired LAN, a 3G line, an LTE line, or the like. The communication interface 213 outputs information received via the network 230 to the processing unit 211. The communication interface 213 transmits information output from the processing unit 211 to the HMD 1, the HMD 500, the PC 220, and the server 200 via the network 230.

  The input unit 214 is, for example, a keyboard, a mouse, a tablet, and an operation pen. The input unit 214 outputs the input information to the processing unit 211. Note that when the display unit 215 has a touch panel function, the display unit 215 may also function as the input unit 214.

The display unit 215 displays an image based on the image data output from the processing unit 211.
As an example, the image captured by the HMD 1, the avatar of the HMD 1, the avatar of the HMD 500, the avatar of the PC 220, and the line drawing image corresponding to the operation of the input unit 214 of the PC 210 are combined and displayed on the display unit 215.

  The camera 216 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) having an image sensor with 5 to 10 million pixels.

  Next, a call procedure performed between the HMD1, the HMD500, the PC 210, and the PC 220 will be described. 9 to 19, each terminal (HMD1, HMD500, PC210, PC220, and server 200) has a communication interface (131, 138, 531, 538, 213, 223, and 203) and a network. A call is made via 230.

First, a procedure for making a call from the HMD 1 to another terminal will be described.
(Step S1) As shown in FIG. 10, the user of the HMD1 operates the touch switch 34 of the HMD1 to make a call with the HMD500, the PC210, and the PC220. Based on the control signal output from the touch switch 34, the processing unit 123 of the HMD 1 performs a call operation indicating that a call is started to the HMD 500, the PC 210, and the PC 220.

(Step S2) The processing unit 123 of the HMD 1 adds the avatar additional information to the avatar of the HMD 1, the avatar of the HMD 500, the avatar of the PC 210, and the avatar of the PC 220, and displays them on the LCD 63.
In the image displayed on the LCD 63 of the HMD 1, the received video and the avatar (including avatar additional information) are displayed on different layers.

Specifically, the avatar generation unit 306 of the processing unit 123 reads the avatar image data of the HMD1, the avatar image data of the HMD500, the avatar image data of the PC 210, and the avatar image data of the PC 220 from the avatar storage unit 307. Next, the avatar generation unit 306 generates image information being called for each terminal of the HMD 500, the PC 210, and the PC 220 based on the reception signal output from the transmission / reception unit 303. Next, the avatar generation unit 306 associates each read avatar image data with the generated image information being called and outputs the image information to the image composition unit 308. Next, the image composition unit 308 displays the image data output from the avatar generation unit 306 on the LCD 63 via the LCD driver 125. That is, the HMD 1 avatar, the HMD 500 avatar and the calling image, the PC 210 avatar and the calling image (calling mark), and the PC 220 avatar and the calling image are displayed on the LCD 63 of the HMD 1.
The displayed avatar is set in advance by the user of the HMD 1 or set in advance by default.
Also in the following process, each part of the process part 123 performs display of an avatar and avatar additional information.

  Next, the processing unit 523 of the HMD 500 extracts call information from the received reception signal. Next, the processing unit 523 reads the avatar image data of the HMD 1 from the storage unit of the processing unit 523 based on the extracted call information. Next, the processing unit 221 generates image information being called based on the extracted call information. Next, the processing unit 523 displays an image based on the read avatar image data of the HMD 1 and the generated image information being called on the LCD 563.

Next, the processing unit 211 of the PC 210 extracts call information from the received reception signal. Next, the processing unit 211 reads the avatar image data of the HMD 1 from the storage unit 212 based on the extracted call information. Next, the processing unit 211 generates image information indicating that a call is in progress based on the extracted call information. Next, the processing unit 211 displays on the display unit 215 an image based on the read avatar image data of the HMD 1 and the generated image information indicating that the call is in progress.
Next, the processing unit 221 of the PC 220 performs processing similar to that of the PC 210 and displays on the display unit 225 an image based on the avatar image data of the HMD 1 read from the storage unit 222 and the generated image information indicating that the call is in progress. To do.

  Next, the processing unit 201 of the server 200 extracts call information from the received reception signal. Next, based on the extracted call information, the processing unit 201 records call information indicating that there is a call from the HMD 1 in the storage unit 202 in association with, for example, time information.

(Step S <b> 3) The processing unit 123 of the HMD 1 controls to display the video (moving image) captured by the camera 64 on the LCD 63.

(Step S <b> 4) The user of the PC 210 confirms the image indicating that the calling is displayed on the display unit 215 and operates the input unit 214 to participate in the call. Based on the information input from the input unit 214, the processing unit 211 of the PC 210 transmits information indicating participation in the call (hereinafter referred to as call participation information) to the HMD1, the HMD 500, the PC 220, and the server 200. Next, the processing unit 211 displays the avatar of the HMD 1 and the avatar of the PC 210 on the display unit 215.

(Step S <b> 5) Based on the call participation information received from the PC 210, the processing unit 123 of the HMD 1 deletes from the LCD 63 image information indicating that the PC 210 is being called.
Next, the processing unit 201 of the server 200 records information indicating that the PC 210 is participating in a call in the storage unit 202 based on the call participation information received from the PC 210.

(Step S6) The user of the HMD 500 confirms the image indicating that the call is being displayed displayed on the LCD 563, operates the touch switch 534, and participates in the call. In this state, the avatar of HMD1 and the avatar of HMD500 are displayed on the LCD 563. Next, the processing unit 523 of the HMD 500 transmits call participation information to the HMD 1, the PC 210, and the server 200 based on the control signal output from the touch switch 534.

(Step S7) The processing unit 523 of the HMD 500 adds and displays the avatar of the PC 210 on the LCD 563.
Next, based on the call participation information received from the HMD 500, the processing unit 123 of the HMD 1 erases from the LCD 63 image information indicating that the HMD 500 is being called.
Next, the processing unit 211 of the PC 210 displays the avatar of the HMD 500 on the display unit 215 based on the call participation information received from the HMD 500.
Next, the processing unit 201 of the server 200 records information indicating that the HMD 500 is participating in a call in the storage unit 202 based on the call participation information received from the HMD 500.
The display of the avatar and the reproduction of the audio signal will be described later.

Next, a procedure for transmitting video captured by the HMD 1 to another terminal will be described.
(Step S8) The user of the HMD 1 touches a predetermined area of the touch switch 34 in order to transmit the video captured by the camera 64 to another terminal. The transmitted video is preset to a low resolution in the initial state. The low resolution is, for example, 320 pixels horizontally, 240 pixels vertically, and 30 frames per second. The reason for transmitting low-resolution video is to some extent, even in the limited network environment where the communication infrastructure is not established and the transfer speed is slow, even if there is blurring of the video due to the movement of the neck of the user of HMD1. This is because an image to be transmitted is transmitted.

(Step S <b> 9) Moving to FIG. 11, the image switching unit 302 of the processing unit 123 of the HMD 1 converts the image captured by the camera 64 into a low resolution based on the control signal input from the touch switch 34. The processing unit 123 transmits the converted low-resolution video signal and HMD1 environment information to the HMD 500, the PC 210, and the server 200 participating in the call.

(Step S10) Each processing unit (523 and 211) of each terminal (HMD500 and PC210) calculates the direction (azimuth) in which the camera 64 of the HMD1 is facing based on the environment information received from the HMD1. Each processing unit displays the calculated azimuth and the video signal received from the HMD 1 on the LCD 563 and the display unit 215.
The processing unit 201 of the server 200 calculates a direction (azimuth) in which the camera 64 of the HMD 1 is facing based on the environment information received from the HMD 1. The processing unit 201 records the calculated orientation and the video data received from the HMD 1 in the storage unit 202.

(Step S11) The processing unit 123 of the terminal HMD1 starts a three-party call with the PC 210 and the HMD 500 based on the call participation information in steps S4 and S6.
Next, the processing unit 523 of the HMD 500 starts a three-party call with the HMD 1 and the PC 210 based on Steps S1 and S4.
Next, the processing unit 211 of the PC 210 starts a three-party call with the HMD 1 and the HMD 500 based on Steps S1 and S6.
Next, the processing unit 201 of the server 200 starts recording the contents (voice data, video data) of the three-party call based on steps S1, S4, and S6.

(Step S12) The user of the HMD 1 performs a predetermined finger operation on the touch switch 34 (third) in order to indicate a specific part of the image captured by the camera 64 to the user participating in the call. Operation). Instead of operating the touch switch 34, the user of the HMD 1 may blink twice, for example. This is detected by the image sensor 72 of the HMD 1 and processed as being equivalent to an operation on the touch switch 34.

(Step S13) The image switching unit 302 of the processing unit 123 of the HMD 1 converts the video signal captured by the camera 64 into a medium resolution based on the control signal input from the touch switch 34.
Next, the processing unit 123 transmits the converted medium resolution video data and environment information to the HMD 500, the PC 210, and the server 200 participating in the call. The medium-resolution video is, for example, 800 pixels in the horizontal direction and 600 pixels in the vertical direction. When transmitting a medium-resolution video, the user of the HMD 1 aims to transmit the video to the user of another terminal, and therefore, the movement of the neck of the user wearing the HMD is small. There is little blur of the camera 64. For this reason, the processing unit 123 switches to the medium resolution video for the purpose of increasing the resolution from the low resolution set in step S9 to the medium resolution and transmitting a video that is easy for other users to view.

If the communication infrastructure environment is poor and it is difficult to secure a communication speed sufficient to transmit at a resolution of 800 pixels in the horizontal direction, 600 pixels in the vertical direction, and 30 frames / second, the processing unit 123 determines that there is little camera shake. In this situation, the frame rate may be decreased, for example, 8 frames / second.
When the angular velocity sensor 134 of the HMD 1 determines that the user's neck rotation is large, the processing unit 123 holds the frame rate at 30 frames / second in order to follow the flow of the image due to the neck rotation. The resolution may be transmitted with 320 pixels in the horizontal direction and 240 pixels in the vertical direction.

(Step S14) Based on the control signal input from the touch switch 34, the processing unit 123 of the HMD 1 displays an image of a light blue cursor 410 as shown in FIG. To do. The images displayed on the LCD (63, 563) and the display unit 215 of each terminal (HMD1, HMD500, and PC210) are the received video and the cursor image displayed on different layers, and To be recorded.

Specifically, the cursor generation unit 305 of the processing unit 123 generates cursor image data and cursor position information based on the control signal output from the input detection unit 301. The cursor generation unit 305 outputs the generated cursor image data and cursor position information to the image composition unit 308. Based on the input cursor image data and the cursor position information, the image composition unit 308 converts the cursor into an image captured by the camera 64 or an image received from another terminal via the communication interface (131, 138). The superimposed images are displayed on the LCD 63.
Also in the following processing, each part of the processing unit 123 displays a cursor image.

  FIG. 21 is a diagram illustrating an example of a configuration of an image displayed on the LCD or the display unit according to the present embodiment. In the example shown in FIG. 21, an image (video or still image) being received or transmitted is displayed on layer 1 (800A). An avatar is displayed on layer 2 (800B). A cursor image by the HMD 1 is displayed on the layer 3 (800C), a line drawing by the HMD 500 is displayed on the layer 4 (800D), and a line drawing by the PC 210 is displayed on the layer 5 (800E). In addition, the images having the layer configuration shown in FIG. 21 are also recorded in the flash memories (122, 522) and the storage units (212, 221 and 202) of each terminal (including server). Note that the example of the layer illustrated in FIG. 21 is an example, and the order of the layers may not be in this order. For example, the layers may be displayed in the order in which they are operated.

FIG. 22 is an example of an image on which a cursor according to the present embodiment is displayed. As shown in FIG. 22, for example, the LCD (63, 563) and the display unit (215, 225) of each terminal (HMD1, HMD500, PC210, and PC220) have a shape indicated by a finger in the image 400a. A cursor 410 is displayed. Note that the shape of the cursor is an example, and other shapes may be used. In the example illustrated in FIG. 22, the user of the HMD 1 is in a state of pointing the price tag 411 with the cursor 410 in the image 400 a.
In addition, since the image transmitted from HMD1 is a moving image, if the user of HMD1 moves the neck, the image | video currently transmitted will also shift. For this reason, the user of the HMD 1 may change the position of the cursor in order to change the position indicated by the cursor. For this reason, a cursor having no trace is displayed on the LCD and the display unit of each device.
Note that the color of the cursor may be determined in advance for each terminal. In this embodiment, the HMD1 cursor is light blue, the PC210 cursor is green, and the HMD500 cursor is blue. In this way, the cursor is color-coded according to each terminal, so that mistakes in communication are less likely to occur.

Next, the processing unit 123 transmits position information for displaying the cursor and cursor color information to the HMD 500, the PC 210, and the server 200.
Next, the processing unit 523 of the HMD 500 displays a light blue cursor image on the LCD 563 based on the received position information for displaying the cursor and the cursor color information. Note that the processing unit 523 may determine the display color of the cursor for each terminal in advance. In this case, the processing unit 523 may acquire only the position information for displaying the cursor and display a light blue cursor because the transmission source is the HMD1.
Next, the processing unit 211 of the PC 210 displays an image of a light blue cursor on the display unit 225 based on the received position information for displaying the cursor and cursor color information.
Next, the processing unit 201 of the server 200 records the position information of the cursor of the HMD 1 in the storage unit 202 based on the received position information for displaying the cursor.

Next, a procedure in which the PC 210 pauses the received video and draws a line drawing on the paused image will be described.
(Step S15) Moving to FIG. 12, the PC 210 user operates the mouse of the input unit 214 so that the cursor image of the PC 210 is displayed on the display unit 215. Next, the user of the PC 210 clicks the mouse to view a still image.

(Step S16) The processing unit 211 of the PC 210 temporarily stops the received video based on the information input from the input unit 214. Next, the processing unit 211 records the paused still image in the storage unit 212 and transmits the still image to the HMD 1, the HMD 500, and the server 200.
The reason for temporarily stopping the video is that it is assumed that the video distributed from the HMD 1 is displayed with a delay on a specific terminal depending on the communication environment. For this reason, in order to superimpose still images, the same still image needs to be displayed on all terminals. By transmitting the paused image, it is possible to observe the same still image on all terminals. In addition, even if each terminal superimposes a line image on a still image, the relationship between the line image and the still image is the same for all terminals, so it is possible to reduce communication errors.
In an environment where it is known that the communication speed is sufficiently high, each terminal may temporarily display the video as a still image in response to the operation of the PC 210 in step S15.
In addition, the pause operation is illustrated as being performed by the PC 210, but it can be performed from the HMD1 that is the transmission source of the video, or from the HMD 500 that has the same components as the HMD1 and is receiving. . For example, when a still image is distributed from the HMD 1, a frame of video displayed on the LCD 63 of the HMD 1 is distributed.

(Step S <b> 17) The processing unit 123 of the HMD 1 receives the still image data transmitted from the PC 210 and records the received still image data in the flash memory 122 of the HMD 1.
Next, the processing unit 523 of the HMD 500 receives the still image data transmitted from the PC 210 and records the received still image data in the flash memory 522 of the HMD 500.
Next, the processing unit 201 of the server 200 receives the still image data transmitted from the PC 210 and records the received still image data in the storage unit 202.

(Step S18) In order to display a line drawing on a still image, for example, the user of the PC 210 draws a line drawing by moving the cursor within the display unit 225 while clicking the mouse. At this time, for example, a cursor shaped like a pencil may be displayed on the display unit 225.
Next, the processing unit 211 of the PC 210 displays a line drawing with a green locus on the display unit 225 based on the information input from the input unit 214. Next, the processing unit 211 sequentially records cursor position information in the storage unit 212. Next, the processing unit 211 sequentially transmits the cursor position information to the HMD 1, the HMD 500, and the server 200.
Note that the color of the line drawing by the PC 210 may be determined in advance for each terminal, or color information may be transmitted from the PC 210 to another terminal.

Specifically, the line drawing generation unit 304 of the processing unit 123 generates line drawing data based on the control signal output from the input detection unit 301. The line drawing generation unit 304 outputs the generated line drawing data to the image composition unit 308. The image composition unit 308 superimposes the input line drawing data on the image data captured by the camera 64 or the image data received from another terminal via the communication interface (131, 138), and superimposes the superimposed image data. The based image is displayed on the LCD 63.
Also in the following processing, each unit of the processing unit 123 displays a line drawing.

(Step S19) The processing unit 123 of the HMD 1 displays a green line drawing on the LCD 63 based on the received cursor position information. Next, the processing unit 123 generates line drawing data based on the cursor position information, and records the generated line drawing data in the flash memory 122.
Next, the processing unit 523 of the HMD 500 displays a green line drawing on the LCD 563 based on the received position information of the cursor. Next, the processing unit 523 generates line drawing data based on the cursor position information, and records the generated line drawing data in the flash memory 522.
Next, the processing unit 201 of the server 200 generates line drawing data based on the received cursor position information, and records the generated line drawing data in the storage unit 202.

FIG. 23 is an example of a line drawing according to the present embodiment. In the example shown in FIG. 23, an arrow 421a and the characters “OK!” 421b are displayed on the image 400b. The arrow 421a and the character 421b are line drawings drawn by the user of the PC 210 using the mouse. 23, for example, in FIG. 22, the user of HMD1 points the price tag 411 to the user of another terminal with the cursor 410, but the user of the PC 210 wants to talk about the destination indicated by the arrow 421a. The price tag 422 is shown.
As described above, when a line image is drawn by being superimposed on a camera image, the camera image is preferably a still image. That is, if the user of the HMD 1 moves his / her head against the intention of the user of the PC 210, the image moves and the position of the subject pointed to by the line drawing is relatively shifted on the screen. However, if a still image is displayed, the image and the line drawing do not relatively shift even if the line drawing is superimposed on the image. For this reason, it is not necessary to be stationary so that the user of the HMD 1 does not move the neck, and an easy-to-use system can be provided.
Note that the line image and the still image are recorded in different layers, and it is possible to delete only the line image at any time.
Further, not only the PC 210 but all terminals are capable of moving a still image at the same time as drawing a line drawing.

Next, a procedure for changing the grid superimposed on the image will be described.
(Step S20) Returning to FIG. 12, the user of the PC 210 uses the input unit 214 to select the type of grid. Based on the signal input from the input unit 214, the processing unit 211 of the PC 210 transmits information indicating the type of the selected grid to the HMD1, the HMD5, and the server 200. The reason for displaying the grid is, for example, that the user of the PC 210 explains a specific area in the image.
FIG. 24 is a diagram illustrating an example of a grid according to the present embodiment. FIG. 25 is a diagram illustrating another example of the grid according to the present embodiment.
The example illustrated in FIG. 24 is an example in which a grid-like grid 431 including vertical lines and horizontal lines is displayed in the image 400c. The example illustrated in FIG. 25 is an example in which a grid 441 including vertical lines, horizontal lines, and concentric circles is displayed in the image 400d. 24 and 25, the grid is indicated by a solid line, but the grid may be a dotted line, a broken line, or the like.
As described above, when the user of the PC 210 changes the grid, the reason for transmitting the changed grid information to another terminal is to prevent a communication error if the grid is different for each user terminal.
Note that the grids shown in FIGS. 24 and 25 are only examples, and other grids may be displayed. In addition, the grid displayed on the LCD or display unit of each terminal is displayed on a layer different from that of the moving image or the still image.

(Step S21) The processing unit 211 of the PC 210 changes the display to the selected grid.
Next, the processing unit 123 of the HMD 1 changes the display to the selected grid based on the received signal.
Next, the processing unit 523 of the HMD 5 changes the display to the selected grid based on the received signal.
Next, the processing unit 201 of the server 200 records the grid change based on the received signal.

Next, a processing procedure when the PC 220 participates in a call from the middle will be described.
(Step S <b> 22) Moving to FIG. 13, the user of the PC 220 confirms the image indicating that the calling is displayed on the display unit 225 and operates the input unit 224 to participate in the call. The processing unit 221 of the PC 220 transmits call participation information to the server 200 based on the information input from the input unit 224.
Next, the processing unit 221 controls the display unit 225 to display the PC 220 avatar.

(Step S23) Based on the call participation information received from the PC 220, the processing unit 201 of the server 200 performs a call participation process of the PC 220 on the HMD1, the HMD 500, the PC 210, and the PC 220. Specifically, the processing unit 201 transmits the same image data as the images (including line drawings, cursors, and grids) displayed on the other LCD and the display unit to the PC 220. The processing unit 221 transmits information indicating that the PC 220 has joined the call and an instruction to display the PC 220 avatar to the HMD 1, the HMD 500, and the PC 210. The processing unit 201 transmits an instruction to display the avatars of the HMD 1, the HMD 500, and the PC 210 to the PC 220.
Next, the processing unit 201 records information indicating that the PC 220 has joined the call in the storage unit 202.

(Step S <b> 24) The processing unit 221 of the PC 220 displays an image based on the image data received from the server 200 on the display unit 225.

(Step S <b> 25) The processing unit 123 of the HMD 1 additionally displays the avatar of the PC 220 on the LCD 63 based on the instruction to display the avatar of the PC 220 received from the server 200, and an image indicating that the PC 220 is being called Erase information.
Next, the processing unit 523 of the HMD 500 adds and displays the PC 220 avatar on the LCD 563 based on the instruction to display the PC 220 avatar received from the server 200.
Next, the processing unit 211 of the PC 210 adds and displays the avatar of the PC 220 on the display unit 215 based on the instruction to display the avatar of the PC 220 received from the server 200.
Next, the processing unit 221 of the PC 220 adds and displays the avatars of HMD1, HMD500, and PC 210 on the display unit 225 based on the instruction to display the avatars of HMD1, HMD500, and PC 210 received from the server 200. .
The display of the avatar displayed on the LCD and the display unit of each terminal here is again from the right from the terminal located east according to the east longitude from the right according to the location of each terminal with the participation of the PC 220. It may be displayed again.

Next, a procedure in which different terminals display a cursor and a line drawing on a common image will be described.
(Step S26) The user of the HMD 500 performs a predetermined finger operation on the touch switch 534 to draw a line drawing. The processing unit 523 of the HMD 500 transmits an instruction to display a line drawing by the HMD 500 to the HMD 1, the PC 210, the PC 220, and the server 200 based on the control signal input from the touch switch 534.

(Step S27) The processing units (123, 211, and 221) of each terminal (HMD1, PC210, and PC220) are red on the LCD 63 and the display units (215, 225) based on the received signal received from the HMD500. Display the line drawing. The processing unit 523 of the HMD 500 displays a red line drawing on the LCD 563 based on the control signal input from the touch switch 534. That is, it is displayed in a color different from the line drawing (green) of the PC 210.
Next, the processing unit of each terminal records the position information of the cursor in the flash memory 122 and the storage unit (212, 222, 202) sequentially based on the received signal received from the HMD 500. Based on the control signal input from the touch switch 534, the processing unit 523 of the HDM 500 sequentially records the cursor position information in the flash memory 522 and the storage units (212, 222, 202).
Alternatively, each processing unit (123, 523, 211, 221 and 201) records the image data of the line drawing drawn by the user of the HMD 500 in the flash memory (122, 522) and the storage unit (212, 222, 202). .
In addition, on the LCD (63, 563) and the display unit (215, 225) of each terminal, the user of the HMD 500 depicted the image temporarily stopped by the PC 210 by moving the finger with the touch switch 534 in step S16. The line drawing is displayed superimposed.
The line drawing by the HMD 500 is stored in each terminal and server as layer information different from the transmitted image and the line drawing drawn by other terminals.

(Step S28) The user of the HMD 1 performs a predetermined finger operation on the touch switch 34 in order to draw a line drawing. The processing unit 123 of the HMD 1 transmits an instruction to display a line drawing by the HMD 1 to the HMD 500, the PC 210, the PC 220, and the server 200.

(Step S29) The processing units (523, 211, and 221) of each terminal display a blue line drawing on the LCD 563 and the display units (215, 225) based on the reception signal received from the HMD1. The processing unit 123 of the HMD 1 displays a blue line drawing on the LCD 63 based on the control signal input from the touch switch 34.
Next, the processing units (523, 211, 221 and 201) of each terminal sequentially store the cursor position information on the flash memory 522 and the storage units (212, 222, 202) based on the received signals received from the HMD1. ). The processing unit 123 of the HMD 1 sequentially records cursor position information in the flash memory 122 based on the control signal input from the touch switch 34. Alternatively, the processing units (123, 523, 211, 221 and 201) record the line drawing image data drawn by the user of the HMD 1 in the flash memory (122, 522) and the storage unit (212, 222, 202).
Note that the LCD (63, 563) and the display unit (215, 225) of each terminal have a green line drawing drawn by the PC 210 in Step S18 on the image paused by the PC 210 in Step S16, and in Step S26. The red line drawing depicted by the HMD 500 is displayed superimposed.
The reason why the line drawing by HMD1 is blue (R: 0, G: 0, B: 255) is that the cursor is displayed in light blue (R: 100, G: 100, B: 255). This is to make it easier to visually understand which user is investigating.

Next, a description will be given of a processing procedure in a case where a video of another terminal is requested according to a request of a user of another terminal, and the video is captured as a still image and image information is shared.
(Step S30) Moving to FIG. 14, for example, the user of the HMD 500 requests the transmission of the video imaged by the camera 64 of the HMD 1 via the call microphone of the HMD 500.
Next, in response to a request from the user of the HMD 500, the user of the HMD 1 touches a predetermined area of the touch switch 34 in order to transmit the video captured by the camera 64 to another terminal at a low resolution. In the initial setting, the resolution at the time of video transmission is set to a low resolution. However, the resolution can be changed by operating the touch switch 34.

(Step S <b> 31) The processing unit 123 of the HMD 1 deletes the image that has been displayed up to Step S <b> 29 and displays an image captured by the camera 64 on the LCD 63.

(Step S32) Based on the control signal output from the touch switch 34, the processing unit 123 of the HMD 1 converts the video captured by the camera 64 into a low resolution. The processing unit 123 transmits the converted low-resolution video data and the environment information of the HMD 1 to the HMD 500, the PC 210, the PC 220, and the server 200 participating in the call.

(Step S33) The processing units (523, 211, and 221) of each terminal (HMD500, PC210, and PC220) are displayed on the LCD 563 and the display units (215, 225) based on the received signal received from the HMD1. Erase the displayed image and line drawing.

(Step S34) The processing units (523, 211, and 221) of each terminal (HMD500, PC210, and PC220) receive the LCD 563 and the display units (215, 225) from the HMD1 based on the received signal received from the HMD1. The video based on the low-resolution video data is displayed.
Next, the processing unit 201 of the server 200 records low-resolution video data received from the HMD 1 in the storage unit 202 based on the reception signal received from the HMD 1.

(Step S35) The user of the HMD 500 touches a predetermined area of the touch switch 534 to make a request for transmitting a high-resolution still image to the HMD 1 to view a detailed image. Next, the processing unit 523 of the HMD 500 transmits a request for transmitting a high-resolution still image to the HMD 1 based on the control signal input from the touch switch 534.

(Step S <b> 36) The processing unit 123 of the HMD 1 displays information indicating a request for transmitting a high-resolution still image received from the HMD 500 on the LCD 63. Note that a high-resolution still image is, for example, an image with 2500 pixels in the horizontal direction and 1875 pixels in the vertical direction.

(Step S37) Based on the received signal from the HMD 500, for example, when the front of the HMD 1 is dark, the processing unit 123 of the HMD 1 turns on the front light 68 for illumination.

(Step S38) Moving to FIG. 15, the processing unit 123 of the HMD 1 combines high-resolution still image data automatically captured by the camera 64 based on the received signal from the HMD 500 with the environment information, the HMD 500, the PC 210, The data is transmitted to the PC 220 and the server 200.

(Step S39) The processing units (523, 211, and 221) of each terminal (HMD500, PC210, and PC220) receive the LCD 563 and the display units (215, 225) from the HMD1 based on the received signal received from the HMD1. The high-resolution still image and information based on the environment information are displayed.
Next, the processing unit 201 of the server 200 records the high-resolution still image received from the HMD 1 in the storage unit 202 based on the reception signal received from the HMD 1.
In this way, high-resolution video that is difficult to distribute as a moving image is photographed with high definition in response to an operation from another terminal, and is distributed to each terminal as a still image. This makes it possible to analyze the environment in which the HMD 1 is used more easily than others.

(Step S40) The user of the PC 210 operates the input unit 214 to enlarge and display the still image received from the HMD 1 on the display unit 215. Based on the control signal input from the input unit 214, the processing unit 211 of the PC 210 transmits an instruction to enlarge the display image to the HMD1, the HMD 500, the PC 220, and the server 200.

(Step S41) The processing units (123, 523, and 221) of each terminal (HMD1, HMD500, and PC220), based on the received signal received from the PC210, the images on the LCD (63,563) and the display unit 225 Is magnified.
Next, the processing unit 201 of the server 200 records the enlarged image data and information indicating that the image has been enlarged in the storage unit 202 based on the received signal received from the PC 210.
As described above, since high-resolution still image data is sent from the HMD 1, an image that does not deteriorate in image quality even when enlarged is displayed on each terminal. The high-resolution still image can be automatically enlarged by simply operating a single terminal, and detailed communication can be performed on this image.

Next, processing for transmitting a medium-resolution video will be described.
(Step S <b> 42) The user of the HMD 500 touches a predetermined area of the touch switch 534 in order to request the HMD 1 to transmit a medium resolution video. Next, the processing unit 523 of the HMD 500 transmits a request for transmitting a medium resolution video to the HMD 1 based on the control signal input from the touch switch 534.

In this case, the LCD 63 of the HMD 1 displays that a medium resolution image is desired. Alternatively, the request may be transmitted by voice call. When responding to this request, the user of the HMD 1 needs to prevent the image from blurring. That is, it can be seen that the user of the HMD 1 is required to stably point the camera at the subject so as not to move the neck much.
In this way, the user of each terminal can know the level of video request from each terminal (still image or moving image). For this reason, since the user of each terminal can know how to point the camera 64 toward the subject, the system is easy to use.

As described above, the medium-resolution video can be transmitted only in a state where the user of the HMD 1 can point the camera 64 toward the subject relatively stably. For this reason, when it is determined that transmission of a medium-resolution video is not preferable, the user of the HMD 1 may refuse this by voice or the like. In this way, by obtaining the consent of the HMD1 user, it is possible to prevent the image of the environment where the HMD1 user is present from being transmitted as it is. That is, the privacy of the user of HMD1 can be protected by obtaining the consent of the user of HMD1.
Even if the HMD1 user approves the transmission of the medium-resolution video, the camera 64 may be severely blurred while the user is actually moving. You may make it set the resolution of the image | video transmitted by the angular velocity sensor 134. FIG. For example, when the output of the angular velocity sensor 134 is larger than a predetermined value, the processing unit 123 of the HMD 1 may perform transmission at a low resolution and a high frame rate even when a request for medium resolution is received.

(Step S <b> 43) The processing unit 123 of the HMD 1 displays information indicating a request for transmitting the medium resolution video received from the HMD 500 on the LCD 63. Next, the user of the HMD 1 confirms the display on the LCD 63 and touches a predetermined area of the touch switch 34 in order to accept the request from the HMD 500.

(Step S44) The processing unit 123 of the HMD 1 transmits the medium resolution video data captured by the camera 64 to the HMD 500, the PC 210, the PC 220, and the server 200 together with the environment information.

Next, a procedure in which one terminal adds information to an image from another terminal and distributes the image to another terminal in a completed state will be described.
(Step S45) The processing units (523, 211, and 221) of each terminal (HMD500, PC210, and PC220) receive from the HMD1 on the LCD 563 and the display units (215, 225) based on the reception signal received from the HMD1. Displays the medium resolution video.
Next, the processing unit 201 of the server 200 records the medium resolution video data received from the HMD 1 in the storage unit 202 based on the reception signal received from the HMD 1.

(Step S46) Moving to FIG. 16, the user of the HMD 500 touches a predetermined area of the touch switch 34 in order to record a still image of the video received from the HMD 1.

(Step S <b> 47) The processing unit 523 of the HMD 500 records the image data of the still image received from the HMD 1 in the flash memory 522 based on the control signal input from the touch switch 534.

(Step S48) The user of the HMD 500 touches a predetermined area of the touch switch 534 in order to superimpose and display a line image on a still image. Next, the user of the HMD 500 moves a finger on the touch switch 534 to draw a line drawing. The processing unit 523 superimposes the drawn line image on the still image and displays it on the LCD 563. The processing unit 523 records image data in which a line image is superimposed on a still image in the flash memory 522.

(Step S49) The user of the HMD 500 touches a predetermined area of the touch switch 534 in order to transmit the image data in which the line image is superimposed on the still image recorded in the flash memory 522 to the HMD 1, the PC 210, the PC 220, and the server 200. To do.

(Step S50) Based on the control signal input from the touch switch 534, the processing unit 523 of the HMD 500 converts the image data obtained by superimposing the line image on the still image recorded in the flash memory 522 to the HMD1, the PC 210, the PC 220, and the server. 200.

(Step S51) Based on the received signal received from the received HMD 500, the processing units (123, 211, and 221) of each terminal (HMD1, PC 210, and PC 220) send the HMD 500 to the LCD 63 and the display units (215, 225). An image based on the image data in which the line drawing is superimposed on the still image received from is displayed.
Next, the processing unit 201 of the server 200 records, in the storage unit 202, image data in which a line drawing is superimposed on the still image received from the HMD 500 based on the received signal received from the received HMD 500.

Next, processing for switching a video from another terminal, and processing for transmitting a low-resolution video by default in that case will be described.
(Step S <b> 52) The user of the HMD 1 touches a predetermined area of the touch switch 34 in order to view the video captured by the camera 564 of the HMD 500. Next, the processing unit 123 of the HMD 1 transmits a video browsing request to the HMD 500.

(Step S <b> 53) The processing unit 523 of the HMD 500 displays information indicating a request to transmit the video received from the HMD 1 on the LCD 563. Next, the user of the HMD 500 confirms the display on the LCD 563 and touches a predetermined area of the touch switch 534 in order to accept the request from the HMD 1.

(Step S54) Moving to FIG. 17, the processing unit 523 of the HMD 500 converts the video captured by the camera 564 into a low resolution. The processing unit 523 transmits the converted low-resolution video signal and the environment information of the HMD 500 to the HMD 1, the PC 210, the PC 220, and the server 200. That is, when there is a transmission request for an image captured by a camera from another terminal, the processing unit of each terminal first transmits a low-resolution video. Thereafter, the resolution is switched according to the request of the user of each terminal.

(Step S55) Based on the received signals from the received HMD 500, the processing units (123, 211, and 221) of the terminals (HMD1, PC 210, and PC 220) transfer the HMD 500 to the LCD 63 and the display units (215, 225). The video based on the low resolution video data received from is displayed. Next, the processing units (123, 211, and 221) of each terminal (HMD1, PC210, and PC220) calculate the imaged direction based on the environment information received from the HMD500, based on the received signal from the HMD500. The calculated imaging direction is displayed on the LCD 63 and the display unit (215, 225).
Next, the processing unit 201 of the server 200 records low-resolution video data and environment information in the still image received from the HMD 500 in the storage unit 202 based on the received signal from the HMD 500.

Next, a procedure for terminating a call by a specific terminal will be described.
(Step S56) The user of the PC 210 operates the input unit 214 to end the connection. Next, based on the control signal input from the input unit 214, the processing unit 211 of the PC 210 transmits information indicating the end of connection to the HMD1, the HMD 500, the PC 220, and the server 200.

(Step S57) The processing units (123, 523, and 221) of each terminal (HMD1, HMD500, and PC220) are displayed on the LCD (63, 563) and the display unit 225 based on the received signal received from the PC210. Delete the avatar of the PC 210 that was on.
Next, the processing unit 201 of the server 200 records the connection end of the PC 210 in the storage unit 202 based on the received signal received from the PC 210.

(Step S58) The user of the HMD 500 operates the touch switch 534 to end the connection. Next, the processing unit 523 of the HMD 500 transmits information indicating the end of connection to the HMD 1, the HMD 500, the PC 220, and the server 200 based on the control signal input from the input unit 214.

(Step S59) The processing units (123 and 523) of each terminal (HMD1 and HMD500) erase the avatar of the HMD 500 displayed on the LCD 63 and the display unit 225 based on the received signal received from the HMD 500.
Next, the processing unit 201 of the server 200 records the connection end of the HMD 500 in the storage unit 202 based on the received signal received from the HMD 500.

Next, a description will be given of a procedure of processing in which the user of each terminal calls and uses an image (video, still image) recorded on the server 200.
(Step S60) In order to select an image recorded in the server 200, the user of the PC 220 operates the input unit 224, for example, presses a predetermined key on the keyboard.

(Step S61) Referring to FIG. 18, it is assumed that a list of thumbnails, which are reduced images of images recorded in the server 200, is displayed on the display unit 225 of the PC 220. The user of the PC 220 operates the input unit 224, for example, presses a predetermined key on the keyboard in order to select an image recorded in the server 200. Note that the thumbnail list may be read from the server 200 and displayed by the user of the PC 220 by operating predetermined keys on the keyboard and the mouse.

(Step S <b> 62) The processing unit 221 of the PC 220 transmits information indicating the selected image to the server 200 based on the control signal input from the input unit 224.

(Step S <b> 63) Based on the received signal received from the PC 220, the processing unit 201 of the server 200 reads out the selected video and the environment information recorded in the storage unit 202 in association with the video. The processing unit 201 calculates an orientation based on the read environment information. The processing unit 201 transmits the read video data and the calculated azimuth information to the HMD 1 and the PC 220.

(Step S64) The processing unit 201 of the server 200 records in the storage unit 202 that the video data has been transmitted to the HMD 1 and the PC 220.
Next, the processing units (123 and 221) of the HMD 1 and the PC 220 display images based on the video data and the orientation information received from the server 200 on the LCD 63 and the display unit 225, respectively.

(Step S <b> 65) The user of the HMD 1 touches a predetermined area of the touch switch 34 (first operation) in order to draw a line drawing on the video transmitted from the server 200. Next, the processing unit 123 of the HMD 1 transmits a request for temporarily stopping the transmitted video to the server 200 based on the control signal input from the touch switch 34. That is, even when a line drawing is drawn on the video read from the server 200, the video is paused to obtain a still image. And each terminal can superimpose a line drawing on this still picture.

(Step S66) The processing unit 201 of the server 200 temporarily stops the video data being transmitted based on the received signal received from the HMD1. Next, the processing unit 201 transmits information indicating that the video data being transmitted has been temporarily stopped to the HMD 1 and the PC 220.

(Step S67) The processing units (123 and 221) of the HMD 1 and the PC 220 record the paused still images in the flash memory 122 and the storage unit 222, respectively, based on the received signal received from the server 200. . Note that the processing unit 123 of the HMD 1 may record the paused still image in the flash memory 122 based on the user touching the touch switch 34 in step S65.

(Step S <b> 68) Moving to FIG. 19, the user of the HMD 1 touches a predetermined area of the touch switch 34 in order to display a line image superimposed on a still image. Next, the user of the HMD 1 moves a finger on the touch switch 34 to draw a line drawing. Next, the processing unit 123 of the HMD 1 sequentially transmits the cursor position to the PC 200 and the server 200 in accordance with the movement of the finger of the touch switch 34.

(Step S69) Each processing unit (123 and 221) of the HMD 1 and the PC 220 superimposes the drawn blue line image on the still image and displays it on the LCD 63 and the display unit 225, respectively. Next, each processing unit (123 and 221) records image data in which a blue line image is superimposed on a still image in the flash memory 122 and the storage unit 222, respectively.

(Step S <b> 70) The user of the HMD 1 touches a predetermined area of the touch switch 34 in order to cancel the temporary stop and resume the transmission of the video. Next, the processing unit 123 of the HMD 1 cancels the temporary stop and transmits a request to resume transmission of video to the server 200.

(Step S <b> 71) The processing unit 201 of the server 200 cancels the pause based on the received signal received from the HMD 1 and resumes transmission of video data. Next, the processing unit 201 transmits to the HMD 1 and the PC 220 an instruction to delete the line drawing displayed in step S69.

(Step S <b> 72) Each processing unit (123 and 221) of the HMD 1 and the PC 220 deletes the blue line drawing displayed on the LCD 63 and the display unit 225 based on the received signal received from the server 200. Next, the respective processing units (123 and 221) of the HMD 1 and the PC 220 are based on the video data and the orientation information whose transmission is resumed to the LCD 63 and the display unit 225, respectively, based on the received signal received from the server 200. Display an image.

Next, a procedure for superimposing a line drawing on the video captured by the HMD 1 itself according to an instruction of the user of the HMD 1 will be described.
(Step S73) The user of the HMD 1 moves his / her finger at a predetermined position on the touch switch 34 in order to draw and transmit a line drawing. In this case, in order to transmit the line drawing, the processing unit 123 of the HMD 1 switches the video captured by the camera 64 from low resolution to medium resolution.

(Step S <b> 74) Moving to FIG. 20, the processing unit 123 of the HMD 1 converts the moving image captured by the camera 64 into a medium-resolution video, and transmits the converted medium-resolution video data to the PC 220 and the server 200.

(Step S75) The user of the HMD 1 touches a predetermined position of the touch switch 34 in order to pause the video being transmitted. Next, the user of the HMD 1 moves his / her finger on the touch switch 34 in order to draw a line drawing on the paused still image.
Next, based on the touch switch 34 being touched, the processing unit 123 of the HMD 1 transmits an instruction to temporarily stop the video being transmitted to the PC 220 and the server 200.
Next, the processing unit 123 sequentially transmits information indicating the position of the cursor to the PC 220 and the server 200 according to the movement of the finger on the touch switch 34.

(Step S76) The processing unit 123 of the HMD 1 superimposes and displays a blue line image on the still image according to the movement of the finger on the touch switch 34.
Next, the processing unit 221 of the PC 220 temporarily stops the video based on the received signal received from the HMD 1. Next, based on the received signal received from the HMD 1, the processing unit 221 superimposes a blue line drawing on the paused still image and displays it on the display unit 225.
Next, the processing unit 201 of the server 200 temporarily stops the video data based on the received signal received from the HMD 1 and records the still image when the video data is stopped in the storage unit 202. Next, the processing unit 201 records a blue line drawing in the storage unit 202 based on the reception signal received from the HMD 1.

Next, a process for ending calls with all call partners using the HMD 1 that started the call in step S1 will be described.
(Step S77) The user of the HMD 1 touches a predetermined position of the touch switch 34 in order to complete a call with all terminals. Next, based on the control signal input from the touch switch 34, the processing unit 123 of the HMD 1 transmits an instruction to complete the call to the PC 220 and the server 200 participating in the call in step S77. Next, the processing unit 123 ends the call with the PC 220 and the server 200.

(Step S78) The processing unit 221 of the PC 220 ends the call processing from the HMD 1 based on the received signal received from the HMD 1.
Next, the processing unit 201 of the server 200 records information indicating that the connection between the HMD 1 and the PC 220 is completed in the storage unit 202 based on the received signal received from the HMD 1.

  In the examples shown in FIGS. 10 to 20, the order of transmission of image data from each terminal, the drawing of line drawings, and the like is an example, and is performed in the order according to the instruction of the user of each terminal.

Next, display of an avatar will be described.
FIG. 26 is a flowchart of an avatar display procedure according to this embodiment.

(Step S101) The user of the HMD 1 operates the touch switch 34 to call the partner terminal.

(Step S102) As shown in FIG. 27, the processing unit 123 of the HMD 1 displays an avatar image and a calling image on the calling party on the LCD 63.
FIG. 27 is a diagram for explaining an example of an image displayed while calling the calling party according to the present embodiment. In FIG. 27, an image 400 includes images 401 to 403 and images 451 to 453. Images 401 to 403 are each avatar of the other party. The images 451 to 453 are images indicating that a call is in progress, and in the example shown in FIG. In addition, since the position information of each call partner cannot be acquired during the call, the processing unit 123 arranges and displays the display positions of the avatars of the call opponents in a predetermined order. May be. In the example illustrated in FIG. 27, the avatars (401 to 403) and the images being called (451 to 453) are displayed on the upper part of the image 400, but are not limited thereto. For example, in accordance with the image displayed on the LCD 63, the image 400 may be displayed side by side, displayed vertically at the right position, and displayed vertically at the left position. The avatar and the image being called may be either color or black and white.
In the example shown in FIG. 27, an example of displaying the calling image (451 to 453) on the right side of the avatar (401 to 403) is shown, but the calling image (451 to 453) , Upper, lower, etc. may be displayed. Furthermore, the image (451 to 453) being called may be a still image or an animation.
Further, in the example shown in FIG. 27, the example in which the sender avatar is not displayed is shown, but the sender avatar may also be displayed.

(Step S103) The processing unit 123 detects that the other party has joined the call in response to a call from the HMD1. If it is detected that the user has joined the call (step S103; Yes), the process proceeds to step S104. If it is not possible to detect participation in the call (step S103; No), the process proceeds to step S105.

(Step S104) When it is detected that the user has joined the call, the processing unit 123 deletes the calling image corresponding to the partner terminal whose participation has been detected as shown in FIG.
FIG. 28 is a view showing an example of a display when the call partner according to the present embodiment participates in a call. In FIG. 28, an image 400a includes avatars 401 to 403 and images 452 and 453 being called. In the example shown in FIG. 28, a call partner corresponding to the avatar 401 participates in the call, and a call partner corresponding to the avatars 402 and 403 does not participate in the call. As shown in FIG. 28, since the calling party corresponding to the avatar 401 has joined the call, the calling image 451 corresponding to the avatar 401 is deleted from the image 400a.

(Step S105) If it is not possible to detect participation in a call, or after step S104 is completed, the processing unit 123 determines whether or not at least one of the called parties has joined the call. If no one is participating in the call (step S105; No), the process returns to step S103. When at least one person is participating in the call (step S105; Yes), the process proceeds to step S106.

(Step S <b> 106) The processing unit 123 acquires environment information of a call partner participating in the call. The processing unit 123 extracts position information from the acquired environment information.

(Step S107) Based on the extracted position information, the processing unit 123 corrects the arrangement position of the calling partner's avatar that is calling one-dimensionally. For example, in the example shown in FIG. 27, the one-dimensional correction is to change the order of arrangement of the avatars 401 to 403 displayed from right to left in the initial state based on the position information. That is.

FIG. 29 is a diagram illustrating an example of a display after rearranging the avatars according to the present embodiment. In FIG. 29, an image 400b includes avatars 401 to 403 and an additional information icon image 461. In the example shown in FIG. 29, since all the avatars 401 to 403 that have been called participate in the call, all the images (451 to 453) that are being called that are displayed in FIG. 27 are deleted. In the example shown in FIG. 29, the call partner position information corresponding to the avatars 402 and 403 can be acquired, and the call partner position information corresponding to the avatar 401 cannot be acquired. In this case, since the longitude in the position information corresponding to the avatar 402 is east relative to the HMD1 relative to the longitude in the position information of the avatar 403, the avatar 402 is arranged on the rightmost side, and the avatar 403 Is placed in the middle. Further, since the position information of the avatar 401 could not be acquired, an image 461 is added to the avatar 401 as additional information and displayed. In the example shown in FIG. 29, since the position information is unknown, “?” Is displayed as the additional information icon 461.
In addition, when there are a plurality of avatars whose position information is unknown, they are displayed in a predetermined order, for example, from the left side. In addition, as in the example illustrated in FIG. 29, the avatars for which position information can be acquired may be arranged in order from the right, and the avatars for which position information cannot be acquired may be arranged next.
In the example shown in FIG. 29, the user's avatar is not displayed on the LCD 63 of the HMD 1 used by the user of the HMD 1. For this reason, since the position of the displayed avatar changes depending on the relative position between the position of the user of the HMD 1 and the communication partner, the display displayed on the LCD or display unit of the terminal used by all users is completely different. It has become.

(Step S108) If the location information of the call partner cannot be acquired in step S107, the processing unit 123 displays the avatars on the LCD 63 in the order of participation in the call. In this case, for example, it may be displayed from the right in the order of participation in the call.

(Step S109) The audio adjustment unit 310 of the processing unit 123 adjusts the left / right balance of the audio signal output to the stereo earphone 100 based on the location information of each call partner corrected in step S107. The adjustment of the audio signal will be described later.

(Step S110) The voice adjustment unit 310 of the processing unit 123 detects the level of each voice signal from the speaking partner participating in the call. When the detected voice level is larger than a predetermined threshold, that is, when it is determined that the voice information is not the ambient noise but the voice of the terminal user's speech, the voice adjustment unit 310 uses the avatar's mouth. Display moving animation. Note that the avatar generation unit 306 of the processing unit 123 performs the display of the animation.

(Step S <b> 111) When the terminal of the other party has a camera like the HMD 500 described above and can transmit an image such as the eyes to the HMD 1, the processing unit 123 animates the facial expression of the avatar corresponding to the other party. Use to change.
For example, if the call partner's eyes are laughing, the avatar's facial expression corresponding to the call partner is also laughed, and if the call partner's eyes are troubled, the avatar corresponding to the call partner is also displayed to show a confused expression. You may make it control so. When the other party's terminal is the HMD 500, the movement of the eyeball is detected by photographing with the image sensor 72, and the detected image or avatar expression information based on the detected result is transmitted to another terminal. The processing unit 123 of the HMD 1 may be linked to the movement of the avatar's eyes corresponding to the HMD 500 based on the reception information received from the HMD 500.

FIG. 30 is a diagram of an example in which the mouth and line of sight of the avatar according to the present embodiment are displayed by animation. In FIG. 30, an image 400c includes avatars 401, 402, and 403a and an additional information icon image 461. The example shown in FIG. 30 is different from the avatar 403 shown in FIG. In the example shown in FIG. 30, since it is determined that the audio signal received from the speaking partner corresponding to the avatar 403a is larger than a predetermined level, the eyes and mouth are displayed as animations.
As shown in FIG. 30, it is easier to understand that the voice is from the terminal corresponding to the avatar by displaying the mouth of the avatar of the other party who is speaking as an animation. Further, when the line of sight is also displayed as an animation, a change in the expression of the avatar can be shown, so that it becomes easy for the user of the HMD 1 to acquire information. For example, based on information indicating these facial expressions, the user of the HMD 1 can acquire information such as whether the other party understands his / her story and what he / she is interested in.

(Step S112) Expression information based on an image captured by the image sensor 72 of the HMD 1 or image data captured by the image sensor 72 is sent to another terminal.

(Step S113) The processing unit 123 determines whether or not the other party participating in the call has ended the call. When it is determined that the other party has not ended the call (step S113; No), the process returns to step S103, and steps S103 to S113 are repeated. When it is determined that the other party has finished the call (step S113; Yes), the process proceeds to step S114.

(Step S114) When it is determined that the other party has finished the call, the processing unit 123 deletes the avatar corresponding to the other party who has finished the call and the image indicating the additional information from the display on the LCD 63. Next, the process part 123 corrects arrangement | positioning of the avatar corresponding to the other party who remains in the call.
FIG. 31 is a diagram for explaining an example of a display when one call partner according to the present embodiment is missed. In FIG. 31, an image 400d includes avatars 402 and 403. In the example illustrated in FIG. 31, the image after the avatar 401 is deleted and the arrangement of the avatars 402 and 403 corresponding to the call partner participating in the call is corrected with respect to FIG. 30. In FIG. 31 as well, the avatars 402 and 403 are arranged from the right in order from the avatar corresponding to the call partner located east relative to the position of the HMD1.
Furthermore, as shown in FIG. 31, a case will be described in which the call partner corresponding to the avatar 401 once participates in the call again during a period in which this session continues after the call partner has left the call. In this case, the processing unit 123 adds the avatar 401 corresponding to the call partner who participated in the call again, and rearranges and displays the three avatars.
That is, the processing unit 123 adds or deletes the avatar based on the participation state (whether the call has been joined or has left the call).

(Step S115) Returning to FIG. 26, the processing unit 123 determines whether or not the other party remains in the call. When the other party remains in the call (step S115; Yes), the process returns to step S103, and steps S103 to S115 are repeated. If the other party does not remain in the call (step S115: No), the process proceeds to step S116.

(Step S116). If the other party does not remain in the call, since all the other parties have finished the call, the processing unit 123 displays a message indicating that the call has ended on the LCD 63 by text display, icon display, or the like. Alternatively, the processing unit 123 may reproduce the end of the call by voice.

  As described above, according to the present embodiment, the avatar is displayed at a position based on the position information (latitude, longitude) of the terminal of the other party, and the balance of the left and right voices is switched according to the position of the avatar. Since a sound corresponding to the position of the sound is heard, good communication can be performed.

Next, an example of the left / right volume balance adjustment performed by the audio adjustment unit 310 will be described with reference to FIGS. 8 and 32. FIG.
FIG. 32 is a flowchart of left and right volume balance adjustment performed by the audio adjustment unit according to the present embodiment.

(Step S <b> 201) The audio adjustment unit 310 of the processing unit 123 acquires audio data from the audio data output from the encoder 129 and the reception signal output from the transmission / reception unit 303.

(Step S202) The avatar generation unit 306 acquires the position information of the HMD 1 and other terminals output from the position calculation unit 309. The avatar generation unit 306 outputs the position information of each avatar to the voice adjustment unit 310 based on the acquired HMD1 and position information of other terminals.

(Step S203) The voice adjustment unit 310 is connected to the audio connector 26 based on the position information of each avatar output from the avatar generation unit 306, based on the acquired voice data of the HMD1 and the voice data of each terminal, based on the position information of each avatar. The left and right sounds are reproduced by the stereo earphone 100 being used.

Here, a specific example will be described in the case where the display unit 60 is arranged on the right eye side of the user, the first speaker 103 corresponds to the right ear, and the second speaker 104 corresponds to the left ear. . If the number of users participating in the call is three as shown in FIG. 9 except for the HMD1 user, for example, the avatar displayed on the LCD 63 as shown in FIG. 403 and the avatar 402 of the PC 220.
In the example shown in FIG. 29, the position of the user of the PC 220 is the most east relative to the position of the HMD 1 in the positional relationship of the three persons. Further, the position of the user of the HMD 500 cannot be acquired. Therefore, as shown in FIG. 29, it is assumed that the user of the HMD 500 is the west most. The position of the user of the PC 210 is between the positions of the PC 220 and the HMD 500. Similar to the arrangement relationship of the avatars, the audio adjustment unit 310 distributes the audio signals of the PC 220 with a higher specific gravity to the right. For example, the audio adjustment unit 310 distributes the right volume to 80% and the left volume to 20%. The audio adjustment unit 310 distributes the audio signal of the HMD 500 to the left with a high specific gravity, for example, the right is assigned to a volume of 20% and the left is assigned to a volume of 80%. The audio adjustment unit 310 distributes the audio signal of the PC 210 equally to the left and right, that is, the right is 50% volume and the left is 50% volume.
The audio adjustment unit 310 synthesizes the audio signals thus distributed and outputs the synthesized audio data to the decoder 121.
As a result of the processing in step S203, the display position of each avatar (401 to 403) displayed on the LCD 63 matches the direction in which the sound heard by the stereo earphone 100 can be heard.

  As described above, the left and right balances of the position information of the visual avatar and the sound from the terminal corresponding to the auditory avatar coincide with each other. It becomes possible to distinguish properly using. According to the present embodiment, for example, even when a plurality of voices having similar voice qualities are heard, the balance between the avatar position and the voice is different depending on the visual and auditory information. Can be recognized as a voice.

  In this embodiment, an example in which avatars are displayed side by side has been described, but avatars may be displayed side by side. When displaying the avatar vertically, the processing unit of each terminal determines the vertical position of the avatar according to the latitude of the other terminal. Also, the sound characteristics are changed by sound processing so that the sound from the terminal corresponding to the avatar arranged above can be heard from above, and the sound from the terminal corresponding to the avatar arranged below can be heard from below. Also good.

Next, another embodiment of the avatar display will be described.
FIG. 33 is a flowchart of a display procedure in another avatar display according to this embodiment. In the following description, it is assumed that all terminals used by users participating in a call are the HMD 1 according to the present embodiment.
(Steps S101 to S105) Processing is performed in the same manner as steps S101 to S105 in FIG.

(Step S <b> 301) The position calculation unit 309 of the processing unit 123 extracts environment information of the communication partner from the reception signal output from the transmission / reception unit 303. The position calculation unit 309 extracts or calculates position information of each terminal and information such as a direction in which the user of each terminal is facing (hereinafter referred to as terminal position / orientation information) from the extracted environment information. The environment information transmitted from each terminal includes various sensors (acceleration sensor 132, geomagnetic sensor 133, angular velocity sensor 134, GPS sensor 135, temperature / humidity sensor 136, heart rate sensor 137) included in the HMD used by each user. Etc.). The position calculation unit 309 outputs the extracted or calculated position / orientation information of each terminal to the avatar generation unit 306.

(Step S302) The avatar generation unit 306 arranges the avatar corresponding to each call partner two-dimensionally in the display image of the LCD 63 based on the position and orientation information of the call partner with the position of the HMD 1 as the center. indicate.

(Step S303) The avatar generation unit 306 observes a change in position information of each terminal for a predetermined period, and determines whether or not the user of each terminal is moving based on the observed result. .
When the user of each terminal is moving, the avatar generation unit 306 calculates the moving speed based on the moving distance moved for a predetermined period and the time required for moving, and based on the calculated result. Thus, the moving means of the user of each terminal is estimated.
When the user of each terminal is not moving, the avatar generation unit 306 determines that the user is using the terminal at home or the like.
Next, the avatar production | generation part 306 produces | generates the additional information icon which shows the estimated stay place, or the additional information icon which shows the estimated moving means.
Next, the avatar production | generation part 306 arrange | positions and displays the additional information icon corresponding to each produced | generated terminal in the vicinity of each avatar.

(Step S304) Thereafter, the same processing as steps S108 to S116 of FIG. 26 is performed.

Here, a specific example of the arrangement procedure of the avatar will be described.
FIG. 34 is a diagram illustrating an example of a display image in which avatars according to the present embodiment are two-dimensionally arranged. In FIG. 34, an image 400e includes avatars 401 to 403, additional information icons 471 to 473, and a mark image 474 indicating the direction of HMD1.

The avatar production | generation part 306 sets the position of HMD1 in the center of LCD63, and displays the mark image 474 which shows the direction of HMD1 in the set center part. In FIG. 34, the vertex 475 of the mark image 474 “Δ” represents, for example, north.
Next, the avatar generation unit 306 compares the position information corresponding to the avatar 401 with the position information of the HMD1, and displays the avatar 401 in the lower left with respect to the mark image 474 as shown in FIG. To do.
Next, the avatar generation unit 306 compares the position information corresponding to the avatar 402 with the position information of the HMD1, and arranges the avatar 402 at the lower right with respect to the mark image 474 as shown in FIG. indicate.
Next, the avatar generation unit 306 compares the position information corresponding to the avatar 403 with the position information of the HMD 1 and displays the avatar 403 in front of the mark image 474 as shown in FIG. To do.
The intervals (L1 to L3) between the mark image 474 “Δ” and each avatar (401 to 403) are calculated by the avatar generation unit 306 based on the distance between the HMD1 and the terminal corresponding to each avatar. Value. In FIG. 34, the distance L1 is shorter than the distance L3, and the distance L3 is shorter than the distance L2.
In the example illustrated in FIG. 34, the user corresponding to the avatar 401 is located closest to the user of the HMD 1. Moreover, it represents that the user corresponding to the avatar 402 is at the position farthest from the user of the HMD 1.
Note that the processing unit 123 may update the arrangement of the avatars shown in FIG. 34 every predetermined period or when it is determined that the position of the other party has changed.

Since the image as shown in FIG. 34 is displayed on the LCD 63, the user of the HMD 1 can more visually recognize the positional relationship of the user of each terminal by checking the image on the LCD 63.
In addition, avatars corresponding to other terminals centering on the position of each terminal are arranged and displayed on the LCD of each terminal.

Also, in the avatar described with reference to FIG. 34, a facial expression can be added to each avatar similarly to steps S110 and S111 (FIG. 26).
For example, when the call partner corresponds to the avatar 403, it is determined that the terminal corresponding to the avatar 403 (hereinafter referred to as the terminal of the avatar 403) is in front of the HMD1, and the terminal of the avatar 403 is facing the direction of the HMD1. Then, the avatar of the face facing the front like the avatar 403 is displayed.
Further, when it is determined that the terminal of the avatar 403 is facing sideways, the processing unit 123 displays the avatar in a state of facing right or left in consideration of the output of the geomagnetic sensor 133 of the terminal of the avatar 403. To do. Further, the processing unit 123 may add additional information of an image indicating sweat to the avatar 403 if the output of the temperature / humidity sensor 136 or the output of the heart rate sensor 137 is equal to or greater than a predetermined value.

Next, in FIG. 34, additional information icons (471 to 473) added to each avatar (401 to 403) will be described.
In step S303 (FIG. 33), the avatar generation unit 306 calculates the moving speed of each avatar's terminal.
As a result of the calculation, if it is determined that the terminal of the avatar 401 is not moving, the avatar generation unit 306 arranges the additional information icon 471 of “house” in the vicinity of the avatar 401 as shown in FIG. indicate.
As a result of the calculation, when the moving speed of the terminal of the avatar 402 is estimated to be moving by an aircraft, the avatar generation unit 306 displays an additional information icon 472 of “aircraft” as shown in FIG. It arrange | positions and displays in the vicinity of the avatar 402.
As a result of the calculation, when the moving speed of the terminal of the avatar 403 is a predetermined speed and the moving speed is estimated to be information indicating the altitude included in the information received from the terminal, the avatar generation unit 306 As shown in FIG. 34, the additional information icon 473 of “ship” is arranged and displayed in the vicinity of the avatar 403. In this case, for example, the terminal of the avatar 403 moves on the sea at almost zero above sea level.
Note that the example in FIG. 34 is a display example when the use of the information output device in the present embodiment in an aircraft is permitted.

As another example of the additional information icon, if it is determined that the vehicle is moving in the direction of the track at a location that seems to be on the track, an icon indicating a train is determined, and the position is determined to be a mountainous area. In this case, a mountain icon may be added, and if the position information is determined to be an amusement park such as a theme park, an icon indicating an amusement park may be added.
In addition, when it is determined that the output of the acceleration sensor 132 continues to be equal to or less than the predetermined value, an icon indicating the universe may be added by determining the weightless state. The determination of these icon types may be performed by the processing unit 201 of the server 200 because the position information and the map data need to be compared.

Next, an example of an avatar display when the display unit 60 of the HMD 1 is rotated to the left by 90 degrees from the display state of FIG. 34 will be described.
FIG. 35 is a diagram for explaining an example of an avatar display when the display unit 60 of the HMD 1 is rotated about 90 degrees to the left in the display of FIG.
In the example shown in FIG. 35, the processing unit 123 of the HMD 1 is a case where the display unit 60 detects that the display unit 60 has been rotated to the left by about 90 degrees based on the output of the geomagnetic sensor 133, and the image 400e in FIG. All displayed avatars are displayed rotated 90 degrees to the right.
As a result, as shown in FIG. 35, the avatar 403 and the additional information icon 473 are displayed in the right direction of the HMD1. Similarly, the avatar 402 and the additional information icon 472 are displayed on the lower left. Similarly, an avatar 401 and an additional information icon 471 are displayed on the upper left.
With this rotation, the audio adjustment unit 310 changes the balance between the left and right sounds adjusted in step 109. The sound adjustment unit 310 changes the sound balance in real time with respect to the movement of the orientation of the HMD 1. On the other hand, the voice adjustment unit 310 slowly changes the display of the avatar with a certain low-pass filter applied. The reason for this is that when the head movement is fast and the orientation of the HMD 1 changes frequently, visual recognition does not follow and the so-called “drunk” may occur. On the other hand, since such a problem does not occur in a change in sound, the left and right balance may be adjusted in real time for the sound.

  As described above, according to the present embodiment, the state of the call partner and the relative positional relationship with the call partner are displayed on the display unit (including LCD) of each terminal, and the voice is relative to the partner. Sounds like it was emitted from the direction. As a result, according to the present embodiment, the positional relationship of the other party can be identified visually and auditorily, so that good communication can be performed.

  Each processing unit of each terminal displays the avatar and the avatar additional information icon shown in FIGS. 27 to 31, 34, and 35 so that the image displayed on the layer 1 can be seen through. Alternatively, it may be displayed translucently. In this case, the brightness may have an initial value set in advance. The brightness may be adjusted by the user operating the touch switch 34. Further, only the avatar may be displayed translucently, and only the avatar additional information icon may be displayed translucently, or vice versa.

  In the present embodiment, an example in which information indicating a plurality of avatars is stored in advance in the avatar storage unit 307 is described, but the present invention is not limited thereto. For example, the processing unit 123 of the HMD 1 may receive information indicating the avatar of the user who is calling with the HMD 1 via the communication interface (131, 138).

  In the present embodiment, the facial expression information of the user is transmitted to another terminal and the facial expression is added to the avatar. However, the facial expression information may be, for example, a touch switch 34 or the like depending on the user's situation. You may make it switch so that transmission may not be permitted by operation.

Note that the LCD 63 of the HMD 1 may be a display unit capable of three-dimensional display with the naked eye. In this case, for example, as shown in FIG. 20, even if the avatars are displayed side by side, the avatars are differentiated in the depth direction of the LCD 63 based on the relative distance to the HMD1, for example. May be displayed.
Further, the camera 64 may be provided with a camera capable of three-dimensional imaging. If the camera 64 is capable of three-dimensional imaging, the image information transmitted to the other terminal is a three-dimensional image if it is known in advance that the other terminal includes a display unit that can display three-dimensionally. Information may be transmitted.

  In the present embodiment, the example in which the processing unit 123 includes the audio adjustment unit 310 has been described. However, for example, an independent functional unit may be provided between the processing unit 123 and the decoder 121, or the decoder 121 May be provided.

In this embodiment, the example in which the sound received from each terminal is distributed to the left and right based on the positional relationship of each terminal has been described. However, at this time, if the sound is smaller than a predetermined sound volume, You may make it distribute after amplifying to right and left. This process may be set to be performed automatically in the initial setting, for example, and the setting may be turned off according to the application. Alternatively, in the initial setting, the setting may be set to the off state, and the user may operate the touch switch 34 to turn on the setting.
Also, the audio adjustment unit 310 may not only amplify but also extract only the audio band and amplify only the extracted band. Alternatively, even if the sound volume is equal to or higher than a predetermined sound volume, if there is a lot of noise, the band other than the sound may be limited to suppress the noise component.

  In the present embodiment, an example has been described in which the sound from the terminal is 100% in the left and right direction, for example, the right volume is 80% and the left volume is 20%, but is not limited thereto. The total value of the left and right volume may not be 100%, may be smaller than 100%, or may be 100% or more. For example, depending on the number of participants in the call, if the other parties are talking, adjust the total left and right volume so that the volume is easy to hear when the audio signal is synthesized You may do it.

  In the present embodiment, an example in which audio is reproduced in stereo using the stereo earphone 100 has been described. However, audio may be reproduced by two speakers (not shown) connected to the audio connector 26.

  In this embodiment, the case where there are a plurality of call partners has been described as an example. However, even if there is only one call partner, the voice adjustment unit 310 distributes the voice signal of the call partner to the left and right. Also good. Even in this case, it is possible to easily recognize by voice the direction in which the other party is located with respect to the HMD 1 of the caller. Further, the processing unit 123 may arrange and display the avatar to be displayed on the LCD 63 based on the relative positional relationship with the HMD 1 and the azimuth even if there is only one call partner. Thereby, the user can easily recognize the positional relationship with the call partner visually.

  In the present embodiment, an example in which the users of the HMD 1 and the HMD 500 perform various operations by operating the touch switches (34, 534) has been described, but the present invention is not limited thereto. For example, the operation may be performed via the remote controller 140. Alternatively, in operations such as switching and selection, the processing units (122 and 522) of each terminal (HMD1 and HMD500) detect the number of blinks of the user, and based on the detected number of blinks, for example, You may control so that a cursor display, the drawing of a line drawing, etc. may be switched.

  In this embodiment, an example in which each terminal draws a line drawing on an image has been shown, but the present invention is not limited to this. For example, a comment input from the keyboard of the input unit 214 of the PC 210 may be superimposed on the received image. Even in this case, the comment is displayed and stored in a different layer from the received image in each terminal and server. For this reason, only the comment can be displayed and deleted.

  In the present embodiment, an example in which the color of the cursor and the color of the line drawing are made similar to each other has been described. However, the color of the avatar may also be made similar. For example, when the cursor by HDM1 displayed on the LCD 563 of the HMD 500 is light blue and the line drawing is blue, the clothes color of the avatar of HMD1 and the color of the hair may be blue. This makes it easier to visually recognize which user is drawing a cursor operation or line drawing.

  In addition, although this embodiment demonstrated the example which produces | generates arrangement | positioning of several avatars based on positional information, and allocates the audio | voice signal corresponding to each avatar on the left and right based on the arrangement | positioning of the produced | generated avatar, Not limited. For example, the processing unit 123 may first allocate audio signals corresponding to each avatar to the left and right based on the position information, and arrange and display the avatar so as to match the allocated audio positional relationship. .

  In this embodiment, an example in which the stereo earphone 100 is connected to the audio connector 26 has been described. However, for example, a 4-channel type earphone (not shown) may be connected. In the case of a 4-channel type earphone, in addition to the left and right spread, the front and rear spread can be enhanced, so the voice adjustment unit 310 allocates the voice signal of the call participant to the left and right including the front and rear. May be. As a result, it is possible to easily identify the other party by hearing. In this case, the processing unit 123 may add direction information to the position information to be used.

In the present embodiment, an example in which the voice signal of the other party is allocated to the left and right based on the position information and then reproduced as it is or amplified is shown, but the present invention is not limited to this. When the stereo earphone 100 is connected and used, depending on the product, there may be a difference in volume between left and right. In such a case, the processing unit 123 outputs a test signal of a sine wave of, for example, 400 [Hz] having the same volume, left and right, to the stereo earphone 100 via the audio connector 26 before starting the call. The user may listen to this test signal and adjust the left / right balance. Then, during a call, the processing unit 123 may adjust the right and left volume to be reproduced in consideration of the left and right balance adjusted by the test signal.
In this way, by performing calibration in the initial state, for example, even a user who hears differently between the left and right ears can be reproduced with a right and left volume balance that is optimal for the user. As a result, a plurality of call partners can be easily identified.

  In this embodiment, an example in which avatars are arranged and audio signals are allocated to the left and right based on position information has been described. However, a user having the same position information may participate in a call. In such a case, for example, the processing unit 123 may display the arrangement of avatars corresponding to the call partner having the same position information from the right (or from the top) in the order of participation in the call. . Then, the voice adjustment unit 310 allocates the voice signals of the other party having the same position information to the left and right based on the position information of the avatars arranged in this way, so that the arrangement of the avatar and the sound signal can be heard. You may make it match.

  When a call is made with a plurality of call partners, when the LCD or display unit of a terminal (HMD) used by one user is in monochrome display and the camera 64 can only perform monochrome imaging, the processing of other terminals The unit may transmit a monochrome image in accordance with the terminal. In this case, there is an effect that data to be transmitted can be reduced. Alternatively, an image captured in color may be transmitted and converted into black and white on the terminal side and displayed.

In the present embodiment, an example in which video (moving image) is transmitted at low resolution in the initial state and switched to medium resolution or the like in response to a request from the user has been described. However, the present invention is not limited to this. There may be more types of resolution. Further, the user may set the resolution and the frame rate by using the touch switch 34. Specifically, the first setting of the resolution and the first frame rate at the low resolution, the second setting of the resolution and the second frame rate at the low resolution, the third setting of the resolution and the third frame rate at the medium resolution, and the like. .
However, when the user sets the resolution, it is necessary to exchange the call at the same resolution on the user terminal of the other terminal, so that information indicating the resolution to be used is transmitted at the start of the call. It may be.
The image to be transmitted may be monochrome or color depending on the application to be used. In the case of color, the image may be 8-bit color, 16-bit color, or the like.

  In the present embodiment, for example, as illustrated in FIG. 29, the example in which each avatar corresponding to the other party is displayed in the same size has been described. However, the processing unit 123 determines the size of the avatar to be displayed. It may be changed based on each position information. In this case, the avatar displayed on the LCD 63 of the HMD 1 is based on the relative positional relationship with the HMD 1. For example, the avatar of the nearest caller is displayed in the largest size, and the farthest caller is displayed. The avatar may be displayed in the smallest size. As a result, even when the avatars are arranged side by side or vertically, the sense of distance between the caller and the other party can be visually recognized.

  In the present embodiment, the example in which the size of the avatar is fixed has been described, but the size may be varied during a call. For example, the processing unit 123 may be varied so that the avatar of the talking partner is greatly displayed. As a result, even when the avatars are arranged side by side or vertically, the sense of distance between the caller and the other party can be visually recognized.

  In the present embodiment, for example, in the example illustrated in FIG. 29, an anthropomorphic image as an avatar has been described, but the present invention is not limited thereto. The avatar may be, for example, an animal or a robot. When such an avatar is used, a facial expression cannot be simply added to the avatar. In such a case, instead of changing the facial expression, the processing unit 123 may perform control such as changing the size of the avatar, changing the color of the avatar, blinking the avatar, or vibrating the avatar.

In the example of FIGS. 10 to 20 of the present embodiment, an example has been described in which a call is made by four parties including the user of the HMD 1 and the same image is shared among all users, but the present invention is not limited to this. For example, the terminal that requested the image may transmit the image information only to the requested terminal. Alternatively, the transmission source may limit the transmission destination or transmit the image information only to grouped members.
Further, not only for images but also for audio signals, the requested terminal may transmit the audio signal only to the requested terminal. Alternatively, the audio signal may be transmitted only to members who are limited by the transmission source or grouped.

  In this embodiment, the example in which the display main body 20 is mounted on the headband 40 has been described, but the present invention is not limited to this. The user may use the display body 20 alone without attaching it to the headband 40.

  In the present embodiment, the example in which the information output device includes the position information acquisition unit, the audio information acquisition unit, and the audio adjustment unit has been described, but the present invention is not limited thereto. For example, a system having an apparatus including each unit may be used.

  The technical scope of the present invention is not limited to the above embodiment. At least one of the requirements described in the above embodiments may be omitted. The requirements described in the above embodiments can be combined as appropriate.

  The above-described HMD 1 (including 500) has a computer system (for example, the processing units 123 and 523) therein. The operation process of each functional unit is stored in a computer-readable recording medium in the form of a program, and the above processing is performed by the computer system reading and executing the program. Here, the computer system includes a CPU, various memories, an OS, and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, it also includes those that hold a program for a certain period of time, such as volatile memory inside a computer system that serves as a server or client in that case. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

DESCRIPTION OF SYMBOLS 1 ... Head mounted display, 20 ... Display main body, 26 ... Audio connector, 34 ... Touch switch, 60 ... Display part, 63 ... Display panel (display part), 64 ... Camera, 72 ... Image sensor, 100 ... Earphone, 103 ... 1st speaker, 104 ... 2nd speaker, 121 ... decoder, 123 ... processing unit, 131 ... WiFi communication circuit, 132 ... acceleration sensor, 133 ... geomagnetic sensor, 137 heart rate sensor, 138 ... 3G / LTE communication circuit, 140 ... Remote control unit 301 ... input detection unit 302 ... image switching unit 303 ... transmission / reception unit 304 ... line drawing generation unit 305 cursor generation unit 306 ... avatar generation unit 307 ... avatar storage unit 308 ... image synthesis unit 309 ... Position calculation unit, 310 ... Audio adjustment unit

Claims (9)

A location acquisition unit that acquires location information of the terminal from a terminal;
An audio acquisition unit for acquiring audio information from the terminal;
Based on the position information acquired by the position acquisition unit, the audio adjustment unit that distributes the audio signal acquired by the audio acquisition unit into left and right audio signals;
An information output device comprising: A display for displaying an image;
An avatar generating unit that generates an avatar that is a substitute of the user of the terminal;
With
The avatar generator is
Based on the position information acquired by the position acquisition unit, determine the arrangement of the generated avatar, display the avatar on the display unit based on the determined arrangement of the avatar,
The voice adjustment unit
The information output device according to claim 1, wherein a volume to be distributed to the left and right audio signals is adjusted based on the arrangement of avatars generated by the avatar generation unit. The voice adjustment unit
The information output device according to claim 1 or 2, wherein the left and right distribution volume of each audio signal acquired from the plurality of terminals is adjusted based on the position information acquired from the plurality of terminals. . The avatar generator is
Based on the position information acquired from the plurality of terminals by the position acquisition unit, determine the arrangement of the plurality of generated avatars,
The voice adjustment unit
The information output device according to claim 2, wherein the left and right distribution volume of each audio signal acquired from the plurality of terminals is adjusted based on the left and right arrangement of the plurality of avatars generated by the avatar generation unit. . The avatar generator is
Based on the participation state of the call from the terminal, adjust the display and arrangement of the generated avatar,
The voice adjustment unit
The information output device according to claim 2, wherein the left and right distribution volume of each audio signal acquired from the terminal is adjusted based on the display and arrangement of the avatar adjusted by the avatar generation unit. The avatar generator is
The information according to claim 2, wherein when the position information cannot be obtained from the terminal, the avatars are arranged at predetermined positions in the order of participation in the call and displayed on the display unit. Output device. The voice adjustment unit
The audio signal acquired by the audio acquisition unit is distributed to left and right audio signals based on the location information acquired from the other terminal by the location acquisition unit and the location information of the terminal itself. The information output device according to claim 6.
The avatar generator is
The said position acquisition part arrange | positions the produced | generated avatar based on the positional information which the said terminal acquired from the said other terminal, and the positional information on an own terminal, and displays it on the said display part. Information output device. An information output method in an information output device,
A position acquisition unit for acquiring position information of the terminal from the terminal;
A voice acquisition procedure in which a voice acquisition unit acquires voice information from the terminal;
A voice adjustment procedure in which the voice adjustment unit distributes the voice signal acquired by the voice acquisition procedure into left and right voice signals based on the position information acquired by the position acquisition procedure;
An information output method comprising: