JP2019110535A - Communication device and remote communication system - Google Patents

Abstract

To provide a technique that presents sight line information to a user.SOLUTION: A communication device is provided at a spot where a user is and communicates with a remote device operated by a cooperator at a second spot distant from the first spot. The communication device includes: a photographing unit for photographing the first spot; an image transmission unit for transmitting a captured image taken with the photographing unit to the remote device; an output control unit for outputting a message received from the remote device to the user; a display unit including a plurality of light sources disposed in a plurality of directions on an outer surface of the communication device; and a display control unit for receiving sight line information indicating a sight line of the cooperator watching the captured image displayed by the remote device, turning on, among the light sources, light sources corresponding to the sight line information, and having a display indicating the sight line of the cooperator made.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a communication device and a remote communication system.

  As a system for realizing communication between people present in a remote place, there are a teleconferencing system, a telepresence system and the like which communicate by communicating two-way video and audio.

  Patent Document 1 proposes a telecommunication system for communicating an intention between a worker who actually performs work on site and a supervisor who supervises this worker at a remote place.

  In addition, the concept of Internet of Abilities (IoA) has been proposed, which provides human and machine capabilities via a network and extends what people can do.

JP 10-257475 A JP, 2011-077652, A JP, 2017-033575, A

"A remote communication system that makes sense of joint outing" Master's thesis, Graduate School of Information Science and Technology, University of Tsukuba, Kei Kei Zhang, March 2011, http://www.iplab.cs.tsukuba.ac.jp/paper /master/tsubaki_master.pdf, search date: October 10, 2017

  When communicating with a person at a remote place via a communication device etc., when talking using a directive such as "that car", "a shelf there", etc., what are you talking about? I sometimes got lost. If it is communication between people, it is possible to know what they are talking about by the line of sight of the person who is talking, but in communication with people who are in distant places, they can not see the other person's line of sight, There is a problem that smooth communication can not be achieved because it is necessary to talk concretely, such as a red car running to the front right and "the top shelf next to a telephone."

  In addition, when a beginner performs work while receiving advice from a skilled person, what is important is whether the advanced person pays attention to the work is important information, but in communication via a communication device, this line-of-sight information can be read There was a problem of not being.

  Then, this invention aims at provision of the technique which shows gaze information to a user.

In order to solve the above-mentioned subject, the communication device of the present invention,
A communication device provided at a first location where a user exists and communicating with a remote device operated by a co-worker located at a second location remote from said first location,
An imaging unit for imaging the first point;
An image transmission unit for transmitting a photographed image photographed by the photographing unit to the remote device;
An output control unit for outputting the message received from the remote device to the user;
A display unit having a plurality of light sources arranged in a plurality of directions on an outer surface of the communication device;
The line-of-sight information indicating the line-of-sight of the collaborator who views the photographed image displayed on the remote device is received, and the light source corresponding to the line-of-sight information is made to emit light among the light sources. And a display control unit configured to perform the display shown.

In order to solve the above problems, the communication system of the present invention is
A telecommunication system comprising a communication device provided at a first point where a user exists and a remote device provided at a second point remote from said first point,
The communication device is
An imaging unit for imaging the first point;
An image transmission unit for transmitting a photographed image photographed by the photographing unit to the remote device;
An output control unit for outputting the message received from the remote device to the user;
A display unit having a plurality of light sources arranged in a plurality of directions on an outer surface of the communication device;
And a display control unit that causes the light source to emit light in accordance with the line-of-sight information received from the remote device, among the light sources.
The remote device is
An image receiving unit that receives the captured image;
A display control unit that causes the display device to display the captured image;
A message acquisition unit for acquiring a message for the user from a collaborator who views the photographed image displayed on the display device;
A line-of-sight information acquisition unit that acquires, as line-of-sight information, information indicating a position within the photographed image that the collaborator is gazing at;
And a transmission control unit that transmits the message and the line-of-sight information to the communication device.

  The display control unit acquires, from the remote device, generation time information indicating the time of generation of the line-of-sight information and the line-of-sight information from the remote device, and displays the line of sight from the time indicated by the generation time information. The direction of the line of sight displayed on the display unit may be corrected based on the amount of change in the direction of the display unit that has changed up to time.

Further, in order to solve the above problems, the communication system of the present invention is
A telecommunication system comprising: a communication device provided at a first point where a user exists, and a remote device operated by a co-worker present at a second point remote from said first point,
The communication device is
An imaging unit for imaging the first point;
An image transmission unit for transmitting a photographed image photographed by the photographing unit to the remote device;
A sound signal transmission unit for transmitting a sound signal indicating the voice of the user to the remote device;
An output control unit for outputting the message received from the remote device to the user;
A display unit having a plurality of light sources arranged in a plurality of directions on an outer surface of the communication device;
The line-of-sight information indicating the line-of-sight of the collaborator who views the photographed image displayed on the remote device is received, and the light source corresponding to the line-of-sight information is made to emit light among the light sources. And a display control unit that makes the display shown.
The remote device is
An image receiving unit that receives the captured image;
A sound signal receiving unit that receives the sound signal;
A display control unit that causes the display device to display the captured image;
A voice recognition unit that recognizes the voice of the user based on the sound signal and converts the voice into a character string;
A request determination unit that determines the user's request based on the character string;
A request target determination unit that determines the target of the request based on the character string;
A message generator for generating a message corresponding to the request;
A target extraction unit that extracts an image indicating a target of the request from the captured image;
A line-of-sight information acquisition unit that acquires, as line-of-sight information, information indicating a position where a subject corresponding to an image indicating the target is present;
And a transmission control unit that transmits the message and the line-of-sight information to the communication device.

  In the communication device, the display control unit may receive the line-of-sight information from each of the plurality of remote devices, and may cause the display unit to display a plurality of lines of sight based on the line-of-sight information.

The communication device includes a drive unit that drives to change the position of at least a part of the communication device, and the output control unit operates the drive unit based on operation instruction information received from the remote device. May cause the user to perform an action.
The communication device includes a plurality of the imaging units, and the imaging unit is disposed such that an imaging range imaged by the imaging unit overlaps with an imaging range of another imaging unit.
The image combining unit is configured to combine overlapping portions of the captured images captured by a plurality of the imaging units, and to generate a composite image having a resolution higher than that of the captured images.
The image transmission unit may transmit the composite image to the remote device.
When the communication device receives an emotion signal indicating the type of emotion of the collaborator from the remote device, the display control unit emits the light source with a light emission pattern according to the type of emotion indicated by the emotion signal. It may be allowed to present the emotions of the collaborator.
In the remote communication system, the communication device includes a plurality of the imaging units, and the imaging units are arranged such that the imaging range imaged by the imaging unit partially overlaps the imaging range of another imaging unit.
The remote device is
The image composition device further includes an image combining unit that combines overlapping parts of captured images captured by a plurality of the capturing units into a composite image having a resolution higher than that of the captured images.
The display control unit may display the composite image on the display device.
The remote communication system includes a transmission control unit that transmits, to the communication device, instruction information that instructs the imaging direction of the communication device.
The communication device may include a drive unit that changes at least the direction of the imaging unit in accordance with instruction information received from the remote device.
The remote communication system includes a storage unit storing attribute information on a plurality of collaborators;
An acquisition unit for acquiring the input information by the user, the sound information at the first point, or the photographed image by the photographing unit as selection information;
A selection unit that selects the collaborator based on the attribute information of the collaborator and the selection information;
A notification unit for notifying that the contact point of the collaborator selected by the selection unit is selected;
May be provided.

  The present invention can provide a technology for presenting the gaze information to the user.

FIG. 1 is a schematic configuration diagram of the remote communication system according to the first embodiment. FIG. 2 is a side view of the avatar. FIG. 3 is a perspective view showing a state in which the cover of the display unit is removed in the avatar. FIG. 4 is a partial cross-sectional view of the display unit. FIG. 5 is a view showing the configuration of the goggles provided with the display unit of the remote device. FIG. 6 is a view showing a state in which the ghost wears the goggles. FIG. 7 is a diagram showing the relationship between the direction of the goggles and the gaze position. FIG. 8 is a diagram showing a process in which the avatar transmits image and voice data. FIG. 9 is a diagram showing processing in which the remote device receives image and audio data. FIG. 10 is a diagram showing a process for transmitting a message and gaze information from the remote device to the avatar. FIG. 11 is a diagram showing a process in which the avatar displays line-of-sight information. FIG. 12 is a diagram showing a process of correcting the light source to be lighted based on the sight line information by the avatar. FIG. 13 is a diagram showing an example of displaying the direction of the line of sight. FIG. 14 is a view showing a usage scene of the remote communication system according to the present embodiment. FIG. 15 is a view showing an avatar according to the first modification. FIG. 16 is a diagram showing processing of transmitting operation instruction information from the remote device to the avatar. FIG. 17 is a diagram showing processing performed by the avatar based on the operation instruction information received from the remote device. FIG. 18 is a schematic configuration diagram of the remote communication system according to the second embodiment. FIG. 19 is a diagram illustrating an example of the request pattern. FIG. 20 is a diagram showing a process in which the remote device according to the second embodiment generates a message and line-of-sight information based on the data received from the avatar and transmits it to the avatar. FIG. 21 is a view showing a second modification of the avatar. FIG. 22 is a view showing a light source of the second modification. FIG. 23 is a diagram showing the configuration of an imaging unit according to the third embodiment. FIG. 24 is a side view of an avatar schematically showing a shooting range. FIG. 25 is a plan view of an avatar schematically showing a shooting range. FIG. 26 is a schematic configuration diagram of the remote communication system according to the third embodiment. FIG. 27 is a diagram showing a process in which the avatar transmits image and voice data. FIG. 28 is an explanatory diagram of an image synthesized at the synthesis step. FIG. 29 is a view showing an example of a subject to be subjected to the combining process. FIG. 30 is a view showing an example of an image distorted by the aberration of the photographing lens. FIG. 31 is a view showing an example of an image distorted by the aberration of the photographing lens. FIG. 32 is a diagram illustrating an example in which distortion of an image in a captured image is corrected. FIG. 33 is a schematic configuration diagram of the remote communication system according to the modified example 3-1. FIG. 34 is a diagram showing processing in which the remote device transmits image and audio data. FIG. 35 is a front view of an avatar according to Modification 3-2. FIG. 36 is a side view of an avatar according to Modification 3-2. FIG. 37 is a plan view of an avatar according to Modification 3-2. FIG. 38 is a schematic configuration diagram of the remote communication system according to the fourth embodiment. FIG. 39 is a diagram showing an example of the ghost DB. FIG. 40 is a diagram showing a process in which the avatar transmits image and voice data. FIG. 41 is a diagram of processing in which the management server identifies a ghost. FIG. 42 is a schematic configuration diagram of the remote communication system according to the fifth embodiment. FIG. 43 is a diagram showing display processing based on gaze information and an emotion signal by the avatar according to the fifth embodiment.

First Embodiment
"Device configuration"
An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic configuration diagram of the remote communication system 100 according to the present embodiment. The remote communication system 100 includes a communication device 1 provided at a first point where a user is present, and a remote device 2 provided at a second point remote from the first point. The communication device 1 photographs the surroundings of the user, and transmits the photographed image and sound to the remote device 2 via a communication line such as the Internet. The remote device 2 outputs the received image or sound to the cooperator, and transmits a message from the coworker to the remote device 2. Thus, for example, communication can be made such that the user asks a question and the collaborator answers. In this case, since the communication device 1 communicates with the user as a substitute of the collaborator, the communication device 1 is hereinafter also referred to as an avatar 1 in the present embodiment. Further, a collaborator who communicates with the user through the avatar 1 is hereinafter also referred to as a ghost.

  The avatar 1 includes a display unit 10, an imaging unit 21, a speaker 22, a microphone 23, a detection unit 24, and a control unit 40.

  The display unit 10 has a plurality of light sources arranged in a plurality of directions on the outer surface of the avatar 1 and indicates the direction of the line of sight of the collaborator. The specific configuration of the display unit will be described later.

  The photographing unit 21 is a camera for photographing a first point where a user exists. Here, the first point is, for example, a range seen from the position where the user is present. Note that the field of view of the user and the imaging range of the imaging unit 21 do not have to completely match, and the range imaged by the imaging unit 21 may include, for example, a range from the user to the viewing angle. Also, a plurality of imaging units 21 may be provided.

  The speaker 22 outputs a sound message or a signal such as music output from the control unit 40 as a sound. The microphone 23 converts sounds around the user and voices of the user into electrical signals (sound signals) and inputs the signals to the control unit 40. The detection unit 24 is a sensor that detects the state of the avatar 1, and for example, a position detection unit that receives a GPS signal and acquires the current position of the avatar 1, and an acceleration sensor that detects the movement direction and rotation direction of the avatar 1. , Geomagnetic sensor etc. which detect geomagnetism and obtain direction.

  The control unit 40 is a computer having a CPU and a memory, and performs processing of the image transmission unit 41, the output control unit 42, the display control unit 43, and the sound signal transmission unit 44.

  The image transmission unit 41 converts the photographed image photographed by the photographing unit 21 according to the specification of the remote device 2, compresses the data size, and transmits the compressed image to the remote device 2. Here, the image transmission unit 41 transmits so that the position of the image photographed in the azimuth used as the reference, such as the front or back of the avatar 1, is the position in the photographed image. For example, the center of the omnidirectional image captured by the imaging unit 21A is the position of the image captured in front of the avatar 1, the center of the omnidirectional image captured by the imaging unit 21B is the position of the image captured behind the avatar 1, etc. To the remote device 2 in a specific format. Note that the process of converting and compressing the captured image is not essential, and the captured image may be transmitted to the remote device 2 as it is. Further, the image transmission unit 41 may transmit shooting information such as a shooting position and a shooting direction of a shot image to the remote device 2 together with the shot image.

  The sound signal transmission unit 44 transmits to the remote device 2 a sound signal indicating the sound around the user and the voice of the user acquired via the microphone 23.

  The output control unit 42 outputs the message received from the remote device 2 to the user. In this embodiment, a voice message of the collaborator's voice is received from the remote device 2 and output from the speaker 22 to the user as a voice message. In addition, when the avatar 1 is provided with a display device capable of displaying characters and images as an output unit, the output control unit 42 receives characters and images by the cooperator from the remote device 2 as a message, and the characters and images May be displayed on the display device. In addition, when the avatar 1 includes a vibrator as an output unit, the output control unit 42 may receive a command instructing vibration by a cooperator from the remote device 2 as a message, and may vibrate the vibrator according to the command. .

  The display control unit 43 receives the line-of-sight information related to the line of sight of the collaborator from the remote device 2 and causes the light source according to the line-of-sight information to emit light among the plurality of light sources provided in the display unit 10. Present the user the direction of Therefore, the display control unit 43 stores, as mapping information, a value such as the direction (angle) of the line of sight indicated by the line-of-sight information and the light source to be lit. When the line-of-sight information is received, the mapping information is referred to, and the light source corresponding to the line-of-sight information is identified and emitted.

  2 is a side view of the avatar 1, FIG. 3 is a perspective view showing the avatar 1 with the cover 14 of the display unit 10 removed, and FIG. 4 is a partial cross-sectional view of the display unit 10.

  The avatar 1 according to the present embodiment includes a main body 31 having a substantially regular dodecahedron outer shape, and a support 32 for supporting the main body 31. The main body 31 includes the imaging units 21 on two parallel surfaces of the twelve surfaces forming the outer shape, and the display unit 10 on the other surfaces.

  In this embodiment, the direction from the user to the front is forward, the opposite direction to the rear, the direction to the right of the user to the right, the opposite direction to the left, and the direction from the user to the upper Upward, let this opposite direction be downward. These directions are shown for the convenience of description, and the configuration of the avatar 1 is not limited to these directions.

  In the example of FIG. 2, among the photographing units 21, those arranged to photograph the front are set as the photographing unit 21 A, and those arranged to photograph the rear are set as the photographing unit 21 B. The imaging units 21A and 21B each have an imaging angle of view of about 190 °, and so-called omnidirectional images can be obtained by connecting the images captured by each.

  The display unit 10 has a substantially regular dodecahedron cover 14 and a substantially regular dodecahedron base 15 surrounded by the cover 14. The cover 14 is provided with an opening that exposes the photographing unit 21 on two parallel surfaces of the regular dodecahedron. In the base 15 of the display unit 10, chip-shaped LEDs (light emitting diodes) 12 are arranged in a plurality of directions as light sources on a pentagonal substrate 13, and a milky white cover 14 is provided on the outside of the LEDs 12. Each of the LEDs 12 is connected to the control unit 40 so that it can be lit individually or in predetermined units. In addition, since the display part 10 of this embodiment is equipped with many LED12, in FIG. 3, the code is attached to only a part of LED12 for convenience, and the code | symbol is abbreviate | omitted and shown about the other LED12. . The same is true for the subsequent figures. The display unit 10 may be a display element such as a liquid crystal display element or an EL panel. In this case, each pixel of the display element corresponds to a light source. The cover 14 spreads the range (viewing angle) in which the display by this light can be seen by diffusing the light output from the LED 12.

  Next, returning to FIG. 1, the remote device 2 will be described. The remote device 2 includes a display unit 51, a speaker 52, a microphone 53, a detection unit 54, and a control unit 60.

  The display unit 51 is a display device that displays the photographed image received from the avatar 1 and presents the image to the collaborator.

  The speaker 52 outputs a signal such as the sound of the first point output from the control unit 60 or the voice of the user as a sound. The microphone 53 converts the voice of the collaborator into an electrical signal and inputs the electrical signal to the control unit 60. The detection unit 54 is, for example, a sensor that detects the position in the captured image at which the ghost is watching.

  The control unit 60 is a computer having a CPU and a memory, and performs processing of the image reception unit 61, the display control unit 62, the message acquisition unit 63, the line-of-sight information acquisition unit 64, the transmission control unit 65, and the sound signal reception unit 66. .

  The image reception unit 61 receives a photographed image from the avatar 1. The display control unit 62 causes the display unit 51 to display the captured image. The display control unit 62 may perform processing of decompressing the data compressed by the avatar 1 and converting the photographed image according to the display unit 51.

  The message acquisition unit 63 acquires a message for the user from the ghost viewing the captured image displayed on the display unit 51. The message acquisition unit 63 acquires, for example, the sound of a ghost as a message by the microphone 53.

  The line-of-sight information acquisition unit 64 acquires, as line-of-sight information, information indicating the position in the captured image at which the ghost is watching. In addition, the specific method of acquiring line-of-sight information will be described later. The transmission control unit 65 transmits information such as the message and the line-of-sight information to the avatar 1. The sound signal receiving unit 66 receives a sound signal indicating the voice or the like of the user from the avatar 1.

  5 shows the configuration of the goggles 70 provided with the display unit 51 of the remote device 2, FIG. 6 shows the goggles 70 attached to a ghost, and FIG. FIG.

  In the example of FIG. 5, the goggles 70 include the display unit 51, the microphone 53, the detection unit 54, and the lens 55 in a housing. The lens 55 is an eyepiece system that guides a light flux emitted from the display surface of the display unit 51 to the ghost eye 39 and forms an image on the retina. This allows the ghost to observe the image as a magnified virtual image.

The detection unit 54 is, for example, an angular velocity sensor or a geomagnetic sensor, and detects the direction of the goggles 70, that is, the direction of the line of sight of the ghost wearing the goggles 70 as shown in FIG. The control unit 60 causes the display unit 51 to display an image corresponding to the direction detected by the detection unit 54. In FIG. 7, the goggle 70 is turned over, and the orientation of the goggle 70 is shown at a clockwise angle θ with respect to the reference direction in the horizontal plane. For example, assuming that the direction corresponding to the front of the avatar 1 is a reference direction (0 °), the right side is indicated as 90 °, the rear side as 180 °, and the left side as 270 °. Also in the avatar 1, when showing the shooting direction and the direction of the line of sight to be displayed, the avatar 1 is turned in the same manner as in FIG. 7 and is indicated by a clockwise angle θ with respect to the reference direction in the horizontal plane.
The imaging unit 21A of the avatar 1 performs imaging at an imaging angle of view of approximately 190 ° with the optical axis of the imaging lens (not shown) directed forward (0 ° in FIG. 7). Further, the imaging unit 21B of the avatar 1 performs imaging at an imaging angle of view of approximately 190 ° with the optical axis of the imaging lens (not shown) directed rearward (180 ° in FIG. 7). The avatar 1 stitch-processes the images captured by the imaging units 21A and 21B, respectively, to obtain an omnidirectional image, and records an image in all directions (360 °). Then, the avatar 1 transmits this omnidirectional image to the remote device 2.
The remote device 2 having received the photographed image from the avatar 1 selects an image of a predetermined range centered on the direction of the goggles 70 among the photographed images of all directions (360 °) photographed by the avatar 1 according to the direction of the goggles 70 It is cut out and displayed on the display unit 51. For example, when the range to be displayed is ± 30 °, if the direction of the goggles 70 is 0 ° shown in FIG. 7, the captured image in the range of 330 ° to 30 ° from the captured image of all directions (360 °) It is cut out and displayed on the display unit 51. Similarly, when the direction of the goggles 70 is 90 °, the captured image in the range of 60 ° to 120 ° is cut out from the captured image of all directions (360 °) and displayed on the display unit 51. Therefore, by changing the orientation of the head, the ghost wearing the goggles 70 can observe the surrounding situation as if looking around at the first point. The control unit 60 acquires the direction of the goggles 70 when the ghost observes the surroundings as gaze information. Although the orientation of the goggles in the horizontal plane is defined in FIG. 7, the orientation of the goggles in the vertical plane can be defined in the same manner. For example, the orientation of the goggle 70 is indicated at a clockwise angle with respect to the reference direction in the vertical plane. Similarly, in the avatar 1, the shooting direction and the direction of the line of sight to be displayed are also indicated by a clockwise angle with respect to the reference direction in the vertical plane. The control unit 60 also acquires the direction of the goggles 70 in the vertical plane as line-of-sight information as in the above horizontal direction, and transmits the information to the avatar 1 for display on the display unit 10.

  In the above example, the direction of the goggles 70 is acquired as gaze information of a ghost. However, the present invention is not limited to this. A captured image received from the avatar 1 is displayed on a display device, and the ghost points a gaze position in the captured image. It is also possible to designate this gaze position as gaze information by designating with the device. In this case, the avatar 1 having received the line-of-sight information determines the gaze position on the first point corresponding to the gaze position designated on the photographed image, for example, and the direction from the position of the avatar 1 to the gaze position is It is displayed on the display unit 10 as the direction.

"Remote communication method"
FIG. 8 is a diagram showing a process in which the avatar 1 transmits image and audio data. When the avatar 1 is activated or when the start of communication is instructed, the avatar 1 starts the process of FIG. In step S10, avatar 1 acquires the user's voice and surrounding sounds via microphone 23, and uses it as a sound signal.

  In step S <b> 20, avatar 1 acquires a photographed image from photographing unit 21. Further, in step S30, avatar 1 converts the photographed image according to the specification of remote device 2. For example, processing for correcting distortion when shooting with a wide-angle lens, and processing for connecting captured images captured by a plurality of imaging units 21A and 21B are performed. When the ghost observes the photographed image with the goggles 70, only a part of the photographed image is displayed according to the direction of the line of sight, so the photographed image to be the display range may be extracted according to the line of sight.

  The avatar 1 compresses the acquired sound and the data of the photographed image in step S40, and transmits the data of the sound and the photographed image to the remote device 2 in step S50. The avatar 1 repeatedly executes the process of FIG. 8 until the power is turned off or until the end of the communication is instructed. Further, when communicating with a plurality of remote devices, avatar 1 may execute the processing of FIG. 8 in parallel, or in step S50, multicast data of sound and photographed image to a plurality of remote devices 2. May be

  FIG. 9 is a diagram showing processing in which the remote device 2 receives image and audio data. When the remote device 2 is activated or the start of communication is instructed, the remote device 2 starts the process of FIG. The remote device 2 receives the sound and the image data from the avatar 1 in step S110, and decompresses the compressed data in step S120.

  In step S130, the remote device 2 outputs the received sound signal from the speaker 52 as a sound.

  At step S140, remote device 2 converts the photographed image according to the specification of display unit 51, and causes display unit 51 to display the photographed image at step S150. The remote device 2 repeatedly executes the processing of FIG. 9 until the power is turned off or until termination of communication is instructed.

  FIG. 10 is a diagram showing a process of transmitting a message and line-of-sight information from the remote device 2 to the avatar 1. When the remote device 2 is activated or the start of communication is instructed, the remote device 2 starts the process of FIG.

  In step S210, information indicating the line of sight of the ghost who views the photographed image displayed on the display unit 51 is acquired as line of sight information. For example, when the ghost observes a captured image using the goggles 70, the direction of the goggles 70 is detected and used as line-of-sight information.

  Also, in step S220, the remote device 2 acquires a ghost voice message via the microphone 53, and compresses the voice message in step S230.

  In step S240, the remote device 2 transmits the gaze information and the voice message to the avatar 1. The remote device 2 repeatedly executes the processing of FIG. 10 until the power is turned off or until termination of communication is instructed.

  FIG. 11 is a diagram showing processing in which the avatar 1 displays line-of-sight information. When the avatar 1 is activated or when the start of communication is instructed, the avatar 1 starts the process of FIG. In step S310, avatar 1 receives the line-of-sight information and the voice message from remote device 2.

  In step S320, the avatar 1 specifies the LED (light source) 12 to be lit based on the line-of-sight information, and presents the direction of the line of sight to the user by turning on the specific LED 12 in step S330.

Further, the avatar 1 decompresses the compressed voice message in step S340, and causes the speaker 22 to output the voice message in step S350. The avatar 1 repeatedly executes the process of FIG. 11 until the power is turned off or until the end of communication is instructed. Further, when the avatar 1 communicates with a plurality of remote devices 2, the process of FIG. 11 may be executed in parallel. In this case, line-of-sight information is obtained from each of the plurality of remote devices 2, and based on the line-of-sight information, the display unit 10 is made to display a plurality of lines of sight. Remote unit 2
The line of sight is displayed in a distinguishable manner by changing the light emission color, the luminance, the blinking pattern and the like of the LED 12 each time.

  FIG. 12 is a diagram showing a process of correcting the light source to be turned on based on the sight line information by the avatar 1. The avatar 1 determines the LED 12 to be lighted on the basis of the line-of-sight information received from the remote device 2 as shown in FIG. 11. For example, when the line-of-sight information is acquired by the remote device 2, If the position and the attitude of the avatar 1 are different at the time when the LED 12 is lighted on the basis, the direction of the display unit 10 is different, and an error occurs in the line of sight displayed on the display unit 10. Therefore, the light source to be turned on is corrected according to the change of the position and posture of the avatar 1. In this case, in order to detect a change in posture, the avatar 1 periodically measures, for example, the direction of the avatar 1 (the direction of the display unit 10) and the photographing position, and stores them as history information together with the measurement time. Further, the remote device transmits generation time information indicating the time of generation of the line-of-sight information to the avatar 1 together with the line-of-sight information. Then, when the avatar 1 is activated or when the start of communication is instructed, the avatar 1 periodically executes the process of FIG. 12. For example, it executes at a cycle shorter than that in FIG.

  When FIG. 12 is started, the avatar 1 acquires line-of-sight information and generation time information received from the remote device 2 in step S405. Also, in step S410, the avatar 1 acquires the current position (shooting position) and the shooting direction.

  In step S420, avatar 1 refers to the history information and compares the shooting position and shooting direction at the time indicated by the generation time information obtained in step S405 with the current shooting position and shooting direction obtained in step S410. And find the difference.

  In step S430, avatar 1 corrects the light source to emit light based on the line-of-sight information based on the difference obtained in step S420. For example, the amount of change in the imaging direction obtained as a difference is subtracted from the direction of the line of sight indicated by the line-of-sight information. Specifically, if the direction of the line of sight indicated by the line-of-sight information is 20 ° with respect to the reference direction shown in FIG. 7 and the current shooting direction deviates 5 ° from this time point, this is subtracted And correct the direction of the line of sight to 15 °.

  In step S440, avatar 1 displays the direction of the line of sight based on the corrected line-of-sight information.

  Although the example of FIG. 12 illustrates an example of correcting the difference between the direction of the line of sight indicated by the line-of-sight information generated on the remote device 2 side and the direction of the line of sight displayed by the avatar 1, the present invention is not limited thereto. 1 transmits so that the position of the image photographed in the reference orientation at the first point is the position in the photographed image, and the remote device makes the reference at the first point Line-of-sight information indicating the direction of the line of sight may be generated and transmitted to the avatar 1 in the direction (angle) with respect to the direction. For example, the avatar 1 indicates the photographing direction when the photographed image is photographed by an angle (azimuth angle) with respect to the azimuth (for example, north) at the first point as a reference, and this to the remote device 2 together with the photographed image Send. Then, the remote device 2 generates line-of-sight information by indicating the direction of the line of sight of the ghost looking at the displayed photographed image by an angle (azimuth angle) with respect to the direction (for example, north) at the first point. Send. As a result, the avatar 1 can indicate the direction of the line of sight without correcting even if the posture at the time of displaying the line of sight changes. If the direction of the line of sight is indicated by an angle to the reference direction such as the horizontal direction or the vertical direction also in the vertical plane and the line-of-sight information is generated, the avatar is not corrected as described above. You can indicate the direction.

FIG. 13 is a diagram showing an example of displaying the direction of the line of sight. In FIG. 13, the lit LEDs 12 are shown with hatching. In FIG. 13 (A) to FIG. 13 (C), the lighted LED 1
The direction of the line of sight is indicated by the arrangement direction of two.

FIG. 14 is a view showing a usage scene of the remote communication system 100 according to the present embodiment. FIG. 14 shows an example in which the avatar 1 is attached to the shoulder of the user. In the example of FIG. 14, the user selects a product while having the following conversation with the ghost via the remote communication system 100 when shopping.
User: "Is this?"
Ghost: "No, no, this is good."
Thus, by using the remote communication system 100, the user can communicate with a ghost located at a remote place. Therefore, even if the user does not have knowledge of the product, it is possible to appropriately select the product using the knowledge of the ghost.

  Also, since the direction of the ghost's line of sight can be presented to the user, it is possible to understand what the ghost is talking while talking, and to communicate smoothly.

  In addition, even if the user can not speak the local language and can not ask the person of the store in overseas travel etc., the ghost who can speak the local language asks the person of the store via the avatar, and the question answer is ghosted Can communicate to the user.

  In this way, even if there is a problem that the user can not solve alone, this problem can be solved by using the ghost's ability at a remote place. On the other hand, ghosts can enjoy shopping and traveling abroad without going out.

  Further, in the case of performing streaming distribution, by communicating with another person using the remote communication system 100, it is possible to reflect an objective opinion of the other person in the streaming content.

Modified Example 1
FIG. 15 is a view showing an avatar 1A according to the first modification. The avatar 1A of FIG. 15 has a rotation mechanism 33 for rotating the main body 31 around the vertical rotation axis VA, and a rotation mechanism 34 for rotating the main body 31 around the horizontal rotation axis HA. The rotation mechanisms 33 and 34 are one form of a drive unit.

  In this modification, the ghost operates the rotation mechanisms 33 and 34, and this operation transmits a message to the user. Since the other configuration is the same as that of the above-described embodiment, the same components are denoted by the same reference symbols, and the description thereof is omitted.

  The remote device 2 transmits operation instruction information for instructing the operation of the avatar 1A to the avatar 1A by the function of the transmission control unit 65. FIG. 16 is a diagram showing processing of transmitting operation instruction information to the avatar 1 from the remote device. The remote device 2 starts the process of FIG. 16 when the ghost inputs an instruction to operate the rotation mechanisms 33 and 34 using input means such as a keyboard and a pointing device.

  In step S510, the remote device 2 acquires, as operation instruction information, an instruction for the rotation mechanisms 33 and 34 input by the operation of the ghost.

  In step S520, the remote device 2 compresses the operation instruction information. The process of compressing the operation instruction information is not essential, and may be omitted when the amount of data is small.

In step S530, the remote device 2 transmits operation instruction information to the avatar 1 via the network.

  FIG. 17 is a diagram showing the process performed by the avatar 1 based on the operation instruction information received from the remote device. In step S570, avatar 1 receives operation instruction information from remote device 2.

  The avatar 1 decompresses the compressed operation instruction information in step S 580, and performs an operation based on the operation instruction information in step S 590. For example, by operating the rotation mechanism 34, the avatar 1 can perform an operation such as nodding. Further, by operating the rotation mechanism 33, it is possible to perform an operation such that the avatar 1 shakes its neck. Thus, the action of the avatar 1 can transmit a message indicating positive or negative to the user.

  In the above embodiment, the imaging unit 21 is configured to capture an omnidirectional image using a wide-angle lens, but in the case of a configuration in which the imaging unit 21 captures a limited range, the imaging range is changed. For this purpose, the rotation mechanisms 33 and 34 may be operated.

  Further, the avatar 1 may be provided with a vibrator, and the avatar 1 may be vibrated based on the operation instruction information instructed by the operation of the ghost.

Second Embodiment
In the above-described embodiment, a person (cooperator) inputs a message as a ghost, and the gaze of the collaborator is detected to be gaze information, but in the present embodiment, the computer operates as a virtual ghost. Is configured to generate line-of-sight information. Here, since the other configuration is the same, the same reference numerals are given to the same elements and the description thereof will not be repeated.

  FIG. 18 is a schematic configuration diagram of the remote communication system 100 according to the present embodiment. The remote device 2A according to the present embodiment is a computer having a CPU and a memory, and the image reception unit 61, the display control unit 62, the message acquisition unit 63, the line-of-sight information acquisition unit 64, the transmission control unit 65, and the sound signal reception unit 66. The voice recognition unit 67, the request determination unit 68, the request target determination unit 69, the message generation unit 71, the target extraction unit 72, and the line-of-sight information acquisition unit 73 are processed.

  The voice recognition unit 67 recognizes the voice of the user based on the sound signal received from the avatar 1 and converts it into a character string. The request determination unit 68 performs morphological analysis and syntax analysis on the character string acquired by the speech recognition unit 67, and determines the user's request from the character string.

  For example, it is determined that the user's request corresponds to the predetermined request pattern shown in FIG. In the example in Fig. 19, go to <facility name>, where is <article>? If there is a request pattern such as eating <food name>, etc., and the request pattern is met, the character string is determined to be a user request, and the content of the request is determined based on the character string. Identify.

  The request target determination unit 69 determines the target of the request based on the character string. For example, it is determined that the object, the subject, and the subject noun, such as <facility name>, <article>, <food name>, and the like, which are included in the request pattern, are to be requested. In this case, if the predicate obtained by the syntactic analysis for the character string matching the required pattern is "go", the object is set to <facility name>. If the predicate is "eat", the object is "food name". If the predicate is "where", the subject is "the article". Furthermore, the word targeted for the request is registered in advance in the dictionary, and among the character strings matching the request pattern, the word corresponding to the facility name dictionary is <facility name> and the word corresponding to the article dictionary is It may be determined that the article]. For example, <article> to be an object, such as a wallet, a smartphone, or a key, is registered in advance in the dictionary.

  The message generator 71 generates a message by performing a process of generating a message associated with the request as shown in FIG. For example, when the request pattern is "Go to <facility name>", route guidance up to the <facility name> is output as a message. If the request pattern is "where is the <article>?", The <article> is retrieved from the past captured image, and a voice indicating the position of the article is output as a message. Furthermore, if the request pattern is "Eat <food name>", the store where the <food name> can be eaten is searched for using the database or search engine on the Internet to find the location of the shop. It is taken as a place, and the route guidance from the present place to the destination is outputted as a message. Here, for route guidance, for example, a map information having data indicating an intersection (node), a road (link) between the intersections, a position of a target, etc. is searched to obtain a route from the current location to the destination Information on the intersection on this route, the road between the intersections, and the target is output as a voice message. For example, the distance from Avatar1's current location to the next intersection or destination, such as "Go 100m in this direction and turn right at the next intersection." "Go straight on the right road to that building." Output the direction as voice.

  The target extraction unit 72 extracts an image indicating the target of the request from the captured image. For example, an image of an <article> to be requested, such as a wallet, a smartphone, or a key, is registered in advance as a target pattern in the storage unit, and a portion matching the target pattern is extracted from the photographed image by pattern matching. Do. Then, the message generation unit 71, in response to a request such as “where is the <article>?”, Determines the position where the subject corresponding to the image of the <article> obtained by the target extraction unit 72 is present. Output a message. For example, "I was in the direction of each place" and "I put it in the direction of the other side". If the image extracted by the object extraction unit 72 can be recognized as information on the position where the <article> is present, such as on a shelf, next to a telephone, etc., include this in the voice message. It may be output like "I placed it on the side of the phone". In addition, when the message extracting unit 71 generates a message indicating a target, such as “A building ...”, “Go straight along the river ...”, etc. The object "building" and "river" are requested (route guidance), and this image is extracted from the photographed image.

  The line-of-sight information acquisition unit 73 acquires, as line-of-sight information, information indicating the position where the subject corresponding to the image indicating the target is present. For example, the position where the subject corresponding to the <article> or the image of the target exists is determined, and the position (for example, latitude and longitude) or the direction viewed from the avatar 1 (for example, as shown in FIG. Information indicating the clockwise angle) is used as the line-of-sight information. Then, the transmission control unit 65 transmits the line-of-sight information to the avatar 1. The avatar 1 having received the line-of-sight information displays the direction of the line of sight on the display unit 10 based on the line-of-sight information. When the line-of-sight information indicates the direction viewed from the avatar 1, the LED 12 indicating the direction is turned on as described above. When the line-of-sight information indicates the position of the subject, the avatar 1 lights the LED 12 so as to indicate the direction from the position of the avatar 1 to the position of the subject indicated by the line-of-sight information. Display.

  As described above, the remote device 2 determines the user's request based on the request pattern, performs message generation processing corresponding to the request, creates a message, and transmits it to the avatar 1. Thus, the remote device 2 functions as a so-called chatter bot that sends back a message in response to the user's request. Note that the remote device 2 according to the present embodiment may determine the user's request and generate a message for the request by AI, instead of fixedly preparing the request pattern and the process of generating the message. . For example, the user sets the multi-layer model for requesting the user and the message for the request, and repeatedly learning about the voice of the user recognized by the voice recognition unit 67 and the photographed image acquired from the avatar 1. Request a pattern to be determined as a request for and a message for the request.

  FIG. 20 is a diagram showing a process in which the remote device 2 of the present embodiment generates a message and line-of-sight information based on the data received from the avatar 1 and transmits it to the avatar 1. The process of transmitting data from the avatar 1 to the remote device 2 is the same as that shown in FIG. When the remote device 2 is activated or the start of communication is instructed, the remote device 2 starts the process of FIG.

  At step S610, the remote device 2 receives the sound and the image data from the avatar 1, and at step S620, the compressed data is decompressed.

  In step S630, remote device 2 recognizes the user's voice based on the sound signal received from avatar 1, and converts it into a character string.

  In step S640, the remote device 2 performs morphological analysis and syntax analysis on the character string acquired in step S630, and determines the user's request from the character string. Also, in step S650, the remote device 2 determines the target of the request based on the character string acquired in step S630.

  In step S660, the remote device 2 generates a message corresponding to the user request obtained in step S630. Also, in step S670, the remote device 2 extracts an image indicating the target of the request from the captured image in the past, and in step 680, information indicating the position where the subject corresponding to the image indicates the target is sight line information Get as.

  Then, in step S690, the remote device 2 transmits the message and the gaze information to the avatar 1. The process in which the avatar 1 receives the message and the line-of-sight information and outputs it to the user is the same as the process in FIG.

  Thus, according to the present embodiment, the user can communicate with the computer via the avatar 1. In addition, the computer, which is a ghost, simulates and displays the line of sight of the avatar 1 so that communication can be smoothly performed.

<Modification 2>
FIG. 21 is a view showing a second modification of the avatar 1 and FIG. 22 is a view showing a light source of the present modification. In the above-mentioned embodiment, although the external shape of avatar 1 was made into approximately regular dodecahedron, the external shape of avatar 1 is not limited to this. In this modification, as shown in FIG. 21, the outer shape of the avatar 1 is substantially spherical. In this modification, the light source 12 is disposed on the arc-shaped film substrates 1F1 to 1F5 and 1R1 to 1R5, and the light source 12 is attached to the outer surface of the substantially spherical casing of the avatar 1. The arc-shaped film substrates 1F1 to 1F5 and 1R1 to 1R5 are connected to the control unit 60 in the housing via the cable IC and the connector IT. Then, the display control unit 43 refers to the mapping information according to the arrangement of the light sources 12, identifies the light source corresponding to the line-of-sight information, and causes the light source to emit light, thereby displaying the direction of the line of sight. Furthermore, the avatar 1 may have another external shape such as a hexahedron or an icosahedron.

Third Embodiment
In the third embodiment, as compared with the first embodiment described above, the imaging unit 21 is disposed such that predetermined portions of the imaging ranges by the plurality of imaging units 21 overlap, and the captured images of the overlapping portions are synthesized to achieve high resolution The configuration for obtaining a composite image of is different. In addition, since the other configuration is the same, the same reference numerals are given to the same elements, and the description thereof will not be repeated.

  FIG. 23 is a view showing the configuration of the photographing unit 21, FIG. 24 is a side view of an avatar 1B schematically showing a photographing range, and FIG. 25 is a plan view of the avatar 1B schematically showing a photographing range.

  As shown in FIG. 23, the imaging unit 21 includes an imaging lens 211 and an imaging element 212. The imaging lens 211 forms an image of a subject on the imaging surface of the imaging element 212. The angle of view of the photographing lens 211 is not particularly limited, but is preferably a lens with a wide angle of view so that a wide range of subjects can be photographed, and a so-called fisheye lens is adopted in this embodiment. The imaging lens 211 is disposed such that the optical axis A 0 is located at the center of the imaging surface of the imaging element 212.

  The imaging element 212 is an imaging device such as a CCD image sensor or a CMOS image sensor, converts an image of light formed on a light receiving surface (imaging surface) into an electrical signal according to the intensity of the light, Image information (photographed image).

  As shown in FIG. 24, the avatar 1B of this embodiment has a photographing unit 21A and a photographing unit 21B, and in side view, a photographing range in which a photographing range photographed by one photographing unit 21A is photographed by the other photographing unit 21B The respective imaging units 21A and 21B are disposed so as to overlap with the range. In addition, since the display part 10 of this embodiment is equipped with many LED12, in FIG. 24, the code is attached to only a part of LED12 for convenience, and the code | symbol is abbreviate | omitted and shown about other LED12. .

  In the first embodiment described above, the imaging units 21A and 21B are disposed in a back-to-back relationship, and the imaging units 21A and 21B are horizontally directed to obtain an omnidirectional image. In this case, the image of the lower part of avatar 1 B is necessarily the image of support 32, which is an area of low value in order to know the situation at the first point. For this reason, in the present embodiment, the imaging units 21A and 21B are disposed more upward than the horizontal direction so that the lower side where the support 32 exists is removed from the imaging range, and the upper imaging range is overlapped.

  The photographing unit 21A sets a range of 190 ° around the optical axis A1 of the photographing lens 211 as a photographing range. Similarly, the imaging unit 21B sets an imaging range of 190 ° centered on the optical axis A2. Then, in a side view, a recess angle θ formed by the optical axis A1 and the optical axis A2 with a point V0 at which the straight line extending the optical axis A1 extending to the image plane or later intersects the straight line extending the optical axis A2 to the image plane Is 120 °. That is, the imaging | photography range of imaging part 21A, 21B overlaps in the range of 50 degrees which make point V0 a vertex.

  On the other hand, in plan view, as shown in FIG. 25, the avatar 1B has the imaging unit 21A forward, the imaging unit 21B backward, and the optical axes A1 and A2 in a straight line. That is, the imaging units 21A and 21B are arranged such that the optical axis A1 of the imaging unit 21A and the optical axis A2 of the imaging unit 21B are directed in directions different by 180 ° around the point V0. Therefore, when the imaging angle of view of the imaging units 21A and 21B is 190 °, the peripheral portion of the imaging range overlaps as in the first embodiment. The arrangement of FIGS. 23 and 24 is an example, and the arrangement of the imaging units 21A and 21B is not limited to this. For example, also in planar view, the imaging regions may be overlapped with the angle of the optical axes A1 and A2 with the point V0 at the top being less than 180 °. When the avatar 1B is attached to the shoulder or arm of the user, the user is photographed in a specific direction in plan view, so the direction of the photographing units 21A and 21B is determined so as to avoid this direction and the user is opposed The imaging area on the side may be made to overlap. Further, three or more cameras may be arranged to overlap at least some of the shooting areas of the cameras.

FIG. 26 is a schematic configuration diagram of the remote communication system 100 according to the present embodiment. The avatar 1B of the present embodiment further includes an image combining unit 45 as compared with the avatar 1 of FIG.

  The image combining unit 45 combines overlapping portions of the captured images captured by the plurality of capturing units 21A and 21B, and generates a composite image having a higher resolution than the captured images.

  FIG. 27 is a diagram showing a process in which the avatar 1B transmits image and audio data. When the avatar 1B is activated or when the start of communication is instructed, the avatar 1B starts the process of FIG. Steps S10 to S30 are the same as the process of FIG. 8 described above.

  In step S35, avatar 1B uses the function of image combining unit 45 to combine overlapping portions of the captured images captured by imaging units 21A and 21B. In this case, the avatar 1B obtains a composite image using both the information of the image photographed by the photographing unit 21A and the information of the image photographed by the photographing unit 21B, so that the resolution is higher than that of the photographed images joined in step S30. The composite image of can be obtained. Then, the avatar 1B compresses in step S40 the captured image obtained in step S30 and the composite image and sound data obtained in step S35, and transmits the compressed image to the remote apparatus 2 in step S50. Note that the avatar 1B repeatedly executes the processing of FIG. 27 until the power is turned off, or until termination of communication is instructed.

  FIG. 28 is an explanatory diagram of an image synthesized at the above-described synthesis step S35. The imaging lens 211 of the imaging unit 21A forms a substantially circular image on the imaging surface, and the imaging element 212 converts the image into an electrical signal to form a captured image 51A. Similarly, the imaging unit 21B forms a substantially circular image on the imaging surface, and the imaging sensor converts the image into an electrical signal to obtain a captured image 51B. As described above, the imaging units 21A and 21B are provided so that the imaging ranges partially overlap, and the shaded portion in FIG. 28 is the overlapping portion 51C of the captured images 51A and 51B.

  The images obtained in the overlapping portion 51C are obtained by shooting the same subject, but have different distortions due to the aberration of the shooting lens. For example, when the rectangular subject 92 shown in FIG. 29 is photographed, the photographed image 51A taken by the photographing unit 21A is distorted and photographed as an image 92A of FIG. In this case, since distortion is larger on the peripheral side than on the center side of the photographed image 51A, the image 92A is photographed as if expanding toward the outside of the circle. Further, also in the photographed image 51B by the photographing unit 21B, the image 92B of the subject 92 is photographed as if expanding toward the peripheral side as shown in FIG. As described above, since the image 92A and the image 92B are distorted differently, the shapes do not match even when the same subject 92 is photographed.

  However, distortion occurring in the images 92A and 92B is determined by the distance from the center of the captured images 51A and 51B. Therefore, the avatar 1B corrects the distortion of the overlapping portion in the captured images 51A and 51B based on the distance from the center and the like. FIG. 32 is a diagram showing an example in which distortion of the images 92A and 92B in the photographed images 51A and 51B is corrected.

  Although the subject 92 shown in FIG. 29 has the same length on the left and right sides 92L and 92R, for example, the image 92A shown in FIG. 30 has a smaller right side 92AR on the center side than the left side 92AL on the peripheral side due to the influence of distortion. It is deformed. That is, the number of pixels forming the right side 92AR is smaller than the number of pixels forming the left side 92AL. Therefore, in order to correct the image 92A as shown in FIG. 32, the left side 92AL is reduced according to the right side 92AR, or the right side 92AR is enlarged according to the left side 92AL.

When the left side 92AL is reduced, the information of the pixels obtained by imaging the left side 92AL is thinned out, so that the entire image 92A has a low resolution that matches the right side 92AR. On the other hand, when the right side 92AR is enlarged, the resolution can be maintained high near the left side 92AL in the image 92A, but the enlargement process is performed based on information of a small number of pixels near the right side 92AR.
Resolution is lower than near AL.

  Further, in the image 92B shown in FIG. 31, contrary to the image 32A, the left side 92BL on the center side is deformed smaller than the right side 92BR on the peripheral side due to the influence of distortion. Therefore, in order to correct the image 92B as shown in FIG. 32, the right side 92BR is reduced according to the left side 92BL or the left side 92BL is enlarged according to the right side 92BR.

  Therefore, when the image 92A is expanded so as to expand the pixels on the right side 92AR, information on the pixels on the right side 92BR in the other image 92B is adopted as information between the expanded pixels, and a composite image is generated. Although FIG. 30 and FIG. 31 show the left sides 92AL and BL and the right sides 92AR and BR of the images 92A and 92B, the same applies to the middle part.

  As described above, when correcting distortion due to the aberration of the imaging lens in one of the images 92A and 92B captured in duplicate, for example, the image 92A, the avatar 1B of the present embodiment And the pixels at the corresponding positions (pixels on the peripheral side) in the other image 92B are used as pixels in the other image 92B as a composite image. That is, the avatar 1B combines the high resolution portions of the images 92A and 92B captured in duplicate to form a composite image, whereby a composite image having a resolution higher than that of the captured image captured by the single capturing unit 21 can be obtained. . In the present embodiment, the resolution is not simply the density of pixels, but is a degree that indicates how well an object can be depicted in a captured image, a composite image, or the like. For example, a test chart in which parallel lines are arranged at equal intervals can be taken, and the number of parallel lines per mm reproduced on an image, and two separated subjects can be photographed and separated and reproduced on an image. The reciprocal of the minimum width can be mentioned.

  The remote device 2 that has received the captured image or the composite image displays the captured image or the composite image as in the case of FIG. 9 or 10 described above. For example, when displaying on the goggles 70, according to the direction of the goggles 70, an image of a predetermined range centered on the direction of the goggles is cut out and displayed on the display unit 51. In the example shown in FIGS. 24 and 25, since the upper side of the avatar 1B is photographed redundantly, the composite image is displayed on the display unit 51 when the ghost wearing the goggles 70 faces upward, and the outside of the overlapping range is observed. Display the captured image in case. Further, the remote device 2 can perform enlarged display or reduced display of a captured image or a composite image according to the operation of the ghost.

  According to the present embodiment, the avatar 1B can capture a part of the area in an overlapping manner by the plurality of imaging units 21, and provide an image of the overlapping area with high resolution. Thus, the ghost can observe the image of the overlapping area in detail. Further, since the image in the overlapping area has a high resolution, the image quality is hardly deteriorated even when the ghost displays the image in the overlapping area in an enlarged scale, and it is easy to observe and observe.

<Modification 3-1>
FIG. 33 is a schematic configuration diagram of the remote communication system 100 according to the modified example 3-1. In the third embodiment described above, the avatar 1 generates a composite image of the overlapping area, but in the present modification, the remote device 2B is configured to generate a composite image of the overlapping area. In addition, since this other structure is the same as 3rd embodiment, the same code | symbol is attached | subjected to the same element and description for the second time is abbreviate | omitted.

The avatar 1B of this modification has a configuration in which a plurality of imaging units 21 overlap and image a part of the area as in the avatar 1B of the third embodiment described above, and the photographed images photographed by the plurality of imaging units 21 Are transmitted to the remote device 2B without being synthesized. In this case, the avatar 1B may not include the image combining unit 45. In addition, both avatar 1B and remote device 2B perform the image combining unit so that the mode for generating a composite image with avatar 1B and the mode for generating a composite image with remote device 2B can be switched according to the user's selection operation. You may have.

  The remote device 2B further includes an image combining unit 75 as compared with the remote device 2 of FIG. The image combining unit 75 receives the captured images captured by the multiple capturing units 21A and 21B from the avatar 1B, combines the overlapping parts, and generates a composite image having a higher resolution than the captured image.

  FIG. 34 is a diagram showing processing in which the remote device 2B transmits image and audio data. When the remote device 2B is activated or the start of communication is instructed, the remote device 2B starts the process of FIG. Steps S110 to S130 are substantially the same as the process of FIG. 9 described above.

  In step S133, the remote device 2B converts the photographed image of the overlapping portion according to the specification of the display unit 51. In this modification, since the photographed image before conversion processing by avatar 1B is acquired from avatar 1B, processing for correcting distortion when photographed with a wide-angle lens, a plurality of photographings, as in step S30 described above A process of connecting the photographed images photographed by the units 21A and 21B is performed.

  In step S135, the remote device 2B combines the overlapping portions of the captured images captured by the imaging units 21A and 21B by the function of the image combining unit 75. In addition, the process of step S135 is the same as that of above-mentioned step S35.

  In step S140A, remote device 2B extracts an image to be displayed according to the specification of display unit 51 from the captured image converted in step S133 and the composite image generated in step S135.

  In step S150, the remote device 2B causes the display unit 51 to display the image extracted in step S140A.

  According to this modification, since the remote device 2B combines the photographed images photographed in duplicate by the plurality of photographing units 21, the processing load on the avatar 1B is reduced.

<Modification 3-2>
35 is a front view of avatar 1C according to modification 3-2, FIG. 36 is a side view of avatar 1C according to modification 3-2, and FIG. 37 is a plan view of avatar 1C according to modification 3-2. It is. The avatar 1C of this modification has a configuration in which a plurality of imaging units 21 overlap and capture a part of the area as in the avatar 1B of the third embodiment described above, and the direction of the overlapping imaging area is changed Thus, it has a rotation mechanism for rotating the main body 31. Since the other configuration is the same as that of the third embodiment described above, the same components are denoted by the same reference symbols, and the description thereof will not be repeated.

  As shown in FIG. 35, the avatar 1C has a rotation mechanism 131 for rotating the main body 31 about the vertical rotation axis VA, and a rotation mechanism 132 for rotating the main body 31 about the horizontal rotation axis HA. There is. The rotation mechanisms 131 and 132 are one form of a drive unit.

  The rotation mechanism 131 is disposed on the support 32 and holds the support post 133 rotatably relative to the support 32. One end of an arc-shaped arm 134 is connected to the upper end of the support column 133, and the rotation mechanism 132 is connected to the other end of the arm 134.

  The rotation mechanism 132 is supported by the upper end of the arm 134 and connected to the body surface of the avatar 1 C to rotate the body 31 relative to the arm 134.

  The rotation mechanism 131 has a configuration in which the rotation axis VA passes through the center of the main body 31 in the vertical direction, and rotates the main body 31 around the vertical axis via the columns 133 and the arms 134, that is, yaws the main body 31.

  When the rotation mechanism 132 has the surface shown in FIG. 35 as the front of the avatar 1C and the direction perpendicular to the paper surface as the front-rear direction, the rotation axis HA is the left-right direction of the avatar 1C and the center of the main body 31 is horizontal. The main body 31 is rotated about the left and right axes, that is, the main body 31 is pitched.

  When the rotation mechanisms 131 and 132 pitch and yaw the main body 31 of the avatar 1C, it is possible to turn the direction of the area where the photographing unit 21 redundantly photographs in any direction. For example, when the ghost operates the remote device 2B and designates the direction to shoot in duplicate, the remote device 2B transmits operation instruction information specifying this direction to the avatar 1C. Then, the avatar 1C having received the operation instruction information operates the rotation mechanisms 131 and 132 based on the operation instruction information to rotate the main body 31, and turns the overlappingly photographed area in the specified direction.

  For example, with the ghost, the front of avatar 1C is 0 °, 90 ° directly above, 180 ° directly behind (back), 270 ° directly below, and the rotation angle of rotation mechanism 132 is within 0 ° to less than 360 °. specify. In addition, the front of avatar 1C is 0 °, the left is 90 °, the rear right (back) is 180 °, the right is 270 °, and the rotation angle of rotation mechanism 131 is specified in the range of 0 ° to less than 360 °. . As a result, it is possible to make the area in which the avatar 1C redundantly shoots be directed in any direction. The configuration for transmitting and receiving the operation instruction information is the same as in FIGS.

  As described above, according to the present modification, the avatar 1C can be redundantly photographed, and the direction of the area for generating the high-resolution composite image can be specified arbitrarily, and the situation in the direction can be observed in detail. , The image of the said direction can be expanded and displayed.

Fourth Embodiment
The remote communication system 100A according to the fourth embodiment has a configuration provided with a management server 8 for selecting a ghost (cooperator) according to the situation at the first point, as compared with the first embodiment and the second embodiment described above. It is different. In addition, since the other configuration is the same, the same reference numerals are given to the same elements, and the description thereof will not be repeated.

  FIG. 38 is a schematic configuration diagram of the remote communication system 100A according to the fourth embodiment. Avatar 1D of this embodiment is further provided with the information acquisition part 46 for selection, and the information transmission part 47 for selection compared with the avatar 1 of FIG.

  The selection information acquisition unit 46 acquires information for selecting a collaborator from the sound signal input from the microphone 23 and the photographed image photographed by the photographing unit 21. For example, the selection information acquisition unit 46 acquires the voice of the user and the photographed image photographed by the photographing unit 21 and uses these as selection information. In addition, the selection information acquisition unit 46 acquires the voice of the user, identifies the user's request by voice recognition, determines which field the request is from, and selects information indicating the relevant field. It may be generated. In addition, the selection information acquisition unit 46 acquires a photographed image photographed by the photographing unit 21, extracts an image matching a predetermined pattern, such as a signboard, a human face, a vehicle, a traffic sign, etc. from the photographed image. Information for selection may be generated. Furthermore, the selection information acquisition unit 46 may recognize the user's gesture from the captured image captured by the imaging unit 21 and acquire the selection information when the user performs a specific gesture. .

  The selection information transmission unit 47 transmits the selection information acquired by the selection information acquisition unit 46 to the management server 8.

  The remote communication system 100A of the present embodiment further includes a management server 8 as compared to the remote communication system 100 of FIG. The management server 8 is a computer having a CPU and a memory (storage unit), and performs processing of the selection information reception unit 81, the selection unit 82, and the notification unit 83. Further, the management server 8 stores a database (hereinafter, referred to as a ghost DB) in which the attribute information on a plurality of ghosts is stored in the memory.

  The selection information receiving unit 81 acquires input information by the user, sound information at the first spot, and the photographed image by the photographing unit as selection information. For example, the voice of the user is recognized, and the user's question or request is acquired as selection information. In addition, the subject in the photographed image is recognized, and the genre related to the subject is acquired as selection information.

  The selection unit 82 refers to the ghost DB and selects a ghost based on the attribute information of each ghost and the information for selection. FIG. 39 is a diagram showing an example of the ghost DB. The ghost DB in FIG. 39 stores ghost identification information (ghost ID), attribute information, contact information, and the like. The ghost ID is information for uniquely identifying a ghost, and is a number, a name, a nickname or the like. The attribute information is information indicating the nature of the ghost, and is information serving as an element of selection by the selection unit 82. The notification destination is a destination for notifying the ghost that the notification has been selected, and is, for example, the network address of the remote device 2. The notification unit 83 notifies the notification destination of the ghost selected by the selection unit that the ghost has been selected.

  FIG. 40 is a diagram showing a process in which the avatar 1D transmits image and audio data. When the avatar 1D is activated or when the start of communication is instructed, the avatar 1D starts the process of FIG. 40 and executes the process in parallel with the process of FIG.

  In step S710, avatar 1D acquires a photographed image photographed by photographing unit 21 in a predetermined period (for example, the last few seconds).

  In step S720, avatar 1D analyzes the image acquired in step S710, and determines whether a specific gesture has been performed. The specific gesture is, for example, an action such as waving a hand, directing a part such as palm or face to avatar 1D, raising a specific finger such as thumb-up or korna (melonic sign) and bringing hand to avatar 1D It can be mentioned.

  If a negative determination is made at step S720, the avatar 1D ends the process of FIG. On the other hand, if it is affirmation determination by step S720, avatar 1D will transfer to step 730 and will acquire the user's voice and surrounding sound inputted via microphone 23 as a sound signal.

  In step S 740, avatar 1 D performs speech recognition on the sound signal, and converts the user's voice into a character string.

  In step S750, the avatar 1D performs morphological analysis and syntax analysis on the character string acquired in step S740, and determines the user's request from the character string. For example, the acquired character string is divided for each sentence, and it is determined that the predicate and the object thereof correspond to the predetermined request pattern shown in FIG. 19 as the user's request. In the example in Fig. 19, go to <facility name>, where is <article>? , What is the <article>? , Eat <food name>, teach <language> etc. Have a required pattern, such as "Go" "Where?" "What?" "Eat" "Teach me" etc. “Facility name”, “article”, “food name”, and “language” are parts of the subject or object (hereinafter, also referred to as request objects).

  If the acquired sentence is "What is the name of this flower?", Then it matches the request pattern "What is the <article>?" And the "flower name" becomes the <article>.

  Moreover, if the acquired sentence is "Tell me the French which is the apology", the request pattern "Tell <Language>" is met, and the "French" becomes the <Language>.

  In step S760, avatar 1D determines the field of the request target obtained in step S750. For example, avatar 1D is provided with a target field table defining request targets corresponding to fields such as “animal”, “plant”, “vehicle”, “English”, “French”, “history”, “geography”, etc. Determine the field corresponding to the requested object you have found. For example, if the request target is "flower name", the field is determined to be "plant". Also, in the above example, if the request target is "French", the field is also determined to be "French".

  In step S 770, avatar 1 D acquires a photographed image photographed by photographing unit 21. In step S780, avatar 1D analyzes the image acquired in step S770, and extracts an image similar to the specific pattern. Here, the specific pattern is an image assumed to be a request target, and examples thereof include signboards, documents, human faces, vehicles, dogs, cats, traffic signs, and the like. When the request object is determined in step S750, an image similar to the pattern of the request object, such as “flower” or “car” may be extracted from the acquired captured image. Also, if the user's request includes a pronoun that indicates proximity, such as "this", such as "What is this?", The object that the user is holding or an object that is near the user The image of may be extracted. Further, when the request object includes a pronoun indicating “far” such as “that”, an image of an object in a place where the user is pointing a finger or a place where the face is directed may be extracted.

  In step S790, avatar 1D determines the field of the image extracted in step S780. For example, the field corresponding to the name of the pattern for which the extracted image is determined to be similar is determined from the target field table as in step S760. That is, when the extracted image is similar to the pattern of “dog” or “cat”, the corresponding field is “animal”.

  If the extracted image contains characters, character recognition is performed, facility names, locations, headings, titles and the like are extracted, and the corresponding fields are obtained from the target field table. For example, when the extracted image is a signboard of a store, and the store name and the handling product read from the signboard are "pasta", "lunch", and "restaurante ○ ○", refer to the target field table, "gourmet" " Seek corresponding fields such as "Italian food".

  In step S795, the avatar 1D connects to the management server 8 via the network, and transmits selection information indicating the field determined in steps S760 and S790 to the management server. Note that the selection information may include, in addition to the information indicating the field, information specifying an age (age), a gender, a skill level, and the like. For example, in advance, if the user is a "fashion" teenager, if it is a "gourmet" teens or less of the same sex, if it is "sport" a beginner (skill level), etc., specify the ghost attribute Information may be input to the avatar 1D, and the avatar 1D may transmit the information by including it in the selection information. Furthermore, not only the information specifying the attribute of the field etc., but also the request of the user such as "Tell me the French which is the problem" determined in step S750 or the image extracted in step S780 may be included in the selection information. Good.

Although the example shown in FIG. 40 shows an example in which the field used for selection information is obtained from both the sound signal and the image, the present invention is not limited to this, and the field may be obtained from one of the sound signal and the image. When the field is determined in step S760 based on the sound signal, steps S770 to S79 are performed.
If the field can not be specified in step S760, for example, 0 is omitted and the request object extracted in step S750 is a pronoun such as "this" or "that", steps S770 to S790 are performed to specify the field from the image. It is also good.

  The avatar 1D repeatedly executes the process of FIG. 40 until the power is turned off or the start of communication is instructed.

  FIG. 41 is a diagram showing processing in which the management server 8 identifies a ghost. The management server 8 starts the process of FIG. 41 when connected to the avatar 1D via the network.

  At step S810, management server 8 acquires information for selection from avatar 1D. In this example, the avatar 1D performs the process of specifying the field, but the avatar 1D transmits a sound signal, a photographed image, and the like as selection information to the management server 8, and the management server 8 performs the above steps S730 to S790. The field may be specified as well.

  In step S820, the management server 8 refers to the ghost DB and obtains a ghost corresponding to the information for selection acquired in step S810. For example, a ghost in which the field indicated in the selection information matches the field of the attribute information is identified. In addition, when information other than the field, such as age, gender, skill level, etc., is included in the information for selection, a ghost having an attribute matching the information is identified. When there are a plurality of ghosts having attribute information that matches the selection information, a predetermined number of ghosts are selected based on the priority, order, and the like of each ghost.

  In step S830, the management server 8 transmits notification information indicating that it has been selected, to the contact point of the ghost selected in step S820, based on the ghost DB. Here, the contact point of the ghost is, for example, the network address of the remote device 2 used by the ghost. Not limited to this, the contact point of the ghost may be the address of another device such as a smartphone used by the ghost. The management server 8 may transmit the notification information including the address of the avatar 1D and information such as the user request.

  When the ghost that has received the notification information approves the communication with the user, the ghost operates the received device and sends back the approval information. If the communication with the user is not accepted, information indicating non-approval may be returned.

  At step S 840, management server 8 determines whether or not the approval information has been received. If the management server 8 does not receive the approval information within a predetermined time or receives information indicating non-approval in step S840, the management server 8 determines that the result is negative and ends the processing of FIG.

  In the case of a positive determination in step S840, the management server 8 proceeds to step S850, notifies the avatar 1D of the address and ghost ID of the remote device 2 used by the selected ghost, and promotes communication with the remote device 2. When the ghost is approved, the remote device 2 may communicate with the avatar 1D based on the address of the avatar 1D indicated by the notification information, and the steps S840 and S850 may be omitted.

  As described above, according to the present embodiment, it is possible to determine the user's request based on the sound signal of the first spot and the photographed image, and to select an appropriate ghost according to the user's request. This allows the user to receive the ghost's cooperation suitable for the request.

The ghost to be selected is not limited to a person, and may be a virtual ghost realized by the processing of the remote device 2A shown in the second embodiment. For example, the field where the computer (remote device 2A) excels, such as route guidance, transit guidance for transportation, search on the Internet, calculation, etc., is registered in the ghost DB as attribute information, and a selection request matching the field Select a virtual ghost when it receives information. Specifically, when the user has a meal with a plurality of people, hold the slip over avatar 1D, and say "How much is it per person when split by 5 people?" Recognize the user's request like In this case, the avatar 1 D sets the field as “calculation”, and transmits it to the management server 8 by including it in the information for selection, since the target of the request is “share”. When the field indicated by the received selection information is "calculation", the management server 8 refers to the ghost DB to select a corresponding virtual ghost, and transmits notification information to the address of the remote device 2A as a contact. . The remote device 2A that has received the notification information connects with the avatar 1D based on the notification information, acquires the image of the slip, reads the amount of money in the total column and divides by 5 to obtain the amount of money per person, the second embodiment Create a message as well as the form and send it to Avatar 1D.

  In this way, by selecting a virtual ghost if it is an area in which the computer is good, and by selecting an existing person as a ghost in other areas, it is possible to select an existing person only when human sensitivities and the like are necessary. It can be selected as a ghost, and human resources can be used effectively.

Fifth Embodiment
In the remote communication system 100B of the fifth embodiment, the remote device 2C transmits an emotion signal indicating a ghost's emotion to the avatar 1E, and the avatar 1E emits a light source according to the emotion signal, as compared with the fourth embodiment described above. The configuration is different. In addition, since the other configuration is the same, the same reference numerals are given to the same elements, and the description thereof will not be repeated.

  FIG. 42 is a schematic configuration diagram of the remote communication system 100A according to the fifth embodiment. The remote device 2C of the present embodiment further includes an emotion detection unit 76 and an emotion signal transmission unit 77, as compared with the remote device 2 of FIG.

  The emotion detection unit 76 detects a ghost's emotion from the sound emitted by the ghost, the facial expression of the ghost, etc., and generates an emotion signal indicating the type of emotion. For example, the emotion detection unit 76 acquires a ghost voice via the microphone 53, performs voice recognition, and converts a user voice into a character string. Then, the emotion detection unit 76 detects predetermined words related to emotions, such as “happy”, “thank”, “sad”, “bad”, “bad”, “stop”, etc. Detect the height of the voice and the loudness of the voice. In addition, the emotion detection unit 76 may detect non-speech voice such as laughter, sigh, sore throat and the like.

  The emotion detection unit 76 determines the type of emotion such as joy, worry, surprise, sadness, anger based on the detection result of a character string or the like, and generates an emotion signal indicating the type and strength of the emotion. For example, if words such as "happy" or "thank you" or laughter are detected, the kind of emotion is determined as "joy" and words such as "all right", "dangerous", or "worry" are detected. To determine the type of emotion as "anxious". In addition, when words such as "sad", "sorry" and sore throats or crying are detected, the type of emotion is determined as "sadness" and when words such as "no good" or "stop" are detected. Determine the type of emotion as "anger". Also, the strength of the emotion may be determined according to the amount of change as compared to the detected utterance speed, the height of the voice, and the size of the voice in advance. For example, as the speaking speed is high, the voice is high, and the voice is high, the intensity of the emotions of "joy", "care", "surprise" and "anger" is set to a strong value. As for negative emotions (for example, "sadness"), the lower the speech speed, the lower the voice, the smaller the voice, and the stronger the emotion strength may be.

  Further, the remote device 2C captures a ghost face with a camera (shooting unit), and detects the type and strength of emotion from the facial expression of the face. In addition, since the technique which detects an emotion from the facial expression of a face can utilize a well-known technique, detailed description is abbreviate | omitted.

  The emotion signal transmission unit 77 transmits the emotion signal generated by the emotion detection unit 76 to the avatar 1. As described above, in the remote device 2C according to the present embodiment, the emotion detection unit 76 detects the ghost voice or the face without performing a special operation such as the ghost selecting the kind of emotion such as "anger" or "joy". An emotion can be detected from the expression, sent to the avatar 1E, and presented to the user. For this reason, the avatar 1E of the present embodiment further includes an emotion signal receiving unit 48, as compared with the avatar 1D of FIG.

  The emotion signal receiving unit 48 receives an emotion signal from the remote device 2C. In addition, when receiving an emotion signal, the display control unit 43 of the present embodiment causes the LED (light source) 12 to emit light with a light emission pattern according to the type of emotion indicated by the emotion signal, and the ghost emotion is To present.

  The display control unit 43 is, for example, predetermined for each type of emotion indicated by the emotion signal, such as “joy” is pink, “care” is green, “surprise” is orange, “sorrow” is blue, and “anger” is red. The LED 12 emits light in color. In addition, you may make it light-emit combining not only light color but "anger" red and purple and "pleasure" multiple colors like pink and green. Further, the display control unit 43 increases the luminance of the LED 12 as the intensity of emotion is stronger. Furthermore, the display control unit 43 may blink the LED 12 and may make the blink speed faster as the intensity of emotion is stronger.

  FIG. 43 is a diagram showing a process in which the avatar 1E according to the fifth embodiment displays based on the line-of-sight information and the emotion signal. When the avatar 1E is activated or when the start of communication is instructed, the avatar 1E starts the process of FIG. In step S310A, avatar 1E receives the line-of-sight information, the emotion information, and the voice message from remote device 2.

  In step S320, the avatar 1E specifies the LED (light source) 12 to be lit based on the line-of-sight information, and in step S330, turns on the specific LED 12 to present the direction of the line of sight to the user.

  In step S335, avatar 1E lights color, brightness, blinking speed, and the like in a predetermined pattern according to the type of emotion indicating emotion information indicating LEDs 12 other than those identified in step S320. In addition, as for LED12 made to light based on gaze information in step S330, and LED12 made to emit light according to an emotion signal, it is desirable to make a color different. For example, the display control unit 43 causes the LED 12 to be lit based on the line-of-sight information to emit light in a specific color (for example, white), and the emotion is controlled so that the color of the LED 12 to emit light according to the emotion signal becomes other than the specific color. The color to be assigned to each type of color is a color other than the specific color (for example, other than white).

  Steps S340 and S350 are the same as in FIG. 11 described above. The avatar 1E repeatedly executes the processing of FIG. 43 until the power is turned off, or until termination of communication is instructed. In the example of FIG. 43, the LED 12 to emit light according to the emotion signal is other than the LED 12 that presents the gaze direction specified in step S320. However, the present invention is not limited thereto. The light may be emitted in a pattern according to

  According to the present embodiment, since the remote device 2C detects the ghost's emotion and causes the avatar 1E to present it, there is no need for the ghost to spend time for presenting the emotion, and concentration on work such as advice to the user it can.

  In addition, said embodiment and modification are an example, and it is not limited to the said structure, You may change in the range which does not deviate from the meaning of invention. For example, the elements of the above-described embodiment and modifications may be combined, or some of the elements may be omitted.

1: Communication device (avatar)
2, 2A: remote device 10: display 12: LED
13: Substrate 14: Cover 15: Base 21: Imaging unit 31: Body 32: Support 40: Control unit 60: Control unit 70: Goggle 100: Remote communication system

Claims (11)

A communication device provided at a first point where a user exists and communicating with a remote device provided at a second point remote from the first point,
A photographing unit for photographing the first spot;
An image transmission unit for transmitting a photographed image photographed by the photographing unit to the remote device;
An output control unit for outputting a message received from the remote device to a user;
A display unit having a plurality of light sources arranged in a plurality of directions on an outer surface of the communication device;
The line-of-sight information indicating the line-of-sight of the collaborator who views the photographed image displayed on the remote device is received, and the light source corresponding to the line-of-sight information is emitted among the light sources to indicate the line-of-sight of the collaborator A display control unit for performing display;
Communication device comprising   The communication device according to claim 1, wherein the display control unit receives the line-of-sight information from each of the plurality of remote devices, and causes the display unit to display a plurality of lines of sight based on the line-of-sight information. A drive unit configured to drive to change a position of at least a part of the communication device;
The communication device according to claim 1 or 2, wherein the output control unit operates the drive unit based on operation instruction information received from the remote device to cause the user to perform an operation.   The display control unit acquires, from the remote device, generation time information indicating the time when the line-of-sight information was generated along with the line-of-sight information, and changed from the time indicated by the generation time information to the time when the line of sight is displayed The communication apparatus according to any one of claims 1 to 3, wherein the direction of the line of sight displayed on the display unit is corrected based on the amount of change in the direction of the display unit. The imaging unit is provided such that the imaging unit has a plurality of imaging units, and the imaging range imaged by the imaging unit overlaps with the imaging range of another imaging unit.
The image combining unit is configured to combine overlapping portions of the captured images captured by a plurality of the imaging units, and to generate a composite image having a resolution higher than that of the captured images.
The communication device according to any one of claims 1 to 4, wherein the image transmission unit transmits the composite image to the remote device.   When the emotion signal indicating the type of emotion of the collaborator is received from the remote device, the display control unit causes the light source to emit light with a light emission pattern according to the type of emotion indicated by the emotion signal, and the cooperation The communication apparatus according to any one of claims 1 to 5, which presents the emotion of the person. A communication system comprising: a communication device provided at a first point where a user exists; and a remote device provided at a second point remote from the first point and in communication with the communication device ,
The communication device is
A photographing unit for photographing the first spot;
An image transmission unit for transmitting a photographed image photographed by the photographing unit to the remote device;
An output control unit for outputting the message received from the remote device to the user;
A display unit having a plurality of light sources arranged in a plurality of directions on an outer surface of the communication device;
The line-of-sight information indicating the line-of-sight of the collaborator who views the photographed image displayed on the remote device is received, and the light source corresponding to the line-of-sight information is emitted among the light sources to indicate the line-of-sight of the collaborator And a display control unit for displaying
The remote device is
An image receiving unit that receives the captured image;
A display control unit that causes the display device to display the captured image;
A message acquisition unit for acquiring a message for the user from the collaborator who views the photographed image displayed on the display device;
A line-of-sight information acquisition unit that acquires, as line-of-sight information, information indicating the line-of-sight of the collaborator who views the photographed image displayed on the display device;
A transmission control unit that transmits the message and the line-of-sight information to the communication device;
Communication system comprising A communication system comprising: a communication device provided at a first point where a user exists; and a remote device provided at a second point remote from the first point and in communication with the communication device ,
The communication device is
A photographing unit for photographing the first spot;
An image transmission unit for transmitting a photographed image photographed by the photographing unit to the remote device;
A sound signal transmission unit for transmitting a sound signal indicating the voice of the user to the remote device;
An output control unit for outputting the message received from the remote device to the user;
A display unit having a plurality of light sources arranged in a plurality of directions on an outer surface of the communication device;
And a display control unit that causes the light source to emit light according to the line-of-sight information received from the remote device, among the light sources.
The remote device is
An image receiving unit that receives the captured image;
A sound signal receiving unit that receives the sound signal;
A display control unit that causes the display device to display the captured image;
A voice recognition unit that recognizes the voice of the user based on the sound signal and converts the voice into a character string;
A request determination unit that determines the user's request based on the character string;
A request target determination unit that determines the target of the request based on the character string;
A message generator for generating a message corresponding to the request;
A target extraction unit that extracts an image indicating a target of the request from the captured image;
A line-of-sight information acquisition unit that acquires, as line-of-sight information, information indicating a position where a subject corresponding to an image indicating the target is present;
A transmission control unit that transmits the message and the line-of-sight information to the communication device;
Communication system comprising The communication device includes a plurality of the imaging units, and the imaging unit is disposed such that an imaging range imaged by the imaging unit partially overlaps with an imaging range of another imaging unit.
The remote device is
The image composition device further includes an image combining unit that combines overlapping parts of captured images captured by a plurality of the capturing units into a composite image having a resolution higher than that of the captured images.
The remote communication system according to claim 7, wherein the display control unit causes the display device to display the composite image. The remote device includes a transmission control unit that transmits instruction information for instructing a photographing direction of the communication device to the communication device.
The communication device includes a drive unit that changes at least the direction of the imaging unit in accordance with instruction information received from the remote device.
The remote communication system according to claim 9. A storage unit storing attribute information on a plurality of collaborators;
An acquisition unit for acquiring the input information by the user, the sound information at the first point, or the photographed image by the photographing unit as selection information;
A selection unit that selects the collaborator based on the attribute information of the collaborator and the selection information;
A notification unit for notifying that the contact point of the collaborator selected by the selection unit is selected;
The remote communication system according to any one of claims 7 to 10, comprising:

Images