JP7115097B2 - Communication terminal, program, display method, recording medium - Google Patents

Description

The present invention relates to communication terminals, programs, display methods, and recording media.

2. Description of the Related Art A communication system in which communication terminals communicate between a plurality of bases via a communication network such as the Internet or a LAN (Local Area Network) to mutually transmit and receive video and audio has become widespread. Communication systems allow remote users to communicate, for example, to hold video conferences.

Also known is a telepresence robot in which a communication system is applied to the robot. A telepresence robot is a robot that combines "teleconference" and "remote control technology". It is a technology that combines remote control from a remote location and robot technology to allow an operator to operate a robot from a distance and make it behave as if the operator exists (presence) in a certain place.

Also, a technique has been devised for mounting a plurality of cameras on a robot (see, for example, Patent Document 1). Patent Literature 1 discloses a moving object equipped with a front camera and a rear camera that capture the front and rear of a traveling device, and a bird's-eye view camera that captures the entire surroundings of the robot.

However, in the conventional technology, the operator can display an arbitrary direction in a wide-angle image such as the entire circumference. There is a problem that it is difficult to grasp whether

For example, if the operator moves the moving device forward while the direction in which the operator is looking is different from the front direction of the moving device, the moving device moves in a direction in which the operator cannot see the situation. sexuality declines.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a communication terminal that allows an operator to easily recognize which direction a mobile device is facing when looking at a wide-angle image.

In view of the above problems, the present invention provides a communication terminal that communicates with another communication terminal mounted on a mobile device via a network and transmits operation instruction information for the mobile device to the other communication terminal, a communication means for receiving a wide-angle video wider than a predetermined one captured by a wide-angle imaging device mounted on a mobile device or another communication terminal; display processing means for displaying a predetermined area image displayed within an area of a display device of the terminal, and information about the orientation of the mobile device on the display device , wherein the display processing means displays the predetermined area image Separately from the above, information indicating in which direction the image of the predetermined area is viewed from the mobile device is displayed together with the image of the mobile device .

It is possible to provide a communication terminal that allows the operator to easily grasp which direction the mobile device is facing when looking at a wide-angle image.

It is an example of the figure explaining the outline of a communication system. FIG. 10 is a diagram for explaining a display example of an omnidirectional video imaged by a wide-angle imaging device; It is a figure which shows the structural example of the communication system which concerns on one Embodiment. It is a figure which shows the hardware structural example of the communication terminal which concerns on one Embodiment. It is a figure which shows the hardware structural example of a communication management system. It is a figure which shows the hardware structural example of a mobile apparatus. It is a hardware block diagram of an example of a wide-angle imaging device. It is an example of a usage image diagram of a wide-angle imaging device. FIG. 4 is a diagram for explaining an outline of processing from an image captured by a wide-angle imaging device to an equirectangular projection image and an omnidirectional image. FIG. 4 is a diagram for explaining an outline of processing from an image captured by a wide-angle imaging device to an equirectangular projection image and an omnidirectional image. FIG. 10 is a diagram showing the positions of a virtual camera and a predetermined area when an omnidirectional image is a three-dimensional solid sphere; It is a figure which shows an example, such as a three-dimensional perspective view of FIG. FIG. 10 is an example of a diagram showing a relationship between predetermined area information and an image of the predetermined area; 1 is a functional block diagram showing, in blocks, functions of a communication management system, a communication terminal 10A, and a communication terminal 10A included in a communication system; FIG. FIG. 10 is a sequence diagram showing an example of processing in the communication preparation stage of the communication system; It is a figure which shows an example of the destination selection screen displayed on a communication terminal. 4 is a sequence diagram showing an example of communication processing of a communication system; FIG. FIG. 10 is an example of a diagram for explaining a reference imaging direction that serves as a reference for the imaging direction and the front direction of the moving device; FIG. 10 is a diagram showing an example of a relationship between an arbitrary imaging direction displayed by an operator and a reference imaging direction; It is an example of a diagram for schematically explaining perspective projection transformation. FIG. 10 is a diagram showing an example of an omnidirectional video display screen displayed in pattern 1; FIG. 10 is an example of a flowchart showing a procedure for the communication terminal to display a predetermined area image on the display unit; FIG. 4 is a diagram showing an example of a polygon model of a telepresence robot and an object of an imaging direction arrow; FIG. 11 is a diagram showing another display example of the imaging direction arrow in pattern 2; FIG. 10 is an example of a flowchart showing a procedure for the communication terminal to display an omnidirectional image on the display unit in pattern 2; FIG. 10 is an example of a flowchart showing a procedure for the communication terminal to display an omnidirectional image on the display unit in pattern 3; It is an example of an image simultaneous display screen that simultaneously displays an omnidirectional image and a front image (or a rear image).

Hereinafter, a communication device and a display method performed by the communication device will be described as an example of a mode for carrying out the present invention.

<Outline of operations of communication system and communication terminal>
First, an outline of a communication system will be described with reference to FIG. FIG. 1(a) shows an example of a schematic diagram of a communication system using a telepresence robot. A telepresence robot is placed in the office 801 and an operator 810 is in the home 802 . The telepresence robot has a communication terminal 10B and a wide-angle imaging device 9 mounted on a mobile device 20. FIG. Also, the operator 810 has a communication terminal 10A.

The wide-angle imaging device 9 transmits an omnidirectional image (an example of a second image), which will be described later, to the communication terminal 10B. The communication terminals 10A and 10B each have a planar imaging device and a microphone, and transmit and receive video and audio to and from each other. That is, the communication terminal 10B transmits the omnidirectional image and the front image 831 (or/and the rear image 832) to the communication terminal 10A, and the communication terminal 10A transmits the front image 831 to the communication terminal 10B.

Application software (hereinafter referred to as an application) that accepts operations related to movement of the mobile device 20 is running on the communication terminal 10A. . The operation instruction input by the operator 810 is transmitted from the communication terminal 10A to the communication terminal 10B.

Communication terminal 10B and mobile device 20 can communicate by short-range wireless communication such as Bluetooth (registered trademark), and communication terminal 10B controls mobile device 20 based on operation instructions. Device 20 can be moved.

By moving the mobile device 20 in this manner, the operator 810 can converse with a desired partner 820 as shown in FIG. 1(b). In FIG. 1B, the operator 810 moves the mobile device 20 to the location of Mr. A 820a and has a conversation with Mr. A, then moves the mobile device 20 to the location of Mr. B 820b and has a conversation with Mr. B. is doing.

In addition to meeting people, the operator 810 can do many things remotely without moving to the site, such as checking products in stores in remote locations, inspecting factories, and participating in exhibitions. It can be done from the ground.

In addition, if the planar imaging device built in the communication terminal 10B can only capture front and rear images, it is difficult for the operator 810 to check the surroundings of the mobile device 20 (for example, feet, left and right). Therefore, in the present embodiment, the communication terminal 10B is equipped with a wide-angle imaging device 9 capable of imaging 360 degrees around the terminal. The wide-angle imaging device 9 may be mounted on the communication terminal 10B or may be mounted on the mobile device 20. FIG. Since the wide-angle imaging device 9 can capture a wide-angle range, for example, it is arranged above where the surroundings are less likely to be blocked. Although it is arranged above the communication terminal 10B in FIG. Also, the wide-angle imaging device 9 may be built in the communication terminal 10B or may be externally attached.

<Display example of omnidirectional video>
Using FIG. 2, a display example will be described in which the operator can grasp the direction in which the mobile device 20 is facing when the operator is looking at a direction in which a 360-degree omnidirectional image of a subject is shown. do. FIG. 2 is a diagram for explaining a display example of a predetermined area image that is a part of the omnidirectional image captured by the wide-angle imaging device 9. As shown in FIG. In this embodiment, three patterns of display examples that allow the operator to grasp the direction in which the mobile device 20 is facing will be described.

Since the orientation of the mobile device 20 is the same as the orientation of the telepresence robot, the orientation of the mobile device 20 and the telepresence robot may be regarded as the same. Also, the direction of the moving device 20 is the direction in which the moving device 20 is facing, for example, the front direction or the forward direction. As long as the operator can grasp the direction in which the mobile device 20 is facing, the mobile device 20 may be oriented sideways, upward, downward, or the like.

FIG. 2A shows an example of an omnidirectional image display screen 1010 that displays an omnidirectional image according to Pattern 1. FIG. In Pattern 1, an image 840 of the telepresence robot is superimposed on the omnidirectional video currently viewed by the operator, so that the operator can grasp the orientation of the mobile device 20 . The operator can see which direction the mobile device 20 is facing. Note that the image 840 of the telepresence robot is displayed in such a manner that the direction of the telepresence robot can be understood.

The operator causes the communication terminal 10A to display an arbitrary predetermined area image. The image displayed on the omnidirectional image display screen 1010 is the predetermined area image. FIG. 2(a) shows part of an open living room. The predetermined area image is an image of a part of the omnidirectional image displayed on the display device (display unit 109 described later) of communication terminal 10A. The display unit 109 provides an area, and the user displays a predetermined area video within this area.

Therefore, it does not necessarily match the front direction of the mobile device 20 . The communication terminal 10A rotates the telepresence robot based on the difference between the front direction of the mobile device 20 and the imaging direction that defines the predetermined area image (the direction indicating the center of the predetermined area image), and rotates the telepresence robot that has been rotated. Display on the sphere image. The displayed image 840 of the telepresence robot is created by CG (computer graphics) rather than the telepresence robot in the real space. CG is an image drawn using a computer. There are mainly two-dimensional and three-dimensional CG, but three-dimensional CG is used in this embodiment.

Therefore, it is possible to know which direction the telepresence robot is facing in the omnidirectional video. In the example of FIG. 2(a), the telepresence robot faces slightly to the right of the operator. By viewing the image 840 of the telepresence robot, the operator can easily determine in which direction the telepresence robot will move if it is moved forward.

In FIG. 2A, for convenience of explanation, the entire image 840 of the telepresence robot is displayed. It is preferable to display 840 closer to the viewpoint (in front). Therefore, the entire image 840 of the telepresence robot need not be shown.

Next, pattern 2 will be described. FIG. 2(b) shows an example of an omnidirectional image display screen 1010 that displays an omnidirectional image according to Pattern 2. As shown in FIG. Pattern 2 displays the direction in which the operator is looking with reference to the telepresence robot, in addition to the predetermined area image 853 that is part of the omnidirectional image that the operator is looking at. Specifically, an image 854 of the telepresence robot and an imaging direction arrow 8 are displayed.

As shown in FIG. 2(b), the communication terminal 10A displays an arbitrary predetermined area image 853 in the omnidirectional image field 1001 according to the operator's operation. In FIG. 2(b), displayed commodities are shown. The communication terminal 10A also has a telepresence robot display field 1002 in addition to the omnidirectional video field 1001 . A telepresence robot display field 1002 displays an imaging direction arrow 8 indicating the direction of the predetermined area image 853 as viewed from the telepresence robot. That is, the imaging direction arrow 8 stereoscopically represents the imaging direction viewed by the operator. Stereoscopic means having azimuth and elevation information. The displayed image 854 of the telepresence robot is not the telepresence robot in the real space but is created by CG. The imaging direction arrow 8 is also the same. In FIG. 2B, the front direction of the telepresence robot is the imaging direction.

In this way, the operator can grasp in which direction the omnidirectional video is viewed from the telepresence robot. The operator can also know which direction the telepresence robot is facing. Also, in FIG. 2B, it can be seen that the telepresence robot approaches the product by moving forward. Compared to pattern 1, there is an advantage that the telepresence robot does not overlap the omnidirectional video.

FIG. 2(c) shows an example of an omnidirectional image display screen 1010 that displays an omnidirectional image according to pattern 3. FIG. The image displayed on the omnidirectional image display screen 1010 is the predetermined area image. In pattern 3, the communication terminal 10A displays the operation button 860 only when the center direction of the predetermined area image (the imaging direction described above) substantially matches the front direction of the telepresence robot. In the left view c-1 of FIG. 2(c), the operation button 860 is not displayed because the imaging direction does not substantially match the front direction of the telepresence robot. In the right view c-2 of FIG. 2(c), the operation button 860 is displayed because the imaging direction and the front direction of the telepresence robot are substantially the same. Substantially matching means matching to the extent that movement of the mobile device 20 is not hindered. If a perfect match is required, the operation button 860 will rarely be displayed and cannot be operated. For example, a deviation of about ±10 to 20 degrees in each of the horizontal and elevation directions may be allowed. Also, the operator may be able to set the magnitude of the deviation.

In pattern 3, the operation button 860 is displayed when the imaging direction and the front direction of the telepresence robot are substantially aligned. and the front direction of the telepresence robot approximately match. Therefore, the imaging direction can be matched with the direction of the omnidirectional image viewed by the operator. In addition, since the operation button 860 is displayed only when the front direction of the telepresence robot and the direction of the omnidirectional image viewed by the operator are matched, the operator can grasp where the telepresence robot will move when moved forward. can.

<Terms>
An operator 810 is a person who operates the mobile device 20 . An operator 810 is also a user who uses the communication terminal 10A, but is referred to as an operator in this embodiment.

A conversation partner 820 is a person in the company or the like, but may be an animal such as a dog or a cat. Also, the mobile device 20 need not be used by the operator 810 to speak to the other party. For example, when the mobile device 20 patrols an office or the like, the communication terminal 10B may transmit video and audio to the communication terminal 10A, and the operator may not talk.

The place where the mobile device 20 moves may be any place where the mobile device 20 can move. In addition, the movement is not limited to a device that moves by power such as a motor, and a human may assist the movement.

The front image refers to an image with a normal angle of view, and the omnidirectional image refers to an image with a wider angle than the front image (wide-angle image). A normal angle of view is an angle of view at which distortion is allowed even if an image is captured with a perspective projection lens, and can be simply called an image captured with a perspective projection lens. A wide-angle image refers to an image captured by a wide-angle lens such as a fisheye lens. Projection methods include stereographic projection, equidistant projection, equisolid angle projection, and orthogonal projection. The wide-angle image is not limited to an omnidirectional image in which the circumference is captured at 360 degrees.

In the present embodiment, a moving image is referred to as a video, and a single still image is referred to as an image. shall not.

<System configuration example>
FIG. 3 is a diagram illustrating a configuration example of a communication system according to one embodiment. The communication system 1 includes a plurality of communication terminals (10A, 10B1, 10B2, 10B3, 10E), a mobile device 20, a relay device 30, a communication management system 50, and the like. In the following description, "communication terminal 10" is used when indicating an arbitrary communication terminal among a plurality of communication terminals (10A, 10B1, 10B2, 10B3, 10E). In addition, when indicating an arbitrary communication terminal among the communication terminals (10B1, 10B2, 10B3), "communication terminal 10B" is used. Also, the number of communication terminals 10 in FIG. 3 is an example.

The communication terminal 10, the relay device 30, and the communication management system 50 are connected to communicate with other communication terminals 10, devices, and systems via the communication network 2, respectively. The communication network 2 may include, for example, a LAN (Local Area Network), the Internet, a mobile phone network, or a dedicated line.

The communication terminal 10 is, for example, a tablet terminal, a smart phone, a general-purpose information processing device such as a PC (Personal Computer), or a communication terminal such as a dedicated video conference device. The communication terminal 10 can, for example, hold a video conference by transmitting and receiving image data, audio data, etc., to and from one or more other communication terminals 10 .

By executing an application corresponding to the communication system 1, the communication terminal 10A can hold a video conference with the communication terminal 10B and remotely control the mobile device 20 via the communication terminal 10B. For example, the communication terminal 10A can move the mobile device 20 on which the communication terminal 10B is mounted, for example, back and forth, left and right, etc., by operating operation buttons displayed on the display screen of the video conference. The communication terminal 10A can also operate browser software, and the browser software can display images and accept operations of the mobile device 20 with respect to the browser software.

The mobile device 20 and the communication terminal 10B (including the wide-angle imaging device 9) are called a telepresence robot. The mobile device 20 is controlled by the communication terminal 10B attached to the mobile device 20, for example, by driving a plurality of wheels to “forward”, “backward”, “turn right”, “turn left”, and “turn left”. It is a device having a traveling function that moves such as When the user instructs to move left or right, it moves forward after "turning right" or "turning left". Note that the appearance of the mobile device 20 shown in FIG. 3 is merely an example. The mobile device 20 only needs to be movable together with the communication terminal 10B in accordance with an operation instruction from the communication terminal 10B mounted on the mobile device 20 .

The relay device 30 is, for example, an information processing device or a system including one or more information processing devices, and relays content data (video, audio, operation instruction information) transmitted and received between a plurality of communication terminals 10 . However, the plurality of communication terminals 10 may directly transmit and receive content data without going through the relay device 30 .

The communication management system 50 is, for example, an information processing device or a system including one or more information processing devices. The communication management system 50 manages, for example, login authentication from the communication terminal 10, management of the communication status of the communication terminal 10, management of the destination list, etc., and session communication between a plurality of communication terminals 10 via the relay device 30. control, etc.

In one embodiment, a session is realized by the relay device 30 relaying content data including image data and sound data (voice and other sounds) between a plurality of communication terminals 10 .

In the above configuration, for example, an operator of communication terminal 10A can communicate with communication terminal 10B and move communication terminal 10B and mobile device 20 by remote control. As a result, the operator of the communication terminal 10A can move the communication terminal 10B and the mobile device 20 to the vicinity of an arbitrary partner to hold a teleconference or the like.

The communication system 1 also includes a data providing system for unidirectionally transmitting content data from one communication terminal 10 to the other communication terminal 10 via the communication management system 50, and a plurality of communication systems via the communication management system 50. It includes a communication system for mutually transmitting information, emotions, etc. between the terminals 10 . This communication system is a system for mutually transmitting information, emotions, etc., between a plurality of communication communication terminals via the relay device 30, and examples include a television (or video) conference system, a videophone system, and the like. .

<Hardware configuration>
<<Hardware configuration of communication terminal>>
FIG. 4 is a diagram showing a hardware configuration example of a communication terminal according to one embodiment of the present invention. The communication terminal 10 includes a general computer configuration, for example, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a flash memory 104, and an SSD (Solid State Drive). ) 105 and the like. The communication terminal 10 also includes a media I/F (Interface) 107, an input unit 108, a display unit 109, a network I/F 111, a camera 112, an imaging device I/F 113, a microphone 114, a speaker 115, and an audio input/output I/F. It has F116 and the like. Furthermore, the communication terminal 10 has an external device connection I/F 117, a short-range wireless communication section 118, a bus 119, and the like.

The CPU 101 is, for example, an arithmetic unit that realizes each function of the communication terminal 10 by reading programs and data from the ROM 102, the flash memory 104, etc., and executing processing. The ROM 102 is a non-volatile memory in which programs such as an IPL (Initial Program Loader) used to start the CPU 101 are stored in advance. A RAM 103 is a volatile memory that is used as a work area for the CPU 101 or the like.

The flash memory 104 is a storage device that stores, for example, an OS (Operating System), application programs, various data, and the like. The SSD 105 controls reading and writing of various data to/from the flash memory 104 under the control of the CPU 101 . The media I/F 107 controls reading or writing (storage) of data to the recording medium 106, which is a recording medium such as a memory card. A computer-readable program for controlling the communication terminal 10 can be stored in the recording medium.

The input unit 108 is, for example, an input device such as a touch panel, a keyboard, or a pointing device for receiving input operations from an operator. Voice input may be possible. The display unit 109 is a display device that displays various information to the operator. Note that the input unit 108 and the display unit 109 may be, for example, the display input unit 110 such as a touch panel display in which a touch panel and a display are integrated.

Network I/F 111 is a communication interface for communication terminal 10 to transmit and receive data using communication network 2 . Camera 112 includes an imaging element for imaging a subject under the control of CPU 101 . The image pickup device I/F 113 controls image pickup by the camera 112 and converts imaged data into predetermined image data. The camera 112 is a planar image pickup device that takes an image with a narrower angle of view than the wide-angle image pickup device 9 , but has a higher resolution than the wide-angle image pickup device 9 . Resolution is the degree of fineness of a digital image, and is expressed by quantifying the fineness of individual unit points (dots, pixels) that make up the digital image. A “dot” is generally used as a unit of resolution. In the case of a display, it is often represented by numbers arranged horizontally and vertically, such as "1024×768 dots". Comparing the same imaging range, the number of dots of the wide-angle imaging device 9 is smaller than that of the camera 112 .

A microphone 114 converts the recorded voice into an electrical signal. The speaker 115 converts the audio signal into sound and outputs the sound. The audio input/output I/F 116 controls input/output of audio by the microphone 114 and the speaker 115 .

The external device connection I/F 117 is an interface for connecting external devices such as USB (Universal Serial Bus). The external device includes, for example, the mobile device 20 shown in FIG.

The short-range wireless communication unit 118 is a communication interface for communicating with an external device (for example, the mobile device 20) by short-range wireless communication such as Bluetooth (registered trademark) or Bluetooth (registered trademark) Low Energy. A bus 119 is commonly connected to each of the above components and transmits address signals, data signals, various control signals, and the like.

<<Hardware Configuration of Communication Management System and Relay Device>>
FIG. 5 is a diagram illustrating a hardware configuration example of a communication management system according to one embodiment. The communication management system 50 has a general computer configuration, and includes, for example, a CPU 501, a ROM 502, a RAM 503, an HD 504, a HDD (Hard Disk Drive) 505, a media drive 507, a display 508, and the like. The communication management system 50 also has a network I/F 509, a keyboard 511, a mouse 512, a CD-ROM drive 514, a bus 510, and the like.

The CPU 501 is an arithmetic unit that realizes each function of the communication management system 50 by reading programs and data stored in the ROM 502, HD 504, etc., and executing processing. A ROM 502 is a non-volatile memory in which programs such as an IPL used to activate the CPU 501 are stored in advance. A RAM 503 is a volatile memory that is used as a work area for the CPU 501 or the like.

The HD 504 is, for example, a storage device that stores an OS, programs such as application programs, and various data. The HDD 505 controls reading and writing of various data to and from the HD 504 under the control of the CPU 501 . The display 508 is a display device that displays various information such as cursors, menus, windows, characters, and images.

A network I/F 509 is a communication interface for performing data communication using the communication network 2 . The keyboard 511 is an example of an input device for accepting input operations such as characters, numerical values, and various instructions by the system administrator. A mouse 512 is an example of a pointing device that receives operations such as selection and execution of various instructions, selection of processing targets, movement of a cursor, and the like by the system administrator.

Note that the display 508, keyboard 511 and mouse 512 may not normally be included in the communication management system 50 or relay device 30. FIG. In this case, they may be connected as required.

A media drive 507 controls reading and writing (storage) of data to a recording medium 506 such as a memory card, for example. A CD-ROM drive 514 controls reading or writing of data to a disk 513 as an example of a removable recording medium. A bus 510 electrically connects the components described above and transmits address signals, data signals, various control signals, and the like.

Note that the hardware configuration of the computer described above is merely an example.

It is assumed that the relay device 30 has a hardware configuration similar to that of the communication management system 50 . Each program for the communication terminal 10, the relay device 30, and the communication management system 50 is recorded on a computer-readable recording medium as a file in an installable format or an executable format and distributed. can be Examples of the recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray disc, USB memory, and the like. Recording media such as a CD-ROM storing each program and the HD 504 storing these programs can be provided domestically or internationally as a program product.

<<Hardware Configuration of Mobile Device>>
FIG. 6 is a diagram illustrating a hardware configuration example of a mobile device according to one embodiment. The mobile device 20 has, for example, a CPU 401, a RAM 402, a ROM 403, an external device I/F 404, a short-range wireless communication section 405, a wheel drive section 406, a steering section 407, and the like.

The CPU 401 is an arithmetic device that implements each function of the mobile device 20 by executing programs stored in the ROM 403 or the like. A RAM 402 is a volatile memory used as a work area for the CPU 401 and the like. The ROM 403 is a non-volatile memory that stores programs and the like for the mobile device 20 . The ROM 403 may be, for example, a rewritable non-volatile memory such as a flash ROM.

The external device I/F 404 is a wired communication interface for communicating with the external device connection I/F 117 of the communication terminal 10 and the like.

The short-range wireless communication unit 405 is, for example, a wireless communication interface for performing wireless communication using the same wireless communication method as the short-range wireless communication unit 118 of the communication terminal 10 . The mobile device 20 only needs to be able to communicate with the communication terminal 10 via the external device I/F 404 or the short-range wireless communication unit 405, for example.

The wheel drive unit 406 is an example of a drive device that drives wheels for moving the mobile device 20 . The wheel driving unit 406 includes, for example, a motor.

The steering unit 407 is an example of a steering device that steers the moving device 20 that is moved by the wheel driving unit 406 . For example, the steering unit 407 may change the direction and inclination of the wheels, or may change the direction of the mobile device 20 by controlling the number of rotations, speed, etc. of the left and right wheels. good.

<<Hardware configuration example of wide-angle imaging device>>
The hardware configuration of the wide-angle imaging device 9 will be described with reference to FIG. FIG. 7 is a hardware configuration diagram of an example of the wide-angle imaging device 9. As shown in FIG. In the following, the wide-angle imaging device 9 is an omnidirectional (omnidirectional) wide-angle imaging device 9 using two imaging elements, but the number of imaging elements may be two or more. In addition, the device does not necessarily have to be dedicated to omnidirectional imaging, and by attaching an omnidirectional imaging unit to an ordinary digital camera, smartphone, or the like, it can have substantially the same function as the wide-angle imaging device 9. good too.

As shown in FIG. 7, the wide-angle imaging device 9 includes an imaging unit 301, an image processing unit 304, an imaging control unit 305, a microphone 308, a sound processing unit 309, a CPU (Central Processing Unit) 311, a ROM (Read Only). Memory) 312, SRAM (Static Random Access Memory) 313, DRAM (Dynamic Random Access Memory) 314, operation unit 315, network I/F 316, communication unit 317, antenna 317a, electronic compass 318, gyro sensor 319, and acceleration sensor 320 are also connected.

Among them, the imaging unit 301 includes wide-angle lenses (so-called fish-eye lenses) 302a and 302b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging units provided corresponding to each wide-angle lens. It has elements 303a and 303b. The imaging elements 303a and 303b are image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert optical images formed by the fisheye lenses 302a and 302b into electrical signal image data and output them. and a timing generation circuit for generating horizontal or vertical synchronizing signals, pixel clocks, etc., and a group of registers for setting various commands and parameters necessary for the operation of this imaging device.

The imaging elements 303a and 303b of the imaging unit 301 are each connected to the image processing unit 304 via a parallel I/F bus. On the other hand, the imaging elements 303a and 303b of the imaging unit 301 are connected separately from the imaging control unit 305 by a serial I/F bus (I2C bus or the like). The image processing unit 304 , imaging control unit 305 and sound processing unit 309 are connected to a CPU 311 via a bus 310 . Furthermore, the bus 310 is also connected with a ROM 312, an SRAM 313, a DRAM 314, an operation unit 315, a network I/F 316, a communication unit 317, an electronic compass 318, and the like.

The image processing unit 304 fetches the image data output from the image sensors 303a and 303b through the parallel I/F bus, performs predetermined processing on each image data, and synthesizes these image data. , to create equirectangular projection image data as shown in FIG. 9(c).

The imaging control unit 305 generally uses the I2C bus with the imaging control unit 305 as a master device and the imaging elements 303a and 303b as slave devices to set commands and the like in registers of the imaging elements 303a and 303b. Necessary commands and the like are received from the CPU 311 . The imaging control unit 305 also uses the I2C bus to capture status data and the like of the registers of the imaging elements 303 a and 303 b and sends them to the CPU 311 .

Further, the imaging control unit 305 instructs the imaging devices 303a and 303b to output image data at the timing when the shutter button of the operation unit 315 is pressed. The wide-angle imaging device 9 may have a preview display function or a function corresponding to moving image display on a display (for example, the display of the communication terminal 10B). In this case, the image data are output continuously from the imaging elements 303a and 303b at a predetermined frame rate (frames/minute).

In addition, as will be described later, the imaging control unit 305 also functions as synchronization control means for synchronizing the output timing of the image data of the imaging elements 303a and 303b in cooperation with the CPU 311. FIG. In this embodiment, the wide-angle imaging device 9 is not provided with a display, but may be provided with a display unit.

The microphone 308 converts sound into sound (signal) data. The sound processing unit 309 takes in sound data output from the microphone 308 through the I/F bus and performs predetermined processing on the sound data.

The CPU 311 controls the overall operation of the wide-angle imaging device 9 and executes necessary processing. ROM 312 stores various programs for CPU 311 . An SRAM 313 and a DRAM 314 are work memories, and store programs to be executed by the CPU 311, data during processing, and the like. In particular, the DRAM 314 stores image data being processed by the image processing unit 304 and data of a processed equirectangular projection image.

An operation unit 315 is a general term for operation buttons such as a shutter button. By operating the operation unit 315, the operator inputs various imaging modes, imaging conditions, and the like.

The network I/F 316 is a general term for interface circuits (such as USB I/F) with external media such as SD cards and personal computers. Also, the network I/F 316 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 314 is recorded on an external medium via the network I/F 316, or is transmitted to an external terminal (such as the communication terminal 10B) via the network I/F 316 as necessary. device).

The communication unit 317 communicates with an external terminal (device) of the communication terminal 10B via an antenna 317a provided in the wide-angle imaging device 9 by short-range wireless communication technology such as Wi-Fi, NFC, and Bluetooth (registered trademark). I do. The communication unit 317 can also transmit the data of the equirectangular projection image to an external terminal (device) such as the communication terminal 10B.

The electronic compass 318 calculates the azimuth of the wide-angle imaging device 9 from the magnetism of the earth and outputs the azimuth information. This orientation information is an example of related information (metadata) conforming to Exif (Exchangeable image file format), and is used for image processing such as image correction of captured images. The related information also includes data such as the date and time when the image was captured and the data volume of the image data.

The gyro sensor 319 detects changes in angles (roll angle, pitching angle, yaw angle) accompanying movement of the wide-angle imaging device 9 . A change in angle is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of a captured image.

Acceleration sensor 320 detects acceleration in three axial directions. Based on the detected acceleration, the posture (angle with respect to the direction of gravity) of the wide-angle imaging device 9 is detected. Having both the gyro sensor 319 and the acceleration sensor 320 improves the accuracy of image correction.

<About omnidirectional images>
Next, using FIG. 8, the usage of the wide-angle imaging device 9 will be described. FIG. 8 is an image diagram of how the wide-angle imaging device 9 is used. The wide-angle imaging device 9, as shown in FIG. 8, is used, for example, to be held by a user to capture an object around the user. In this case, the imaging elements 303a and 303b shown in FIG. 7 capture images of subjects around the user, respectively, so that two hemispherical images can be obtained.

Next, with reference to FIGS. 9 and 10, an outline of the process until the equirectangular projection image EC and the omnidirectional image CE are created from the image captured by the wide-angle imaging device 9 will be described. 9A is a hemispherical image (front side) captured by the wide-angle imaging device 9, FIG. 9B is a hemispherical image (back side) captured by the wide-angle imaging device 9, and FIG. FIG. 2 is a diagram showing an image represented by the ranged cylindrical projection (hereinafter referred to as an “equidangular projection image”). FIG. 10(a) is a conceptual diagram showing a state in which an equirectangular projection image covers a sphere, and FIG. 10(b) is a diagram showing an omnidirectional image.

As shown in FIG. 9A, the image obtained by the imaging element 303a is a hemispherical image (front side) curved by a fisheye lens 302a, which will be described later. Also, as shown in FIG. 9B, the image obtained by the imaging device 303b is a hemispherical image (rear side) curved by the fisheye lens 302b. Then, the hemispherical image (front side) and the 180-degree inverted hemispherical image (rear side) are combined by the wide-angle imaging device 9 to form an equirectangular projection image EC as shown in FIG. is created.

Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular projection image is pasted so as to cover the spherical surface as shown in FIG. A spherical image CE as shown in b) is created. Thus, the omnidirectional image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the omnidirectional image CE may be a still image or a moving image.

As described above, since the omnidirectional image CE is an image pasted so as to cover the spherical surface, it gives a sense of incongruity when viewed by humans. Therefore, by displaying a predetermined area (hereinafter referred to as the "predetermined area image" described above) of the omnidirectional image CE as a planar image with little curvature, it is possible to display the image without causing discomfort to humans. . This will be described with reference to FIGS. 11 and 12. FIG. The predetermined area image may be a moving image or a still image.

FIG. 11 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of the operator viewing the omnidirectional image CE displayed as a three-dimensional sphere. 12(a) is a three-dimensional perspective view of FIG. 11, and FIG. 12(b) is a diagram showing a predetermined area image displayed on a display. In addition, in FIG. 12(a), the omnidirectional image shown in FIG. 11 is represented by a three-dimensional solid sphere CS. Assuming that the omnidirectional image CE generated in this way is the stereosphere CS, the virtual camera IC is positioned inside the omnidirectional image CE as shown in FIG. A predetermined area T in the omnidirectional image CE is an imaging area of the virtual camera IC, and is specified by predetermined area information indicating an imaging direction and an angle of view of the virtual camera IC in a three-dimensional virtual space including the omnidirectional image CE. be. Further, the zooming of the predetermined area T can also be expressed by moving the virtual camera IC closer to or farther from the omnidirectional image CE. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. Therefore, the predetermined area T can be identified by the angle of view α and the distance f from the virtual camera IC to the omnidirectional image CE (see FIG. 13).

Then, the predetermined area image Q shown in FIG. 12(a) is displayed as an image of the imaging area of the virtual camera IC on a predetermined display, as shown in FIG. 12(b). The image shown in FIG. 12B is a predetermined area video represented by default predetermined area information. In the following description, the imaging direction (ea, aa) and the angle of view (α) of the virtual camera IC will be used. Note that the predetermined area T may be indicated by the imaging area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the angle of view α and the distance f.

Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 13 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. As shown in FIG. 13, "ea" indicates an elevation angle, "aa" indicates an azimuth angle, and "α" indicates an angle of view. That is, the posture of the virtual camera IC is changed so that the point of gaze of the virtual camera IC indicated by the imaging direction (ea, aa) becomes the center point CP of the predetermined area T that is the imaging area of the virtual camera IC. Become. As shown in FIG. 13, when the diagonal angle of view of the predetermined area T represented by the angle of view α of the virtual camera IC is α, the center point CP is the (x, y) parameter of the predetermined area information. becomes. f is the distance from the virtual camera IC to the center point CP. L is the distance between an arbitrary vertex of the predetermined area T and the center point CP (2L is a diagonal line). In FIG. 13, a trigonometric function generally represented by the following (Equation 1) holds.

L/f=tan(α/2) (Formula 1)
<About functions>
Next, functions of the communication system will be described with reference to FIG. FIG. 14 is a functional block diagram showing functions of the communication management system 50, the communication terminal 10B, and the communication terminal 10A included in the communication system 1 in block form.

<<Function Configuration of Communication Terminal 10A>>
The communication terminal 10A is, for example, a communication terminal 10 that accepts operation instructions for controlling equipment such as the mobile device 20 but does not have a function for controlling equipment such as the mobile device 20 . The communication terminal 10A has a transmission/reception section 11, an operation input reception section 12, a communication control section 13, an imaging section 14, an audio input section 15a, an audio output section 15b, a display control section 16, and a storage/readout section 17. . Each of these units is realized by any one of the components shown in FIG. function or means to The communication terminal 10 also has a storage unit 18 implemented by the RAM 103 shown in FIG. 4 and the flash memory 104 shown in FIG.

<<Function Configuration of Communication Terminal 10B>>
The communication terminal 10B is the communication terminal 10 mounted on the mobile device 20. FIG. The communication terminal 10B is, for example, an example of the communication terminal 10 having a device control function such as the mobile device 20 or the like. The communication terminal 10B has, in addition to the functional configuration of the communication terminal 10A described above, an operation instruction receiving section 19a, an omnidirectional video receiving section 19b, a device control section 19c, an inter-device communication section 19d, and the like.

(Each functional configuration of the communication terminal 10)
Next, each functional configuration of the communication terminal 10 (communication terminal 10A and communication terminal 10B) will be described in detail.

The transmission/reception unit 11 transmits/receives various data (or information) to/from another communication terminal, device, or system via the communication network 2 . The transmitting/receiving unit 11 starts receiving each state information indicating the state of each communication terminal as a destination candidate from the communication management system 50 before starting a call with a desired destination terminal. This state information includes not only the operating state (on-line or off-line state) of each communication terminal 10, but also detailed states such as whether a call is possible or whether a call is in progress in the case of an on-line state. indicates

The operation input reception unit 12 receives various inputs to the communication terminal by the operator. For example, when the operator performs an operation to turn on the power of the communication terminal 10, the operation input reception unit 12 receives the operation and controls to turn on the power.

For example, when the power is turned on, the communication control unit 13 sends login request information indicating a login request to the communication management system 50 from the transmission/reception unit 11 via the communication network 2, and the current time of the request source terminal. automatically send the IP address of Further, when the operator performs an operation to turn off the power of the communication terminal 10, the operation input reception unit 12 turns off the power after the transmission/reception unit 11 transmits the state information indicating that the power is turned off to the communication management system 50. to As a result, the communication management system 50 side can grasp that the power of the communication terminal 10 has changed from ON to OFF.

The communication control unit 13 also performs various communication controls such as establishment and disconnection of a session for transmitting and receiving content data to and from another communication terminal 10 via the relay device 30 . The communication control unit 13 according to the present embodiment adds the communication ID (Identification) of the communication terminal 10 to session control information (for example, start request information, start response information, etc., which will be described later) to be transmitted to the communication management system 50. send with.

A communication ID is an example of identification information of an account that can participate in a session for transmitting and receiving content data using the communication terminal 10 . The communication ID may be, for example, a user ID that is identification information of an operator, an application ID that is identification information of an application, a contract ID that is identification information of a contractor of the communication terminal 10, or the like. Further, the communication ID includes information in which at least two of characters, numbers, symbols, and various signs are combined. An email address or the like may be used.

The imaging unit 14 converts imaging data obtained by imaging a subject into predetermined image (video) data and outputs the data. There is one imaging unit 14 on the front side of the communication terminal (capturing a front image) and one on the rear side (capturing a rear image). A larger number of imaging units may be provided.

After the operator's voice is converted into a voice signal by the microphone 114, the voice input unit 15a converts the voice signal into predetermined voice data and outputs the voice data. The audio output unit 15b converts the audio data into an audio signal and outputs the audio signal to the speaker, causing the speaker 115 to output audio.

The display control unit 16 causes the display unit 109, the display input unit 110, or the like to display image data included in the content data received by the communication terminal 10, for example. Further, the display control unit 16 can transmit the information of the destination list received from the communication management system 50 to the display unit 109 and display the destination list on the display unit 109 .

The storage/readout unit 17 stores various data in the storage unit 18 and reads out various data stored in the storage unit 18 .

The storage unit 18 stores, for example, the above-described communication IDs and authentication information such as passwords corresponding to the communication IDs. The storage unit 18 overwrites and stores image data and audio data that are received when making a call with the destination terminal each time they are received. Among them, an image is displayed on the display unit 109 or the like with the image data before being overwritten, and sound is output from the speaker 115 with the audio data before being overwritten.

Next, each functional configuration included in the communication terminal 10B will be described.

The omnidirectional image receiving unit 19b receives the equirectangular projection image from the wide-angle imaging device 9 by, for example, wireless communication such as Bluetooth (registered trademark) or wired communication such as a USB cable. An equirectangular projection image is a moving image that is repeatedly transmitted at a frequency that can be regarded as a moving image. However, it may be a still image, or it may be possible to switch between a moving image and a still image.

The operation instruction receiving unit 19a receives, via the transmission/reception unit 11, operation instruction information from the communication terminal 10A requesting the communication terminal 10B to control the device (the mobile device 20). This operation instruction information includes, for example, the communication ID of the communication terminal 10 that has transmitted the operation instruction information, an operation instruction indicating the content of the requested control, and the like.

The device control unit 19c controls the mobile device 20 based on the operation instruction included in the operation instruction information received by the operation instruction reception unit 19a. The device-to-device communication unit 19 d communicates with the mobile device 20 using the short-range wireless communication unit 118 .

<<Functional Configuration of Mobile Device>>
The mobile device 20 has, for example, an inter-device communication unit 21, a travel control unit 22, and the like. The device-to-device communication unit 21 is implemented, for example, by commands from the CPU 401 shown in FIG. Here, it is assumed that the device-to-device communication unit 21 uses the short-range wireless communication unit 405 to communicate with the communication terminal 10B.

The travel control unit 22 controls the wheel drive unit 406 and the steering unit 407 shown in FIG. Controls movement (driving) such as turning.

<<Functional configuration of wide-angle imaging device>>
The wide-angle imaging device 9 has, for example, a wide-angle image transmission section 31, a wide-angle image imaging section 32, and the like. The wide-angle image capturing unit 32 captures a 360-degree wide-angle equirectangular projection image at a predetermined frame rate. A still image may be captured. The wide-angle image transmission unit 31 is implemented by the communication unit 317 shown in FIG. 7, and transmits an equirectangular projection image of a moving image or a still image to the communication terminal 10B.

<< Functional configuration of communication management system >>
The communication management system 50 includes a transmission/reception section 51, a terminal authentication section 52, a terminal management section 53, a terminal extraction section 54, a session management section 55, a storage/read section 57, and the like. 5 operates according to instructions from the CPU 501 according to the program for the communication management system 50 developed on the RAM 503 from the HD 504. or a means of functioning. The communication management system 50 also has a storage unit 5000 implemented by the HD 504 or the like shown in FIG.

(Each functional configuration of the communication management system)
Next, each functional configuration of the communication management system 50 will be described in detail. The transmission/reception unit 51 transmits/receives various data (or information) to/from another communication terminal, device, or system via the communication network 2 .

Terminal authentication unit 52 determines whether or not the combination of communication ID and password included in the login request information received via transmission/reception unit 51 is included in authentication management DB (Database) 5002. The terminal 10 is authenticated.

The terminal management unit 53 stores and manages, in the terminal management DB 5003, the destination name, operating status, reception date and time of the request information, IP address of the requesting terminal, etc. for each communication ID in association with each other. For example, when the operator turns the power of the communication terminal 10 from ON to OFF, the terminal management unit 53 updates the terminal management DB 5003 based on the state information indicating that the power is to be turned OFF, which is sent from the communication terminal 10 . Change the operating state from ON line to OFF line.

The terminal extracting unit 54 searches the destination list management DB 5004 using the communication ID of the requesting terminal that requested the login as a key, and extracts the communication ID of the destination terminal that can communicate with the requesting terminal. Furthermore, the terminal extracting unit 54 searches the destination list management DB 5004 to extract the communication IDs of other communication terminals that have registered the communication ID of the requesting terminal as a destination terminal candidate.

Further, the terminal extracting unit 54 searches the above-described terminal management DB 5003 using the communication ID of the extracted candidate of the destination terminal as a search key, and reads the operating state for each extracted communication ID. As a result, the terminal management unit 53 can acquire the operation status of the destination terminal candidates that can communicate with the request source terminal that has made the login request. The terminal management unit 53 also searches the above-described terminal management DB 5003 using the communication ID of the requester as a search key, and acquires the operation status of the requestor terminal that has requested the login.

The session management unit 55 controls sessions managed by the communication management system 50 . The session control includes, for example, control for establishing a session, control for joining the communication terminal 10 to the established session, control for disconnecting the session, generation of a session ID, and the like. In addition, the session management unit 55 stores the communication ID of the communication terminal 10 requesting the start of the session, the communication ID of the destination terminal, etc. in the session management DB 5005 in association with the session ID, which is the identification information of the session. ,to manage.

The storage/reading unit 57 is implemented by the instructions from the CPU 501 and the HDD 505 shown in FIG. 5, or by the instructions from the CPU 501 . The storage/readout unit 57 stores various data in the storage unit 5000 and reads out various data from the storage unit 5000 .

<Example of information managed by the communication management system>
Each management DB stored in the storage unit 5000 of the communication management system 50 will be described.

通信管理システム５０の記憶部５０００に記憶された認証管理ＤＢ５００２には、例えば、表１に示されているような認証管理テーブル６０２が含まれる。この認証管理テーブル６０２では、通信管理システム５０によって管理される通信端末１０の通信ＩＤと、通信ＩＤに対応するパスワードとが関連付けられて管理されている。例えば、表１に示されている認証管理テーブル６０２において、通信ＩＤが「０１ａａ」の通信端末１０のパスワードは、「ａａａａ」であることが示されている。

The authentication management DB 5002 stored in the storage unit 5000 of the communication management system 50 includes, for example, an authentication management table 602 as shown in Table 1. In this authentication management table 602, communication IDs of communication terminals 10 managed by the communication management system 50 and passwords corresponding to the communication IDs are associated and managed. For example, in the authentication management table 602 shown in Table 1, the password of the communication terminal 10 with the communication ID "01aa" is "aaaa".

通信管理システム５０の記憶部５０００に記憶された端末管理ＤＢ５００３には、例えば、表２に示されているような端末管理テーブル６０３が含まれる。この端末管理テーブル６０３では、通信端末１０の通信ＩＤ毎に、各通信端末１０を宛先とした場合の宛先名、各通信端末１０の稼動状態、後述のログイン要求情報が通信管理システム５０で受信された受信日時、及び通信端末１０のＩＰアドレスが関連付けられて管理される。例えば、表２に示されている端末管理テーブル６０３において、通信ＩＤが「０１ａａ」の通信端末１０は、端末名が「日本 本社」で、稼動状態が「ＯＮライン（通信可能）」であることが示されている。また、通信ＩＤが「０１ａａ」の通信端末１０は、通信管理システム５０でログイン要求情報が受信された日時が「２０ｘｘ年４月１０日の１３時４０分」で、ＩＰアドレスが「１．２．１．３」であることが示されている。

The terminal management DB 5003 stored in the storage unit 5000 of the communication management system 50 includes, for example, a terminal management table 603 as shown in Table 2. In this terminal management table 603, the communication management system 50 receives, for each communication ID of the communication terminal 10, the destination name when each communication terminal 10 is the destination, the operation status of each communication terminal 10, and login request information described later. The received date and time and the IP address of the communication terminal 10 are associated and managed. For example, in the terminal management table 603 shown in Table 2, the communication terminal 10 with a communication ID of "01aa" has a terminal name of "Japan headquarters" and an operating state of "online (communicable)". It is shown. For the communication terminal 10 with the communication ID "01aa", the date and time when the login request information was received by the communication management system 50 was "13:40 on April 10, 20xx", and the IP address was "1.2 .1.3".

通信管理システム５０の記憶部５０００に記憶された宛先リスト管理ＤＢ５００４には、例えば、表３に示されているような宛先リスト管理テーブル７０１が含まれる。この宛先リスト管理テーブル７０１では、テレビ会議における通信の開始を要求する要求元端末の通信ＩＤに対して、宛先端末の候補として登録されている宛先端末の通信ＩＤが全て関連付けられて管理される。例えば、表３に示されている宛先リスト管理テーブル７０１において、通信ＩＤが「０１ａａ」である要求元端末から通信の開始を要求することができる宛先端末の候補は、通信ＩＤが「０１ｂ１」、「０１ｂ２」、及び「０１ｂ３」の通信端末である。この宛先端末の候補は、任意の要求元端末から通信管理システム５０に対する追加又は削除の要請により、通信管理システム５０によって追加又は削除されることで更新される。

The destination list management DB 5004 stored in the storage unit 5000 of the communication management system 50 includes, for example, a destination list management table 701 as shown in Table 3. In this destination list management table 701, all the communication IDs of the destination terminals registered as destination terminal candidates are associated with the communication ID of the request source terminal requesting the start of communication in the video conference and managed. For example, in the destination list management table 701 shown in Table 3, candidate destination terminals that can be requested to start communication from a request source terminal with a communication ID of "01aa" are: These are the communication terminals "01b2" and "01b3". This destination terminal candidate is updated by being added or deleted by the communication management system 50 in response to an addition or deletion request to the communication management system 50 from any request source terminal.

According to such a configuration, the request source terminal (for example, 01aa) can start communication only with a pre-registered destination terminal candidate (for example, 01b1). This destination terminal (eg, 01b1) cannot communicate with the requesting terminal unless the requesting terminal (eg, 01aa) is registered in the destination list management table 701 as a destination terminal. Such a mechanism is preferable in that it can reduce the possibility that unexpected communication terminals 10 communicate with each other. However, the communication terminals 10 may arbitrarily communicate with each other without registration in the destination list management table 701 .

通信管理システム５０の記憶部５０００に記憶されたセッション管理ＤＢ５００５には、例えば、表４に示されているようなセッション管理テーブル７０２が含まれる。このセッション管理テーブル７０２には、セッションの識別情報であるセッションＩＤ毎に、中継に使用される中継装置３０の中継装置ＩＤ、要求元端末の通信ＩＤ、宛先端末の通信ＩＤ、及びセッション参加日時等の情報が管理される。例えば、表４に示されているセッション管理テーブル７０２において、セッションＩＤ「ｓｅ２」のセッションは、要求元端末の通信ＩＤ「０１ａｄ」と宛先端末の通信ＩＤ「０１ｃａ」との間で行われていることが示されている。また、セッションＩＤ「ｓｅ２」のセッションは、中継装置ＩＤ「１１１ｂ」の中継装置３０を介して、「２０ｘｘ／４／１０ １３：１１：１１」に開始されたことが示されている。

The session management DB 5005 stored in the storage unit 5000 of the communication management system 50 includes, for example, a session management table 702 as shown in Table 4. In this session management table 702, the relay device ID of the relay device 30 used for relay, the communication ID of the request source terminal, the communication ID of the destination terminal, session participation date and time, etc. are stored for each session ID, which is session identification information. information is managed. For example, in the session management table 702 shown in Table 4, the session with the session ID "se2" is between the communication ID "01ad" of the request source terminal and the communication ID "01ca" of the destination terminal. is shown. It also shows that the session with the session ID "se2" was started at "20xx/4/10 13:11:11" via the relay device 30 with the relay device ID "111b".

<Process flow>
Next, the processing flow of the communication system 1 will be described.

(Preparatory stage processing)
FIG. 15 is a sequence diagram showing an example of processing in the communication preparation stage of the communication system 1. As shown in FIG. Here, as an example, processing in a preparatory stage before starting a session between the communication terminal 10A and the communication terminal 10B1 will be described. In the following description, it is assumed that the communication ID of communication terminal 10A is "01aa" and the communication ID of communication terminal 10B1 is "01b".

First, when the operator of the communication terminal 10A, which is the request source terminal, performs an operation to turn on the power of the communication terminal 10A, for example, the operation input reception unit 12 receives the operation to turn on the power, and the communication terminal 10A is turned on. The power is turned on (step S21).

When the power is turned on, the communication control unit 13 transmits login request information for requesting login from the transmission/reception unit 11 to the communication management system 50 via the communication network 2 (step S22). It is an example that the login request information is transmitted by the operation of turning on the power of the communication terminal 10A. can be

Further, the login request information includes a communication ID (communication ID of the request source terminal) and a password for identifying the communication terminal 10A as the request source. These communication ID and password are, for example, information read from the storage unit 18 via the storage/readout unit 17 . Further, when the login request information is transmitted from the communication terminal 10A to the communication management system 50, the communication management system 50 on the receiving side can grasp the IP address of the communication terminal 10A on the transmitting side.

Next, the terminal authentication unit 52 of the communication management system 50 searches the above-described authentication management table 602 using the communication ID and password included in the login request information received via the transmission/reception unit 51 as search keys. The terminal authentication unit 52 performs authentication based on whether the combination of the communication ID and password included in the login request information received from the communication terminal 10A is included in the authentication management table 602 (step S23).

When the terminal authentication unit 52 determines that the login request is from the communication terminal 10A having valid use authority, the terminal management unit 53 recognizes the communication ID "01aa" of the communication terminal 10A recorded in the terminal management table 603. " is changed to "ON LINE (Communication enabled)". At this time, the terminal management unit 53 updates the date and time of reception, and if necessary, updates the IP address of the communication terminal 10 (step S24). As a result, in the terminal management table 603, the communication ID "01aa" of the communication terminal 10A, the operation status "ON line (communication possible)", the reception date and time "20xx.4.10.13:40", and the communication terminal 10A The IP address "1.2.1.3" is associated and managed.

The transmitting/receiving unit 51 of the communication management system 50 is the request source terminal that has sent the authentication result information showing the authentication result obtained by the terminal authentication unit 52 via the communication network 2. It is transmitted to the communication terminal 10A (step S25). Here, a case in which the terminal authentication unit 52 determines that the communication terminal has valid use authority will be described below.

The terminal extraction unit 54 of the communication management system 50 searches the destination list management table 701 using the communication ID "01aa" of the request source terminal (communication terminal 10A) that requested the login as a search key. As a result, the terminal management unit 53 extracts the communication ID of the destination terminal candidate that can communicate with the request source terminal (communication terminal 10A) (step S26). Here, as an example, it is assumed that "01b1", "01b2", and "01b3" are extracted as the communication IDs of the destination terminals corresponding to the communication ID "01aa" of the request source terminal (communication terminal 10A).

Next, the terminal extraction unit 54 searches the terminal management table 603 using the communication IDs (“01b1”, “01b2”, “01b3”) of the extracted destination terminal candidates as search keys. Accordingly, by reading the operating status for each extracted communication ID, the operating status of each communication ID (“01b1”, “01b2”, “01b3”) is obtained (step S27).

Next, the transmitting/receiving unit 51 transmits the destination state information including the operating state of each of the communication IDs (“01b1”, “01b2”, “01b3”) of the destination terminal candidates to the request source terminal (communication terminal 10A). (step S28). As a result, the request source terminal (communication terminal 10A) confirms the current operating status of the communication IDs (“01b1”, “01b2”, “01b3”) that are candidates for the destination terminal of this request source terminal (communication terminal 10A). can grasp. Communication terminal 10A displays a destination selection screen shown in FIG.

Further, the terminal extracting unit 54 of the communication management system 50 searches the destination list management table 701 using the communication ID "01aa" of the request source terminal (communication terminal 10A) that has made the login request as a search key. As a result, the terminal extracting unit 54 extracts the communication IDs of other requesting terminals in which the communication ID "01aa" of the requesting terminal (communication terminal 10A) is registered as a destination terminal candidate (step S29). In the destination list management table 701 shown in Table 3, the extracted communication IDs of the other requesting terminals are "01b1", "01b2" and "01b3".

Next, the terminal management unit 53 of the communication management system 50 searches the terminal management table 603 using the communication ID "01aa" of the request source terminal (communication terminal 10A) that has sent the login request as a search key. As a result, the terminal management unit 53 acquires the operating state of the request source terminal (communication terminal 10A) that has issued the login request (step S30).

Then, the terminal management unit 53 of the communication management system 50 determines that the operation status in the terminal management table 603 is "ON LINE ( communication possible)” (“01b1”, “01b2”, “01b3”) are extracted.

Further, the transmitting/receiving unit 51 sends the communication ID "01aa" of the requesting terminal (communication terminal 10A) and the operating state to the communication terminal 10B1 corresponding to the extracted communication IDs ("01b1", "01b2", "01b3"). Destination status information including "on line (communication possible)" is transmitted (step S31).

On the other hand, the other communication terminal 10B1 also performs the same processing as the processing of steps S22 to S32 in response to the operation of turning on the power, for example. The communication terminal 10B1 is turned on by, for example, an administrator of the mobile device 20 or the like.

<Destination selection screen>
FIG. 16 is a diagram showing an example of a destination selection screen displayed on communication terminal 10A. A destination selection screen 1201 shown in FIG. 16 displays a message 1202 prompting the operator to select the destination communication terminal 10 and a plurality of buttons 1203 for selecting the destination communication terminal 10 . The operator of the communication terminal 10A selects a destination terminal requesting participation in the session, for example, by selecting one button 1203 out of the plurality of buttons 1203 displayed. That is, which mobile device 20 is to be controlled is selected. As shown in FIG. 16, since the operator can select a plurality of communication terminals 10B, the operator can grasp the situation of different bases from the same place.

<Communication processing>
FIG. 17 is a sequence diagram showing an example of communication processing of the communication system 1. As shown in FIG. Here, an example of a communication management method for starting communication between the communication terminal 10A and the communication terminal 10B1, which is a device control terminal capable of controlling the mobile device 20, will be described.

In step S901, the operation input receiving unit 12 of the communication terminal 10A receives a selection operation of the destination terminal (communication terminal 10B1) by the operator of the communication terminal 10A.

The transmission/reception unit 11 of the communication terminal 10A transmits start request information for requesting the start of a session to the communication management system 50 (step S902). This start request information includes, for example, the communication ID of the requesting terminal, which is the communication ID of the communication terminal 10A, which is the requesting terminal, and the communication ID of the destination terminal, which is the communication ID of the communication terminal 10B1 which is the destination terminal. . The start request information also includes information such as the IP address (request source IP address) of the communication terminal 10A.

In step S903, the terminal management unit 53 of the communication management system 50 that has received the start request information from the communication terminal 10A, based on the communication ID "01aa" of the request source terminal (communication terminal 10A) included in the start request information, Update the management DB 5003 . For example, the terminal management unit 53 changes the operating state information corresponding to the communication ID "01aa" of the communication terminal 10A to "online (in communication)" and also updates the reception date and time information.

In step S904, the session management unit 55 of the communication management system 50 transmits start request information requesting the start of a session to the communication terminal 10B1, which is the destination terminal. This start request information includes, for example, the communication ID of the request source terminal of the communication terminal 10A, which is the request source terminal.

In step S 905 , the communication terminal 10 B 1 that has received the start request information from the communication management system 50 transmits start response information to the communication management system 50 . This start response information includes, for example, the communication ID of the destination terminal of the communication terminal 10B1. In the present embodiment, it is assumed that the start response information is transmitted without requiring any operation on the communication terminal 10B1 side, but the condition may be that the administrator operates the communication terminal 10B1.

In step S906, the terminal management unit 53 of the communication management system 50 that has received the start response information from the communication terminal 10B1 updates the terminal management DB 5003 based on the communication ID "01b1" of the communication terminal 10B1 included in the start response information. . For example, the terminal management unit 53 changes the operating state information corresponding to the communication ID "01b1" of the communication terminal 10B1 to "online (communicating)" and updates the reception date and time information.

In step S907, the session management unit 55 of the communication management system 50 numbers a session ID, which is identification information for identifying a session. Also, the session management unit 55 associates the created session ID with the communication ID of the request source terminal (the communication ID of the communication terminal 10A) and the communication ID of the destination terminal (the communication ID of the communication terminal 10B1) to manage the session. Store in DB5005.

In step S<b>909 , the session management unit 55 of the communication management system 50 transmits session information to the relay device 30 . This session information includes, for example, information such as the session ID created in step S907. The relay device 30 can acquire session information from the session management DB 5005 based on the session ID.

In step S910a, the session management unit 55 of the communication management system 50 transmits start instruction information for instructing the communication terminal 10A to start a session. Similarly, in step S910b, the session management unit 55 of the communication management system 50 transmits start instruction information for instructing the start of a session to the communication terminal 10B1. A session ID is included in the start instruction information, and the communication terminal 10 can acquire session information from the session management DB 5005 based on the session ID.

In step S911a, communication terminal 10A establishes a session with relay device 30 based on the received start instruction information. Similarly, in step 911b, communication terminal 10B1 establishes a session with relay device 30 based on the received start instruction information. This allows the communication terminals 10A and 10B1 to participate in the same session.

In step S911c, the wide-angle image transmission unit 31 of the wide-angle imaging device 9 transmits the equirectangular projection image to the communication terminal 10B1. For example, when the communication terminal 10B1 establishes a session, the communication terminal 10B1 instructs the wide-angle imaging device 9 to start capturing and transmitting an equirectangular projection image. Alternatively, while the wide-angle imaging device 9 is powered on, the wide-angle imaging device 9 may always transmit the equirectangular projection image to the communication terminal 10B1. Also, the equirectangular projection video may be transmitted before the session is established.

The communication terminal 10A and the communication terminal 10B1 participate in a session with the same session ID, and transmit and receive content data such as image data and audio data, thereby enabling, for example, a video conference (S912).

Also, the communication terminal 10A can transmit and receive information regarding the operation of a device (for example, the mobile device 20) with the communication terminal 10B1 using the established session.

It should be noted that the transmission and reception of information related to the operation of the device may be transmitted and received using a control session via the communication management system 50 without using a session, or via the communication network 2 or the like. It may be transmitted and received between 10A and communication terminal 10B1.

Here, the following description is given assuming that the communication terminal 10A uses the established session to transmit and receive information regarding the operation of a device (for example, the mobile device 20) with the communication terminal 10B1.

In step S913, when the operator of the communication terminal 10A performs input on the operation screen of the device, operation instruction information corresponding to the input operation is transmitted using the session. This operation instruction information includes, for example, the communication ID of the communication terminal 10A and an operation instruction according to the operation content of the operator.

In step S914, the device control unit 19c of the communication terminal 10B1 controls the mobile device 20 via the device-to-device communication unit 19d based on the operation instruction included in the notified operation instruction information.

<Pattern 1>
Next, for each of the patterns 1 to 3, a display example will be described in which the operator can grasp which direction the mobile device 20 is facing when the operator is looking at a certain direction.

First, pattern 1 will be described with reference to FIGS. 18 to 21. FIG. FIG. 18 is an example of a diagram for explaining a reference imaging direction that serves as a reference for the imaging direction and the front direction of the moving device 20. As shown in FIG. Since the wide-angle imaging device 9 is fixed to the mobile device 20 , the front direction of the mobile device 20 can be represented by the imaging direction of the wide-angle imaging device 9 . The imaging direction that is the same as the front direction of the mobile device 20 is defined as a reference imaging direction (ea ₀ , aa ₀ ).

When the imaging direction viewed by the operator is the reference imaging direction, the composition is such that the operator views the front of the telepresence robot from the back of the telepresence robot.

Next, FIG. 19 is a diagram showing the relationship between an arbitrary imaging direction (ea, aa) displayed by the operator and the reference imaging direction (ea ₀ , aa ₀ ). Since the reference imaging direction (ea ₀ , aa ₀ ) is fixed, the difference (Δea, Δaa) in the imaging direction (ea, aa) with respect to the reference imaging direction (ea ₀ , aa ₀ ) is as follows.
Δea = ea ₀ - ea
Δaa = aa ₀ - aa
Only the horizontal direction is considered for ease of explanation. When the imaging direction is changed by Δaa from the reference imaging direction, it is rotated by Δaa as seen from the telepresence robot. For example, when the imaging direction increases by 90 degrees in the horizontal direction, the telepresence robot rotates in the direction to decrease by 90 degrees in the horizontal direction. When the imaging direction is horizontally reduced by 90 degrees, the telepresence robot rotates horizontally by 90 degrees. The center of rotation may be the center of gravity of the telepresence robot.

Therefore, the display control unit 16 rotates the telepresence robot by (-Δea, -Δaa) and displays it. (−Δea,−Δaa) may be calculated not by the difference from the reference imaging direction (ea ₀ ,aa ₀ ) but by the difference from the previous imaging direction (ea, aa).

The display control unit 16 rotates a prepared polygonal model of the telepresence robot in the horizontal direction and the elevation angle direction. Coordinate transformation is performed to rotate around two axes of an orthogonal coordinate system passing through the center of the polygon model of the telepresence robot.

FIG. 20 is an example of a diagram schematically explaining perspective projection transformation. Perspective projection transformation is transformation that projects an object onto a projection plane 830 perpendicular to the Z-axis, with the viewpoint on the Z-axis. The objects in this embodiment are the polygon model of the telepresence robot and the predetermined area image Q of the omnidirectional image. The origin O is the center of the solid sphere as shown in FIG. The projection plane 830, the telepresence robot polygon model 839, and the predetermined area image Q are arranged in order from the origin O. FIG. Although the polygon model 839 of the telepresence robot is not arranged in normal perspective projection transformation for displaying the omnidirectional video on the display unit 109, in pattern 1, the polygon model 839 of the telepresence robot is arranged on the Z axis. . The distance from projection plane 830 may be determined experimentally. The wide-angle imaging device 9 is mounted on the upper side of the communication terminal 10A, and since the omnidirectional image is captured by the wide-angle imaging device 9, the polygon model 839 closer to the projection surface 830 is the viewpoint and the wide-angle imaging device 9. less deviation.

Note that the polygon model 839 of the telepresence robot does not have to be on the Z-axis. For example, it may be arranged so as to be projected on one of the four corners of the predetermined area image Q. FIG.

The polygonal model 839 of the telepresence robot is rotated in the local coordinate system due to the difference (Δea, Δaa) in the imaging direction (ea, aa) from the reference imaging direction (ea ₀ , aa ₀ ), as described with reference to FIG. there is After that, it is arranged in a coordinate system for projecting an omnidirectional image as shown in FIG. As a result, the direction in which the mobile device 20 is facing in the omnidirectional video can be displayed in the image of the telepresence robot.

FIG. 21 is an example of an omnidirectional image display screen 1010 displayed in pattern 1. As shown in FIG. In the omnidirectional image display screen 1010 of FIG. 21(a), the communication terminal 10A displays the omnidirectional image in the reference imaging direction. Therefore, the image 840 of the telepresence robot is facing the operator. Next, the operator rotated the omnidirectional image to the right. For convenience of explanation, it is assumed that the image is rotated 90 degrees in the horizontal direction. During this time, the moving device 20 is not moving or turning.

FIG. 21(b) shows an image 840 of the telepresence robot after the operator rotates the omnidirectional video to the right. Since the telepresence robot in the real space faces 90 degrees to the left with respect to the imaging direction of the omnidirectional video in FIG. 21(b), the image 840 of the telepresence robot also faces 90 degrees to the left. The image 840 of the telepresence robot changes its direction according to the amount of change when the predetermined area image Q changes.

Therefore, even if the operator arbitrarily rotates the omnidirectional image, the operator can easily grasp which direction the telepresence robot in the real space is facing.

FIG. 22 is an example of a flowchart showing a procedure for displaying a predetermined area image on display unit 109 by communication terminal 10A. The processing in FIG. 22 is executed each time the user changes the imaging direction.

First, the operation input reception unit 12 receives an operation of the imaging direction by the operator (S1001).

Next, the display control unit 16 calculates the difference (Δea, Δaa) in the imaging direction with respect to the reference imaging direction (S1002). Instead of using the reference imaging direction as a reference, the difference from the immediately preceding imaging direction may be calculated.

Next, the display control unit 16 rotates the polygon model of the telepresence robot by (−Δea, −Δaa) opposite to the difference (S1003).

Then, the display control unit 16 performs perspective projection conversion on the predetermined area image and the polygon model and displays them (S1004).

The image of the telepresence robot is preferably translucent. It is also preferred that the degree of translucency be determined by the operator. Furthermore, it is preferable that the operator can set display or non-display of the image of the telepresence robot. As a result, it is possible to prevent the image of the telepresence robot from obstructing viewing of the omnidirectional video.

<Pattern 2>
Next, a method of displaying the orientation of the telepresence robot in pattern 2 will be described. Pattern 2 is a display method in which, apart from the omnidirectional video viewed by the operator, the direction of the omnidirectional video viewed from the telepresence robot is displayed by the telepresence robot and the imaging direction arrow 8. is.

FIG. 23 shows an example of a polygon model of a telepresence robot and an object of an imaging direction arrow. A polygon model 850 of the telepresence robot is prepared in advance. A polygonal model 850 of the telepresence robot is arranged in a predetermined coordinate system. The imaging direction arrow object 851 is also formed of polygons. Since the imaging directions determined by the operator are represented by ea and aa, the display control unit 16 generates an imaging direction arrow object 851 of a predetermined length indicating the imaging directions ea and aa. For example, an object 851 having an imaging direction arrow prepared in advance is arranged in the imaging directions ea and aa.

As illustrated, the display control unit 16 aligns the arrowhead of the imaging direction arrow object 851 with the wide-angle imaging device 9 and arranges them in a predetermined coordinate system. Thereafter, the image may be arranged on the display unit 109 after perspective projection conversion. As a result, the imaging direction arrow 8 can be displayed stereoscopically. The imaging direction arrow 8 serves as information indicating in which direction the predetermined area image Q is viewed from the mobile device 20 . Therefore, the imaging direction arrow 8 is an image of an arrow whose starting point is the telepresence robot image 854 (mobile device 20).

The image 854 of the telepresence robot and the imaging direction arrow 8 are displayed side by side with the predetermined area image 853 as shown in FIG. 2(b). However, in order to cope with a case where the screen size of the display unit 109 is small, the display control unit 16 may display only one of them. For example, when the operator drags the predetermined area image 853 with a mouse or swipes with a finger, an image 854 of the telepresence robot and the imaging direction arrow 8 appear.

Further, in FIG. 23, perspective projection may be performed after the object 851 of the telepresence robot is rotated according to the imaging direction. This is because the viewpoint from which the imaging direction arrow 8 can be easily seen differs depending on the imaging direction. For example, the object 851 of the telepresence robot in the case of the reference imaging direction (the imaging direction arrow 8 in the front direction is at an easy-to-see horizontal angle) is horizontally rotated by Δea. This makes it easier for the operator to grasp where the imaging direction arrow 8 is directed regardless of the imaging direction.

As shown in FIG. 24, in pattern 2, the imaging direction arrow 8 may be displayed separately for the horizontal direction and the elevation angle direction. FIG. 24 shows another display example of the imaging direction arrow 8 in pattern 2. In FIG. The omnidirectional image display screen 1010 in FIG. 24A has an omnidirectional image field 1001 , a horizontal direction instruction field 1003 , and an elevation direction instruction field 1004 . The horizontal direction indication column 1003 has an image 855 of the telepresence robot viewed from above, and only the horizontal direction among the imaging directions is indicated by the horizontal direction arrow 8a. The elevation direction indication field 1004 has an image 854 of the telepresence robot, and only the elevation direction of the imaging directions is indicated by the elevation direction arrow 8b.

Displaying the horizontal direction and the elevation angle direction separately makes it easier for the operator to determine how much of the imaging direction is in the horizontal direction and how much is in the elevation direction.

In FIG. 24A, the horizontal direction of the imaging directions is included in the front 180 degrees, so the elevation direction arrow 8b of the elevation direction instruction field 1004 is displayed in front of the telepresence robot. On the other hand, in FIG. 24B, since the horizontal direction of the imaging direction is included in the rearward 180 degrees, the elevation direction arrow 8b in the elevation direction indication field 1004 is displayed behind the telepresence robot. By switching the display position of the elevation direction arrow in this way, the operator can intuitively grasp whether the horizontal direction is forward or backward.

The images 854 and 855 of the telepresence robot, the horizontal direction arrow 8a, and the elevation direction arrow 8b in FIG. can be reduced.

FIG. 25 is an example of a flowchart showing a procedure for communication terminal 10A to display an omnidirectional image on display unit 109 in pattern 2. As shown in FIG. The processing in FIG. 25 is executed each time the user changes the imaging direction.

First, the operation input reception unit 12 receives an operation of the imaging direction by the operator (S2001).

Next, the display control unit 16 performs perspective projection conversion on the predetermined area image and displays it (S2002).

Next, the display control unit 16 determines the viewpoint of the telepresence robot with respect to the polygon model 850 according to the imaging direction (S2003).

Next, the display control unit 16 arranges the imaging direction arrow object 851 in a predetermined coordinate system (S2004). That is, the display control unit 16 aligns the arrowhead of the object 851 of the imaging direction arrow 8 with the wide-angle imaging device 9, and arranges the object of the telepresence robot in the coordinate system where the object is arranged.

The display control unit 16 performs perspective projection transformation on the polygon model 850 of the telepresence robot and the imaging direction arrow object 851, and displays them on the display unit 109 (S2005).

<Pattern 3>
Pattern 3 displays the operation buttons only when the imaging direction and the front direction of the mobile device 20 substantially match. That is, the display control unit 16 displays the operation button 860 only when the imaging direction (ea, aa)≈(ea ₀ , aa ₀ ). See FIG. 2(c) for the omnidirectional image display screen 1010 of pattern 3. FIG.

FIG. 26 is an example of a flowchart showing a procedure for communication terminal 10A to display an omnidirectional image on display unit 109 in pattern 3. As shown in FIG. The processing in FIG. 26 is executed each time the user changes the imaging direction.

First, the operation input reception unit 12 receives an operation of the imaging direction by the operator (S3001).

Next, the display control unit 16 performs perspective projection conversion on the predetermined area image Q and displays it (S3002).

Next, the display control unit 16 determines whether or not the imaging direction substantially matches the reference imaging direction (S3003).

If the determination in step S3003 is Yes, the display control unit 16 displays the operation button 860 superimposed on the omnidirectional image (S3004).

If the determination in step S3003 is No, the display control unit 16 does not display the operation button 860 in the omnidirectional video.

Therefore, since the operation button 860 is displayed only when the front direction of the mobile device 20 and the imaging direction of the omnidirectional image viewed by the operator are substantially matched, where will the mobile device 20 move when the mobile device 20 is moved forward? The operator can grasp whether Also, when the operation button 860 is displayed, the orientation of the mobile device 20 can be grasped.

Although the operation button 860 is not displayed at all in the left diagram c-1 of FIG. 2(c), when the front direction of the mobile device 20 and the imaging direction of the omnidirectional video viewed by the operator do not substantially match. Additionally, the operation button 860 may be displayed in a manner that the operator cannot operate it. The mode in which the operation cannot be performed means, for example, displaying with low luminance, displaying in gray, or displaying in semi-transparency (increasing transparency). Even if the operator presses the operation button 860, the operation input accepting unit 12 does not accept the operation.

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is by no means limited to such examples, and various modifications can be made without departing from the scope of the present invention. and substitutions can be added.

For example, although the front image (or rear image) captured by the front imaging device is not described in the present embodiment, the front image (or rear image) is appropriately displayed in each of patterns 1 to 3.

FIG. 27 is an example of an image simultaneous display screen 2001 that simultaneously displays an omnidirectional image and a front image (or rear image). The image simultaneous display screen 2001 has an entire display column 2002 and a front display column 2003 . An image of one of patterns 1 to 3 is displayed in the entire display column 2002 . The front display field 2003 displays a front image (or rear image) captured by the front imaging device. Therefore, the operator can display an arbitrary direction in the entire display field 2002 while always confirming the front or the back.

Further, in the present embodiment, the wide-angle imaging device 9 transmits the equirectangular projection image to the communication terminal 10B, and the communication terminal 10B transmits the equirectangular projection image to the communication terminal 10A. A session may be established with the terminal 10A to transmit the equirectangular projection image. In this case, the wide-angle imaging device 9 becomes one form of the communication terminal 10, and becomes the communication terminal 10 that transmits but does not receive video.

Further, although it has been described that the rear image is transmitted in the present embodiment, the communication terminal 10B does not have to transmit the rear image. Also, the communication terminal 10B may transmit an image with a normal angle of view separately from the front image and the rear image.

Further, in the present embodiment, the mobile device 20 that travels on land has been described as an example, but the mobile device 20 may fly in the air like a drone. Also, it may navigate on the sea or move in the water. It may also move underground, narrow passages, and underground. When moving on land, it may move on wheels, may move on multiple legs such as two, three, or four legs, or may move on caterpillars (registered trademark).

In addition, the display method of the present embodiment may be applied not only to telepresence robots, but also to images captured by the wide-angle imaging device 9 and the planar imaging device mounted on a device that is moved by humans.

Also, in the present embodiment, an example in which one operator operates one mobile device 20 has been mainly described, but it is also possible for one operator to operate a plurality of mobile devices 20 . In this case, the communication terminal 10A establishes sessions with a plurality of communication terminals 10B. Conversely, it is also possible for a plurality of operators to operate one mobile device 20 . In this case, for example, the operation instruction information last transmitted to the communication terminal 10B may be valid, or the concept of operation authority may be introduced so that only the communication terminal 10A having the operation authority can transmit the operation instruction. . The operation right can be transferred by the operation of the operator or the like.

Further, although it has been described that the operation instruction information is transmitted in a communication session in the present embodiment, it may be transmitted in a content data session.

Also, in this embodiment, the communication terminal 10 communicates via the relay device 30, but the communication terminal 10 may communicate without the relay device. For example, WebRTC (Web Real-Time Communication) is known as a communication protocol for such communication.

Further, the configuration examples in FIG. 14 and the like are divided according to main functions in order to facilitate understanding of processing by the communication management system 50 and the communication terminal 10 . The present invention is not limited by the division method or name of the unit of processing. The processing of the communication management system 50 and the communication terminal 10 can also be divided into more processing units according to the content of the processing. Also, one processing unit can be divided to include more processing.

Further, in this embodiment, for convenience of explanation, the communication management system 50 and the relay device 30 are described as separate devices, but a device in which the functions of both are integrated provides the functions of the communication management system 50 and the relay device 30. may

Further, the communication system 1 may have a plurality of communication management systems 50, and the functions of the communication management system 50 may be distributed and installed in a plurality of servers.

Moreover, one or more of the databases that the communication management system 50 has in the storage unit 5000 may exist on the communication network 2 .

The transmission/reception unit 11 is an example of communication means, the display control unit 16 is an example of display processing means, and the operation input reception unit 12 is an example of reception means.

REFERENCE SIGNS LIST 1 communication system 9 wide-angle imaging device 10 communication terminal 20 mobile device 50 communication management system

JP 2013-112030 A

Claims (11)

A communication terminal that communicates with another communication terminal mounted on a mobile device via a network and transmits operation instruction information for the mobile device to the other communication terminal,
a communication means for receiving a wide-angle video wider than a predetermined one captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal;
Display processing for displaying, on a display device, a predetermined area image displayed within an area of a display device of the communication terminal and information about the direction of the mobile device, from among a portion of the wide-angle image received by the communication means. means and
has
The display processing means displays information indicating in which direction the predetermined area image is viewed from the mobile device together with the image of the mobile device, separately from the predetermined area image. terminal. 2. The communication terminal according to claim 1 , wherein said display processing means superimposes an image of said mobile device represented in a manner in which the orientation is known on said predetermined area image. The image of the moving device, which is displayed in a manner in which the orientation can be recognized, changes in orientation according to the amount of change when the predetermined area image displayed on the display device among the wide-angle images changes. 3. The communication terminal according to item 1 or 2. The display processing means performs perspective projection transformation on the wide-angle image pasted on the solid sphere and the model of the moving device represented in a manner in which the orientation is known, so that the predetermined area image is displayed in a manner in which the orientation is known. 4. A communication terminal according to claim 2 or 3, characterized in that it displays an image of the represented mobile device superimposed. 2. The communication terminal according to claim 1 , wherein the information indicating in which direction the image of the predetermined area is viewed from the mobile device is an image of an arrow starting from the image of the mobile device. . The display processing means performs perspective projection transformation on a model of the moving device and a model of arrows indicating in which direction the predetermined region video is viewed from the moving device, so that the predetermined region video is: 6. The communication terminal according to claim 5 , wherein information indicating the direction of the image as viewed from the mobile device is displayed. Information indicating in which direction the image of the predetermined area is viewed from the mobile device is an image of an arrow in each of the horizontal direction and the elevation angle direction with the image of the mobile device as a starting point. A communication terminal according to claim 1 . The information about the orientation of the mobile device is an operation button for accepting the operation instruction information, which is displayed when the center direction of the predetermined area image substantially matches the front direction of the mobile device. The communication terminal according to claim 1. a communication terminal that communicates with another communication terminal mounted on a mobile device via a network and transmits operation instruction information for the mobile device to the other communication terminal;
a communication means for receiving a wide-angle image wider than a predetermined angle captured by a wide-angle imaging device connected to the mobile device or the other communication terminal;
Display processing for displaying, on a display device, a predetermined area image displayed within an area of a display device of the communication terminal and information about the direction of the mobile device, from among a portion of the wide-angle image received by the communication means. means and
has
The display processing means is characterized in that, in addition to the predetermined area image, information indicating the direction of the image of the predetermined area as viewed from the mobile apparatus is displayed together with the image of the mobile apparatus . program for. A display method performed by a communication terminal that communicates with another communication terminal mounted on a mobile device via a network and transmits operation instruction information for the mobile device to the other communication terminal,
a step in which the communication means receives a wide-angle image wider than a predetermined angle captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal;
A display processing means displays a predetermined area image displayed within an area of a display device of the communication terminal and information about the orientation of the mobile device, from among the part of the wide-angle image received by the communication means. and the steps to display in
has
The display processing means has a step of displaying, in addition to the image of the predetermined area, information indicating in which direction the image of the predetermined area is viewed from the mobile apparatus together with the image of the mobile apparatus. display method. a communication terminal that communicates with another communication terminal mounted on a mobile device via a network and transmits operation instruction information for the mobile device to the other communication terminal;
a communication means for receiving a wide-angle video wider than a predetermined one captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal;
Display processing for displaying, on a display device, a predetermined area image displayed within an area of a display device of the communication terminal and information about the direction of the mobile device, from among a portion of the wide-angle image received by the communication means. having means and
The display processing means is characterized in that, apart from the image of the predetermined area, information indicating the direction of the image of the image of the predetermined area as viewed from the mobile device is displayed together with the image of the mobile device. A computer-readable recording medium that stores a program for functioning.

Images