JP7346830B2 - Communication terminals, programs, display methods, recording media, systems - Google Patents

Description

The present invention relates to a communication terminal, a program, a display method, a recording medium, and a system .

2. Description of the Related Art Communication systems in which communication terminals communicate between a plurality of bases and send and receive video and audio to and from each other via communication networks such as the Internet and LAN (Local Area Network) have become widespread. Communication systems allow users in remote locations to communicate, for example to hold video conferences.

Furthermore, telepresence robots are known in which a communication system is applied to the robot. A telepresence robot is a robot that combines "video conferencing" and "remote control technology." This is a technology that combines remote control from a remote location and robot technology, allowing an operator to operate a moving device that moves the robot from a distance, making it behave as if the operator were present at a certain location.

Furthermore, a technique for mounting multiple cameras on a robot has been devised (for example, see Patent Document 1). Patent Document 1 discloses a moving body that is equipped with a front camera and a rear camera that take pictures of the front and back of a traveling device, and an overhead camera that can take pictures of the entire surrounding area of the robot.

However, the conventional technology has a problem in that it does not take into consideration the controller that controls the mobile device. That is, when a plurality of videos are transmitted from a mobile device, there is no disclosure as to how to display the controller while displaying the videos.

SUMMARY OF THE INVENTION In view of the above problems, an object of the present invention is to provide a communication terminal that allows an operator to control a robot while viewing images from a plurality of imaging devices transmitted from a mobile device.

In view of the above-mentioned problems, the present invention provides operation instruction information for the mobile device that communicates with another communication terminal installed in a mobile device or a communication unit included in the mobile device via a network, and that corresponds to an input operation from an operator. a first video transmitted by the other communication terminal or the communication unit, and a communication terminal that is connected to or connected to the other communication terminal or the mobile device. a communication means for receiving a wide-angle second image captured by an integrated wide-angle imaging device; and controlling to display both the first image and the second image received by the communication means on a display screen. and display processing means for displaying a controller that controls the movement of the mobile device over the first image displayed on the display screen, the controller controlling the movement of the mobile device from above the display screen. It is characterized by accepting the input operation.

It is possible to provide a communication terminal that displays images of a plurality of imaging devices transmitted from a mobile device in a manner that is easy for an operator to view.

1 is an example of a diagram illustrating an outline of a communication system. FIG. 4 is a diagram illustrating a display example of a front image or a rear image captured by a flat imaging device and a spherical image captured by a wide-angle imaging device. 1 is a diagram illustrating a configuration example of a communication system according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a communication terminal according to an embodiment. 1 is a diagram showing an example of a hardware configuration of a communication management system. FIG. 2 is a diagram illustrating an example of a hardware configuration of a mobile device. 1 is a hardware configuration diagram of an example of a wide-angle imaging device. It is an example of the usage image diagram of a wide-angle imaging device. FIG. 2 is a diagram illustrating an outline of processing until an equirectangular cylindrical projection image and a celestial sphere image are created from an image captured by a wide-angle imaging device. FIG. 2 is a diagram illustrating an outline of processing until an equirectangular cylindrical projection image and a celestial sphere image are created from an image captured by a wide-angle imaging device. FIG. 3 is a diagram showing the positions of a virtual camera and a predetermined area when the omnidirectional image is a three-dimensional solid sphere. FIG. 12 is a diagram showing an example of a three-dimensional perspective view of FIG. 11; FIG. 4 is an example of a diagram showing a relationship between predetermined area information and an image of the predetermined area. It is a functional block diagram showing in block form the communication management system, the communication terminal 10A, and the functions of the communication terminal 10A that the communication system has. FIG. 2 is a sequence diagram illustrating an example of processing in a communication preparation stage of the communication system. FIG. 3 is a diagram showing an example of a destination selection screen displayed on a communication terminal. FIG. 2 is a sequence diagram illustrating an example of communication processing of the communication system. 3 is a diagram illustrating an example of a video display screen that a communication terminal displays on a display unit in pattern 1. FIG. 7 is an example of a flowchart diagram illustrating a procedure for switching images in an example of arrangement of images of pattern 1. FIG. 7 is a diagram illustrating an example of a video display screen that a communication terminal displays on a display unit in pattern 2. FIG. 7 is an example of a flowchart showing a procedure for displaying images in an example of arrangement of images of pattern 2. FIG. FIG. 7 is a diagram illustrating an example of a video display screen displayed by the communication terminal on the display unit in another display example of pattern 2; 23 is an example of a flowchart showing a procedure for displaying a video in another display example of pattern 2 in FIG. 22. FIG. 12 is a diagram illustrating an example of a video display screen that a communication terminal displays on a display unit in pattern 3. FIG. 11 is an example of a flowchart diagram showing a procedure for displaying images in an example of arrangement of images of pattern 3. FIG. 7 is a diagram illustrating an example of a video display screen displayed by the communication terminal on the display unit in another display example of pattern 3. FIG. 27 is an example of a flowchart showing a procedure for displaying a video in another display example of pattern 3 in FIG. 26. FIG. FIG. 2 is an example of a diagram illustrating an opportunity for automatic pattern switching; FIG.

DESCRIPTION OF THE PREFERRED EMBODIMENTS As an example of a mode for carrying out the present invention, a communication terminal and a video display method performed by the communication terminal will be described below with reference to the drawings.

<Outline of operation of communication system and communication terminal>
First, the outline of the communication system will be explained using FIG. FIG. 1(a) shows an example of a schematic configuration diagram of a communication system using a telepresence robot. A telepresence robot is placed in a company 801, and an operator 810 is present in a home 802. The telepresence robot has a communication terminal 10B mounted on a mobile device 20 and a wide-angle imaging device 9. Further, the operator 810 has a communication terminal 10A.

The wide-angle imaging device 9 transmits a spherical image (an example of a second image), which will be described later, to the communication terminal 10B. The communication terminals 10A and 10B each have a planar imaging device and a microphone, and transmit and receive video and audio to and from each other. That is, the communication terminal 10B transmits the spherical image and the front image 831 (or/and the rear image 832, an example of the first image) to the communication terminal 10A, and the communication terminal 10A transmits the front image 831 to the communication terminal 10B. do.

Further, application software (hereinafter referred to as an application) is running on the communication terminal 10A, and the operator 810 inputs operation instructions while checking the video transmitted by the communication terminal 10B. . Operation instructions input by operator 810 are transmitted from communication terminal 10A to communication terminal 10B.

The communication terminal 10B and the mobile device 20 can communicate via short-range wireless communication such as Bluetooth (registered trademark), and the communication terminal 10B controls the mobile device 20 based on operation instructions, so the operator 810 can move from a remote location (home). Device 20 can be moved.

By moving the moving device 20 in this manner, the operator 810 can have a conversation with a desired party 820, as shown in FIG. 1(b). In FIG. 1B, the operator 810 moves the mobile device 20 to the location of Mr. A 820a and has a conversation with Mr. A, and then moves the mobile device 20 to the location of Mr. B 820b and has a conversation with Mr. B. are doing.

In addition to meeting people, the operator 810 can also do many things remotely without having to travel to the location, such as checking products at stores in remote locations, inspecting factories, and participating in exhibitions. It can be done from the ground.

Furthermore, if the planar imaging device built into the communication terminal 10B can only capture images from the front and rear, it is difficult for the operator 810 to check the surroundings of the mobile device 20 (for example, underfoot, left and right). Therefore, in this embodiment, the communication terminal 10B is equipped with a wide-angle imaging device 9 that can capture images of the surrounding area 360 degrees. The wide-angle imaging device 9 may be installed in the communication terminal 10B or may be installed in the mobile device 20. Since the wide-angle imaging device 9 can image a wide-angle range, it is placed, for example, above where the surroundings are less likely to be obstructed. Although it is placed above the communication terminal 10B in FIG. 1, it may be placed on the left or right side, or on the pole of the mobile device 20.

Using FIG. 2, an outline of how the communication terminal 10A of this embodiment displays images of the flat imaging device and the wide-angle imaging device 9 that can capture images of the surrounding 360 degrees will be explained. FIG. 2 is a diagram illustrating a display example of a front image or a rear image captured by the flat imaging device and a spherical image captured by the wide-angle imaging device 9. In this embodiment, three patterns will be described as examples of the arrangement of front images, rear images, and omnidirectional images.

FIG. 2(a) shows an example of the arrangement of pattern 1. Pattern 1 is an arrangement example in which a spherical image 840 or a front image (rear image) 841 is switched and displayed. In the left diagram of FIG. 2(a), the communication terminal 10A displays a spherical image 841 on the entire surface, and in the right diagram of FIG. 2(a), the front image (or rear image) 841 is displayed on the entire surface. . The term "full screen" does not necessarily mean that the display does not include any other elements; it may also include the soft keys of an application or the screen of another application. The operator can switch between the front image (or rear image) and the spherical image, both of which can be viewed on a large screen.

FIG. 2(b) shows an example of the arrangement of pattern 2. Pattern 2 is an arrangement example in which a spherical image and a front image (rear image) are displayed side by side on the same screen. In FIG. 2(b), an equirectangular projection image 842, in which a spherical image is converted to a plane using an equirectangular projection, is displayed in a small size at the top, and a front image (or rear image) 843 is displayed in a large size at the bottom. There is. The operator can check the front or back side using the front image (or rear image) 843 while checking the entire image using the equirectangular projection image 842.

FIG. 2(c) shows an example of the arrangement of pattern 3. Pattern 3 is an arrangement example in which either a spherical image or a front image (rear image) is used as a basic image, and a main screen and a plurality of sub screens are combined. In the left diagram of FIG. 2C, a spherical image 844 is displayed on the main screen, and two sub-screens (a front image 845 and a rear image 846) are arranged below the main screen. In the right diagram of FIG. 2(c), a spherical image is displayed on the main screen 847, and a plurality of sub-screens 848 are arranged to overlap the main screen. On the sub-screen 848, a front image, a rear image, and a reproduction of a part of the viewing angle of the omnidirectional image can be arbitrarily displayed.

In FIG. 2, a mobile device operation button 614 (an example of a controller) is displayed on at least one of the spherical image or the front image (or rear image). Therefore, the operator can move the moving device 20 while viewing the omnidirectional image or the front image (or rear image).

In this way, in the communication system of this embodiment, the communication terminal 10A of the operator 810 displays the omnidirectional image and the front image (rear image) in an appropriately arranged manner. You can also check the current situation, as well as high-resolution images of what's in front of you or behind you.

<About terms>
Operator 810 is a person who operates mobile device 20. The operator 810 is also a user who uses the communication terminal 10A, but is referred to as an operator in this embodiment.

The conversation partner 820 is a person within the company, but may also be an animal such as a dog or a cat. Additionally, mobile device 20 may not be used by operator 810 to talk to a party. For example, when the mobile device 20 patrols a company, the communication terminal 10B only needs to transmit video and audio to the communication terminal 10A, and the operator may not speak.

The location to which the mobile device 20 moves may be any location to which the mobile device 20 can move. Furthermore, the device is not limited to a device that moves using power such as a motor, and a human may assist the movement.

The first image refers to an image with a normal angle of view, and the second image refers to an image with a wider angle than the first image. A normal angle of view is an angle of view that allows distortion even when an image is captured using a perspective projection lens, and can simply be said to be an image captured using a perspective projection lens. A wide-angle image refers to an image captured using a wide-angle lens such as a fisheye lens. Projection methods include stereoscopic projection, equidistant projection, equisolid angle projection, and orthogonal projection. The wide-angle video is not limited to a spherical video that captures 360 degrees of the surrounding area, but may also be a video that captures a hemisphere or a video that captures 180 to 360 degrees in the horizontal direction.

"Large", such as the first video or the second video being displayed in a large size, refers to the fact that the display area of the first video or the second video is large, that is, the size of the area that occupies the screen. The number of pixels per unit length may be larger.

In this embodiment, when a moving image is assumed, it is referred to as a video, and when a single still image is assumed, it is referred to as an image, but a video has multiple images, and a video and an image are strictly distinguished. shall not be done.

Controlling to display both the first video and the second video refers to displaying the first video and the second video at the same time, or displaying only one of them, preferably at the same time. This means that the user can select whether to display the first video or the second video.

A controller that controls a mobile device refers to a user interface, soft keys, display components, etc. that can send some control signals to the mobile device. The control signals are not limited to control signals for movement, but may include control signals for turning on/off the power, stopping/resuming imaging, lighting a lamp, playing a moving image, and the like.

<System configuration example>
FIG. 3 is a diagram illustrating a configuration example of a communication system according to an embodiment. The communication system 1 includes a plurality of communication terminals (10A, 10B1, 10B2, 10B3, 10E), a mobile device 20, a relay device 30, a communication management system 50, and the like. In the following description, "communication terminal 10" is used to refer to any communication terminal among the plurality of communication terminals (10A, 10B1, 10B2, 10B3, 10E). Further, "communication terminal 10B" is used to indicate any communication terminal among the communication terminals (10B1, 10B2, 10B3). Further, the number of communication terminals 10 in FIG. 3 is an example.

The communication terminal 10, the relay device 30, and the communication management system 50 are each connected to other communication terminals 10, devices, and systems via the communication network 2 so that they can communicate with each other. The communication network 2 may include, for example, a LAN (Local Area Network), the Internet, a mobile phone network, a dedicated line, or the like.

The communication terminal 10 is, for example, a tablet terminal, a smartphone, a general-purpose information processing device such as a PC (Personal Computer), or a communication terminal such as a dedicated video conference device. The communication terminal 10 can conduct, for example, a video conference or the like by transmitting and receiving, for example, image data, audio data, etc., with one or more other communication terminals 10.

By executing an application compatible with the communication system 1, the communication terminal 10A can hold a video conference with the communication terminal 10B and remotely control the mobile device 20 via the communication terminal 10B. For example, the communication terminal 10A can move the mobile device 20 equipped with the communication terminal 10B back and forth, left and right, etc., by operating an operation button displayed on the display screen of the video conference. Note that the communication terminal 10A can also operate browser software, and the browser software can display images and accept operations from the mobile device 20 on the browser software.

The mobile device 20 and the communication terminal 10B (including the wide-angle imaging device 9) are referred to as a telepresence robot. The mobile device 20 can perform "forward", "reverse", "turn right", "turn left" by driving a plurality of wheels, for example, in accordance with control from the communication terminal 10B attached to the mobile device 20. It is a device with a traveling function that allows movement such as ``. When the user instructs to move left or right, the robot moves forward after "turning right" or "turning left." Note that the appearance of the moving device 20 shown in FIG. 3 is just an example. The mobile device 20 only needs to be movable together with the communication terminal 10B in response to an operation instruction from the communication terminal 10B mounted on the mobile device 20.

Further, the mobile device 20 and the communication terminal 10B may be integrated. Integral means that the communication terminal 10B is attached to the mobile device 20 from the time of shipment to the distribution stage, and that it is obvious from the outside that the communication terminal 10B exists as a part of the mobile device 20. Integrity in design, manufacturer, and vendor is recognized, or the mobile device 20 and the communication terminal 10B are inseparable and difficult to separate, or even if they can be separated, part of the function will be impaired; etc. Note that when the mobile device 20 and the communication terminal 10B are integrated, the communication terminal 10B is treated as a communication unit included in the mobile device 20.

When the mobile device 20 and the communication terminal 10B are integrated, the wide-angle imaging device 9 may also be built into the mobile device 20, or the wide-angle imaging device 9 may be externally attached to the mobile device 20. When the wide-angle imaging device 9 is built into the mobile device 20, the wide-angle imaging device 9 is treated as a wide-angle imaging unit included in the mobile device 20.

Furthermore, even if the communication terminal 10B is externally attached to the mobile device 20, the wide-angle imaging device 9 may be built into the mobile device 20, or even if the wide-angle imaging device 9 is externally attached to the mobile device 20. good.

Moreover, the wide-angle imaging device 9 may be built into the communication terminal 10B, or may be attached externally. When the wide-angle imaging device 9 is built into the communication terminal 10B, the wide-angle imaging device 9 is treated as a wide-angle imaging unit included in the communication terminal 10B.

The relay device 30 is, for example, an information processing device or a system including one or more information processing devices, and relays content data (video, audio, operation instruction information) transmitted and received between the plurality of communication terminals 10. However, the plurality of communication terminals 10 may directly transmit and receive content data without going through the relay device 30.

The communication management system 50 is, for example, an information processing device or a system including one or more information processing devices. The communication management system 50 manages, for example, login authentication from the communication terminal 10 , management of communication status of the communication terminal 10 , management of destination list, etc., and session communication between a plurality of communication terminals 10 via the relay device 30 . control, etc.

In one embodiment, a session is realized by the relay device 30 relaying content data including image data and sound data (audio and other sounds) between a plurality of communication terminals 10.

In the above configuration, for example, the operator of the communication terminal 10A can communicate with the communication terminal 10B and move the communication terminal 10B and the mobile device 20 by remote control. Thereby, the operator of the communication terminal 10A can move the communication terminal 10B and the mobile device 20 close to an arbitrary party and hold a video conference or the like.

The communication system 1 also includes a data providing system that transmits content data in one direction from one communication terminal 10 to the other communication terminal 10 via the communication management system 50, and a data provision system that transmits content data in one direction from one communication terminal 10 to the other communication terminal 10 via the communication management system 50. A communication system for mutually transmitting information, emotions, etc. between the terminals 10 is included. This communication system is a system for mutually transmitting information, emotions, etc. between a plurality of communication communication terminals via the relay device 30, and examples include a television (or video) conference system and a video telephone system. .

<Hardware configuration>
<<Hardware configuration of communication terminal>>
FIG. 4 is a diagram showing an example of the hardware configuration of a communication terminal according to an embodiment of the present invention. The communication terminal 10 includes a general computer configuration, and includes, for example, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a flash memory 104, and an SSD (Solid State Drive). ) 105 etc. The communication terminal 10 also includes a media I/F (Interface) 107, an input unit 108, a display unit 109, a network I/F 111, a camera 112, an image sensor I/F 113, a microphone 114, a speaker 115, and an audio input/output I/F. It has F116 etc. Furthermore, the communication terminal 10 includes an external device connection I/F 117, a short-range wireless communication section 118, a bus 119, and the like.

The CPU 101 is an arithmetic device that realizes each function of the communication terminal 10 by reading programs and data from, for example, the ROM 102 and the flash memory 104 and executing processes. The ROM 102 is a nonvolatile memory that stores in advance a program used to start the CPU 101, such as an IPL (Initial Program Loader). The RAM 103 is a volatile memory used as a work area for the CPU 101, etc.

The flash memory 104 is, for example, a storage device that stores an OS (Operating System), application programs, various data, and the like. The SSD 105 controls reading and writing of various data to and from the flash memory 104 under the control of the CPU 101. The media I/F 107 controls reading or writing (storage) of data to a recording medium 106, which is a recording medium such as a memory card, for example. A program for controlling the communication terminal 10 may be stored in the recording medium.

The input unit 108 is, for example, an input device such as a touch panel, a keyboard, or a pointing device for receiving input operations from an operator. Voice input may also be possible. The display unit 109 is a display device that performs various displays for the operator. Note that the input unit 108 and the display unit 109 may be, for example, a display input unit 110 such as a touch panel display in which a touch panel and a display are integrated.

The network I/F 111 is a communication interface through which the communication terminal 10 uses the communication network 2 to transmit and receive data. Camera 112 includes an image sensor for capturing an image of a subject under the control of CPU 101. The image sensor I/F 113 controls imaging by the camera 112 and converts the captured data into predetermined image data. The camera 112 is a planar imaging device that captures images at a narrower angle of view than the wide-angle imaging device 9, but has a higher resolution than the wide-angle imaging device 9. Resolution refers to the degree of fineness of a digital image, and resolution is expressed by quantifying the fineness of each unit point (dot, pixel) that makes up the digital image. A "dot" is generally used as a unit to express resolution. In the case of a display, it is often represented by numbers arranged horizontally and vertically, such as "1024 x 768 dots." Comparing the same imaging range, the number of dots in the wide-angle imaging device 9 is smaller than the number of dots in the camera 112.

The microphone 114 converts the recorded audio into an electrical signal. The speaker 115 converts the audio signal into audio and outputs it. The audio input/output I/F 116 controls input/output of audio from the microphone 114 and the speaker 115.

The external device connection I/F 117 is, for example, an interface for connecting an external device such as a USB (Universal Serial Bus). The external device includes, for example, the mobile device 20 in FIG. 3 and the like.

The short-range wireless communication unit 118 is a communication interface for communicating with an external device (for example, the mobile device 20) by short-range wireless communication such as Bluetooth (registered trademark) or Bluetooth (registered trademark) Low Energy. The bus 119 is commonly connected to each of the above components and transmits address signals, data signals, various control signals, and the like.

<<Hardware configuration of communication management system and relay device>>
FIG. 5 is a diagram illustrating an example hardware configuration of a communication management system according to an embodiment. The communication management system 50 has a general computer configuration, and includes, for example, a CPU 501, a ROM 502, a RAM 503, an HD 504, an HDD (Hard Disk Drive) 505, a media drive 507, a display 508, and the like. The communication management system 50 also includes a network I/F 509, a keyboard 511, a mouse 512, a CD-ROM drive 514, a bus 510, and the like.

The CPU 501 is an arithmetic device that realizes each function of the communication management system 50 by reading programs and data stored in the ROM 502, HD 504, etc., and executing processing. The ROM 502 is a non-volatile memory that stores in advance programs used to start the CPU 501 such as IPL. The RAM 503 is a volatile memory used as a work area for the CPU 501, etc.

The HD 504 is, for example, a storage device that stores an OS, programs such as application programs, and various data. The HDD 505 controls reading and writing of various data to the HD 504 under the control of the CPU 501. The display 508 is a display device that displays various information such as, for example, a cursor, menu, window, characters, or images.

The network I/F 509 is a communication interface for performing data communication using the communication network 2. The keyboard 511 is an example of an input device for receiving input operations such as characters, numerical values, and various instructions by a system administrator. The mouse 512 is an example of a pointing device that accepts operations such as selecting and executing various instructions, selecting a processing target, and moving a cursor by a system administrator.

Note that the communication management system 50 or the relay device 30 does not usually need to have the display 508, the keyboard 511, and the mouse 512. In this case, they may be connected as necessary.

The media drive 507 controls reading and writing (storage) of data from, for example, a recording medium 506 such as a memory card. A CD-ROM drive 514 controls data reading from or writing to a disk 513, which is an example of a removable recording medium. The bus 510 electrically connects each of the above components and transmits address signals, data signals, various control signals, and the like.

Note that the hardware configuration of the computer described above is just an example.

It is assumed that the relay device 30 has the same hardware configuration as the communication management system 50. Note that each program for the communication terminal 10, the relay device 30, and the communication management system 50 is recorded on a computer-readable recording medium and distributed as an installable or executable file. It's okay. Furthermore, examples of the recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray disc, USB memory, and the like. Furthermore, recording media such as CD-ROMs in which each program is stored, and the HD 504 in which these programs are stored can be provided domestically or internationally as a program product.

<<Hardware configuration of mobile device>>
FIG. 6 is a diagram illustrating an example hardware configuration of a mobile device according to an embodiment. The mobile device 20 includes, for example, a CPU 401, a RAM 402, a ROM 403, an external device I/F 404, a short-range wireless communication section 405, a wheel drive section 406, a steering section 407, and the like.

The CPU 401 is a calculation device that implements each function of the mobile device 20 by executing programs stored in the ROM 403 and the like. The RAM 402 is a volatile memory used as a work area for the CPU 401 and the like. The ROM 403 is a nonvolatile memory that stores programs for the mobile device 20 and the like. The ROM 403 may be, for example, a rewritable nonvolatile memory such as a flash ROM.

The external device I/F 404 is a wired communication interface for making a wired connection with the external device connection I/F 117 of the communication terminal 10 and performing communication.

The short-range wireless communication unit 405 is, for example, a wireless communication interface for performing wireless communication using the same wireless communication method as the short-range wireless communication unit 118 of the communication terminal 10. The mobile device 20 only needs to be able to communicate with the communication terminal 10 using the external device I/F 404 or the short-range wireless communication unit 405, for example.

The wheel drive unit 406 is an example of a drive device that drives wheels for moving the moving device 20. The wheel drive unit 406 includes, for example, a motor.

The steering unit 407 is an example of a steering device that steers the moving device 20 that is moved by the wheel drive unit 406. The steering unit 407 may be, for example, one that changes the orientation and inclination of the wheels, or one that changes the orientation of the moving device 20 by controlling the number of rotations, speed, etc. of the left and right wheels. good.

<<Hardware configuration example of wide-angle imaging device>>
The hardware configuration of the wide-angle imaging device 9 will be explained using FIG. 7. FIG. 7 is a hardware configuration diagram of an example of the wide-angle imaging device 9. As shown in FIG. In the following, the wide-angle imaging device 9 is a spherical (omnidirectional) wide-angle imaging device 9 that uses two imaging devices, but the number of imaging devices may be any number greater than or equal to two. In addition, it does not necessarily have to be a device exclusively for omnidirectional imaging, but by attaching an aftermarket omnidirectional imaging unit to a normal digital camera, smartphone, etc., it can essentially have the same functions as the wide-angle imaging device 9. Good too.

As shown in FIG. 7, the wide-angle imaging device 9 includes an imaging unit 301, an image processing unit 304, an imaging control unit 305, a microphone 308, a sound processing unit 309, a CPU (Central Processing Unit) 311, and a ROM (Read Only). Memory) 312, SRAM (Static Random Access Memory) 313, DRAM (Dynamic Random Access Memory) 314, operation unit 315, network I/F 316, communication unit 317, antenna 317a, electronic compass 318, gyro sensor 319, and acceleration sensor 320 etc. are also connected.

Among these, the imaging unit 301 includes wide-angle lenses (so-called fisheye lenses) 302a and 302b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging units provided corresponding to each wide-angle lens. It includes elements 303a and 303b. The image sensors 303a and 303b are image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert optical images formed by fisheye lenses 302a and 302b into electrical signal image data and output the image data. It has a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set.

The imaging elements 303a and 303b of the imaging unit 301 are each connected to the image processing unit 304 via a parallel I/F bus. On the other hand, the imaging elements 303a and 303b of the imaging unit 301 are connected to the imaging control unit 305 via a serial I/F bus (such as an I2C bus). The image processing unit 304, the imaging control unit 305, and the sound processing unit 309 are connected to the CPU 311 via a bus 310. Furthermore, a ROM 312, an SRAM 313, a DRAM 314, an operation unit 315, a network I/F 316, a communication unit 317, an electronic compass 318, and the like are also connected to the bus 310.

The image processing unit 304 takes in image data output from the image sensors 303a and 303b through the parallel I/F bus, performs predetermined processing on each image data, and then synthesizes these image data. , create equirectangular cylindrical projection image data as shown in FIG. 9(c).

Generally, the imaging control unit 305 uses the I2C bus to set commands and the like in register groups of the imaging devices 303a and 303b, using the imaging control unit 305 as a master device and the imaging devices 303a and 303b as slave devices. Necessary commands and the like are received from the CPU 311. The imaging control unit 305 also uses the I2C bus to take in status data and the like of the register groups of the imaging elements 303a and 303b, and sends it to the CPU 311.

Further, the imaging control unit 305 instructs the imaging elements 303a and 303b to output image data at the timing when the shutter button of the operation unit 315 is pressed. Depending on the wide-angle imaging device 9, it may have a preview display function or a function corresponding to video display on a display (for example, the display of the communication terminal 10B). In this case, image data is continuously output from the image sensors 303a and 303b at a predetermined frame rate (frames/minute).

The imaging control unit 305 also functions as a synchronization control unit that synchronizes the output timing of image data of the imaging elements 303a and 303b in cooperation with the CPU 311, as will be described later. Note that in this embodiment, the wide-angle imaging device 9 is not provided with a display, but may be provided with a display section.

Microphone 308 converts sound into sound (signal) data. The sound processing unit 309 takes in sound data output from the microphone 308 through the I/F bus, and performs predetermined processing on the sound data.

The CPU 311 controls the overall operation of the wide-angle imaging device 9 and executes necessary processing. ROM312 stores various programs for CPU311. The SRAM 313 and DRAM 314 are work memories that store programs executed by the CPU 311, data being processed, and the like. In particular, the DRAM 314 stores image data that is currently being processed by the image processing unit 304 and data of processed equirectangular projection images.

The operation unit 315 is a general term for operation buttons such as a shutter button. By operating the operation unit 315, the operator inputs various imaging modes, imaging conditions, and the like.

The network I/F 316 is a general term for an interface circuit (USB I/F, etc.) with external media such as an SD card, a personal computer, etc. Further, the network I/F 316 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 314 may be recorded on an external media via this network I/F 316, or may be transferred to an external terminal such as the communication terminal 10B via the network I/F 316 as necessary. device).

The communication unit 317 communicates with an external terminal (device) of the communication terminal 10B via an antenna 317a provided on the wide-angle imaging device 9 using short-range wireless communication technology such as Wi-Fi, NFC, or Bluetooth (registered trademark). I do. This communication unit 317 can also transmit the data of the equirectangular projection image to an external terminal (device) such as the communication terminal 10B.

The electronic compass 318 calculates the orientation of the wide-angle imaging device 9 from the earth's magnetism and outputs orientation information. This orientation information is an example of related information (metadata) in accordance with Exif (Exchangeable image file format), and is used for image processing such as image correction of captured images. Note that the related information also includes data such as the date and time when the image was captured and the data capacity of the image data.

The gyro sensor 319 detects changes in angle (roll angle, pitching angle, yaw angle) accompanying movement of the wide-angle imaging device 9. A change in angle is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of a captured image.

Acceleration sensor 320 detects acceleration in three axial directions. The attitude (angle with respect to the direction of gravity) of the wide-angle imaging device 9 is detected based on the detected acceleration. Having both the gyro sensor 319 and the acceleration sensor 320 improves the accuracy of image correction.

<About spherical images>
Next, using FIG. 8, the usage situation of the wide-angle imaging device 9 will be explained. Note that FIG. 8 is an image diagram of how the wide-angle imaging device 9 is used. As shown in FIG. 8, the wide-angle imaging device 9 is used, for example, in a user's hand to capture an image of a subject around the user. In this case, two hemispherical images can be obtained by capturing images of objects around the user using the imaging device 303a and the imaging device 303b shown in FIG. 7, respectively.

Next, an outline of the process for creating an equirectangular projection image EC and a celestial sphere image CE from an image captured by the wide-angle imaging device 9 will be explained using FIGS. 9 and 10. Note that FIG. 9(a) is a hemispherical image (front side) captured by the wide-angle imaging device 9, FIG. 9(b) is a hemispherical image (backside) captured by the wide-angle imaging device 9, and FIG. FIG. 2 is a diagram showing an image expressed by the ectorectal projection method (hereinafter referred to as an "equirectangular projection image"). FIG. 10(a) is a conceptual diagram showing a state where a sphere is covered by an equirectangular projection image, and FIG. 10(b) is a diagram showing a celestial sphere image.

As shown in FIG. 9A, the image obtained by the image sensor 303a becomes a hemispherical image (front side) curved by a fisheye lens 302a, which will be described later. Further, as shown in FIG. 9B, the image obtained by the image sensor 303b becomes a hemispherical image (rear side) curved by the fisheye lens 302b. The hemisphere image (front side) and the hemisphere image (rear side) inverted by 180 degrees are combined by the wide-angle imaging device 9, and as shown in FIG. 9(c), an equirectangular projection image EC is created. is created.

Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular projection image is pasted so as to cover the spherical surface, as shown in Figure 10(a). A spherical image CE as shown in b) is created. In this way, the spherical image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the omnidirectional image CE may be a still image or a moving image.

As described above, since the spherical image CE is an image pasted to cover a spherical surface, it feels strange when viewed by humans. Therefore, by displaying a predetermined area (hereinafter referred to as a "predetermined area image") of a part of the spherical image CE as a flat image with less curvature, it is possible to perform a display that does not give a sense of discomfort to humans. This will be explained using FIGS. 11 and 12.

Note that FIG. 11 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional solid sphere. The virtual camera IC corresponds to the position of the viewpoint of the operator who views the spherical image CE displayed as a three-dimensional solid sphere. Further, FIG. 12(a) is a three-dimensional perspective view of FIG. 11, and FIG. 12(b) is a diagram showing a predetermined area image when displayed on a display. Moreover, in FIG. 12(a), the omnidirectional image shown in FIG. 11 is represented by a three-dimensional solid sphere CS. Assuming that the spherical image CE generated in this way is a three-dimensional sphere CS, the virtual camera IC is located inside the spherical image CE, as shown in FIG. 12. The predetermined region T in the spherical image CE is an imaging region of the virtual camera IC, and is specified by predetermined region information indicating the imaging direction and angle of view of the virtual camera IC in the three-dimensional virtual space including the spherical image CE. Ru. Furthermore, zooming of the predetermined area T can also be expressed by moving the virtual camera IC closer to or farther away from the omnidirectional image CE. The predetermined area image Q is an image of a predetermined area T in the omnidirectional image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the spherical image CE (see FIG. 13).

The predetermined area image Q shown in FIG. 12(a) is displayed on a predetermined display as an image of the imaging area of the virtual camera IC, as shown in FIG. 12(b). The image shown in FIG. 12(b) is a predetermined area image represented by initialized (default) predetermined area information. In the following, description will be given using the imaging direction (ea, aa) and angle of view (α) of the virtual camera IC. Note that the predetermined area T may be indicated by the imaging area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the angle of view α and the distance f.

Next, the relationship between the predetermined area information and the image of the predetermined area T will be explained using FIG. 13. Note that FIG. 13 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 13, "ea" indicates elevation angle, "aa" indicates azimuth angle, and "α" indicates angle of view. That is, the attitude of the virtual camera IC is changed so that the point of view of the virtual camera IC indicated by the imaging direction (ea, aa) becomes the center point CP of the predetermined area T that is the imaging area of the virtual camera IC. Become. As shown in FIG. 13, when α is the diagonal angle of view of the predetermined area T represented by the angle of view α of the virtual camera IC, the center point CP is the (x, y) parameter of the predetermined area information. becomes. f is the distance from the virtual camera IC to the center point CP. L is the distance between an arbitrary vertex of the predetermined region T and the center point CP (2L is a diagonal line). In FIG. 13, the trigonometric function expressed by the following (Equation 1) generally holds true.

L/f=tan(α/2)...(Formula 1)
<About functions>
Next, the functions of the communication system will be explained using FIG. 14. FIG. 14 is a functional block diagram showing in block form the functions of the communication management system 50, the communication terminal 10B, and the communication terminal 10A included in the communication system 1.

<<Functional configuration of communication terminal 10A>>
The communication terminal 10A receives, for example, operation instructions for controlling equipment such as the mobile device 20. The communication terminal 10A includes a transmitting/receiving section 11, an operation input receiving section 12, a communication control section 13, an imaging section 14, an audio input section 15a, an audio output section 15b, a display control section 16, and a storage/reading section 17. . Each of these parts is realized by one of the components shown in FIG. 4 operating according to an instruction from the CPU 101 according to a program for the communication terminal 10 loaded from the flash memory 104 onto the RAM 103. It is a function or means to The communication terminal 10 also includes a storage unit 18 implemented by a RAM 103 shown in FIG. 4 and a flash memory 104 shown in FIG. 4.

<<Functional configuration of communication terminal 10B>>
Communication terminal 10B is communication terminal 10 mounted on mobile device 20. The communication terminal 10B is an example of the communication terminal 10 that has a control function for devices such as the mobile device 20, for example. In addition to the functional configuration of the communication terminal 10A described above, the communication terminal 10B includes an operation instruction receiving section 19a, a spherical video receiving section 19b, a device control section 19c, an inter-device communication section 19d, and the like. Note that the communication terminal 10A also has these additional configurations.

(Functional configuration of communication terminal 10)
Next, each functional configuration of the communication terminal 10 (communication terminal 10A and communication terminal 10B) will be described in detail.

The transmitting/receiving unit 11 transmits and receives various data (or information) to and from other communication terminals, devices, or systems via the communication network 2 . The transmitting/receiving unit 11 starts receiving status information indicating the status of each communication terminal as a destination candidate from the communication management system 50 before starting a call with a desired destination terminal. Note that this status information includes not only the operating status of each communication terminal 10 (ON line or OFF line status), but also detailed status such as whether a call is possible or a call is in progress in the case of an ON line. shows.

The transmitting/receiving unit 11 of the communication terminal 10A receives the first video transmitted by the communication terminal 10B (or the communication unit included in the communication terminal 10B) and the image captured by the wide-angle imaging device 9 connected to or integrated with the communication terminal 10B or the mobile device. Receive spherical images.

The operation input reception unit 12 receives various inputs from an operator to a communication terminal. For example, when an operator performs an operation to turn on the power of the communication terminal 10, the operation input reception unit 12 receives the operation and controls the power to be turned on.

For example, in response to the reception of the power ON, the communication control unit 13 sends login request information indicating that a login is requested from the transmitting/receiving unit 11 to the communication management system 50 via the communication network 2, and the current information of the request source terminal. automatically sends the IP address of Further, when the operator performs an operation to turn off the power of the communication terminal 10, the transmission/reception unit 11 transmits status information indicating that the power is to be turned off to the communication management system 50, and then the operation input reception unit 12 turns off the power. Make it. Thereby, the communication management system 50 side can grasp that the power of the communication terminal 10 has changed from being powered on to being powered off.

Further, the communication control unit 13 performs various communication controls such as establishing and disconnecting sessions for transmitting and receiving content data with other communication terminals 10 via the relay device 30 . Note that the communication control unit 13 according to the present embodiment includes the communication ID (Identification) of the communication terminal 10 in the session control information (for example, start request information, start response information, etc. described later) to be sent to the communication management system 50. Include and send.

The communication ID is an example of identification information of an account that can participate in a session for transmitting and receiving content data using the communication terminal 10. The communication ID may be, for example, a user ID that is identification information of an operator, an application ID that is identification information of an application, a contract ID that is identification information of a subscriber of the communication terminal 10, or the like. In addition, the communication ID may include information in which at least two of letters, numbers, symbols, and various marks are combined. It can also be an email address.

The imaging unit 14 converts imaging data obtained by imaging a subject into predetermined image (video) data and outputs the data. There is one imaging unit 14 on the front side of the communication terminal (capturing the front image) and one on the back side (capturing the rear image). There may be more imaging units than this.

After the operator's voice is converted into an audio signal by the microphone 114, the audio input unit 15a converts the audio signal into predetermined audio data and outputs the same. The audio output unit 15b converts the audio data into an audio signal, outputs it to the speaker, and causes the speaker 115 to output audio.

For example, the display control unit 16 causes the display unit 109, the display input unit 110, or the like to display image data included in the content data received by the communication terminal 10. Further, the display control unit 16 can transmit the destination list information received from the communication management system 50 to the display unit 109 and cause the display unit 109 to display the destination list.

The display control unit 16 of the communication terminal 10A controls the display of either the front image or the rear image received by the transmitting/receiving unit 11, and superimposes the mobile device operation button 614 that controls the mobile device 20 on the front image or the rear image. displayed on the display device.

The storage/readout section 17 performs processing of storing various data in the storage section 18 and reading out various data stored in the storage section 18 .

The storage unit 18 stores, for example, the above-mentioned communication ID and authentication information such as a password corresponding to the communication ID. The storage unit 18 overwrites and stores image data and audio data that are received when making a call with a destination terminal each time the data is received. Of these, an image is displayed on the display unit 109 or the like using the image data before being overwritten, and audio is output from the speaker 115 using the audio data before being overwritten.

Next, each functional configuration included in the communication terminal 10B will be explained.

The omnidirectional image receiving unit 19b receives an equirectangular cylindrical projection image from the wide-angle imaging device 9, for example, by wireless communication such as Bluetooth (registered trademark) or wired communication such as a USB cable. Equirectangular projection video is a moving image that is repeatedly transmitted with enough frequency that it can be considered a moving image. However, it may be a still image, or it may be possible to switch between a moving image and a still image.

The operation instruction receiving section 19a receives, via the transmitting/receiving section 11, operation instruction information requesting the communication terminal 10B to control a device (mobile device 20) from the communication terminal 10A. This operation instruction information includes, for example, the communication ID of the communication terminal 10 that transmitted the operation instruction information, and an operation instruction indicating the requested control content.

The device control unit 19c controls the mobile device 20 based on the operation instructions included in the operation instruction information received by the operation instruction receiving unit 19a. The device-to-device communication unit 19d communicates with the mobile device 20 using the short-range wireless communication unit 118.

<<Functional configuration of mobile device>>
The mobile device 20 includes, for example, an inter-device communication section 21, a travel control section 22, and the like. The inter-device communication unit 21 is realized, for example, by a command from the CPU 401 shown in FIG. 6, an external device I/F 404, or a short-range wireless communication unit 405. Here, it is assumed that the inter-device communication unit 21 communicates with the communication terminal 10B using the short-range wireless communication unit 405.

The travel control unit 22 controls the wheel driving unit 406 and the steering unit 407 in FIG. 6 according to the control content acquired from the communication terminal 10A, for example, to move the moving device 20 forward, backward, turn left, or turn right. Control movement (travel) such as turning.

<<Functional configuration of wide-angle imaging device>>
The wide-angle imaging device 9 includes, for example, a wide-angle image transmitting section 31, a wide-angle image capturing section 32, and the like. The wide-angle image capturing unit 32 captures a wide-angle equirectangular cylindrical projection image of 360 degrees at a predetermined frame rate. A still image may be captured. The wide-angle image transmitting unit 31 is realized by the communication unit 317 shown in FIG. 7, and transmits an equirectangular projection video of a moving image or a still image to the communication terminal 10B.

<<Functional configuration of communication management system>>
The communication management system 50 includes a transmitting/receiving section 51, a terminal authentication section 52, a terminal managing section 53, a terminal extracting section 54, a session managing section 55, a storage/reading section 57, and the like. The functions of each of these parts are realized when any of the components shown in FIG. or a means of functioning. The communication management system 50 also includes a storage unit 5000 implemented by the HD 504 shown in FIG. 5 or the like.

(Functional configuration of communication management system)
Next, each functional configuration of the communication management system 50 will be explained in detail. The transmitting/receiving unit 51 transmits and receives various data (or information) to and from other communication terminals, devices, or systems via the communication network 2 .

The terminal authentication unit 52 determines whether or not the combination of communication ID and password included in the login request information received via the transmission/reception unit 51 is included in the authentication management DB (Database) 5002. The terminal 10 is authenticated.

The terminal management unit 53 stores and manages the destination name, operating status, reception date and time of request information, etc., and IP address of the requesting terminal in association with each communication ID in the terminal management DB 5003. For example, when the operator turns the power of the communication terminal 10 from ON to OFF, the terminal management unit 53 updates the terminal management DB 5003 based on the status information sent from the communication terminal 10 indicating that the power is to be turned OFF. Change the operating state from ON line to OFF line.

The terminal extraction unit 54 searches the destination list management DB 5004 using the communication ID of the requesting terminal that has made the login request as a key, and extracts the communication ID of the destination terminal that can communicate with the requesting terminal. Furthermore, the terminal extraction unit 54 searches the destination list management DB 5004 and extracts the communication IDs of other communication terminals that have registered the communication ID of the requesting terminal as a candidate for the destination terminal.

Further, the terminal extraction unit 54 searches the terminal management DB 5003 described above using the communication ID of the extracted destination terminal candidate as a search key, and reads out the operating status for each extracted communication ID. Thereby, the terminal management unit 53 can obtain the operating status of the destination terminal candidate that can communicate with the requesting terminal that has made the login request. The terminal management unit 53 also searches the terminal management DB 5003 described above using the communication ID of the requester as a search key, and also acquires the operating state of the requester terminal that has made the login request.

The session management unit 55 controls sessions managed by the communication management system 50. The session control includes, for example, control for establishing a session, control for making the communication terminal 10 participate in the established session, control for disconnecting the session, generation of a session ID, and the like. Additionally, the session management unit 55 stores the communication ID of the communication terminal 10 that requested the start of a session, the communication ID of the destination terminal, etc. in the session management DB 5005 in association with the session ID, which is session identification information. ,to manage.

The storage/reading unit 57 is realized by the instruction from the CPU 501 and the HDD 505 shown in FIG. 5, or by the instruction from the CPU 501. The storage/readout unit 57 stores various data in the storage unit 5000 and reads various data from the storage unit 5000.

<Example of information managed by the communication management system>
Each management DB stored in the storage unit 5000 of the communication management system 50 will be explained.

通信管理システム５０の記憶部５０００に記憶された認証管理ＤＢ５００２には、例えば、表１に示されているような認証管理テーブル６０２が含まれる。この認証管理テーブル６０２では、通信管理システム５０によって管理される通信端末１０の通信ＩＤと、通信ＩＤに対応するパスワードとが関連付けられて管理されている。例えば、表１に示されている認証管理テーブル６０２において、通信ＩＤが「０１ａａ」の通信端末１０のパスワードは、「ａａａａ」であることが示されている。

The authentication management DB 5002 stored in the storage unit 5000 of the communication management system 50 includes, for example, an authentication management table 602 as shown in Table 1. In this authentication management table 602, the communication ID of the communication terminal 10 managed by the communication management system 50 and the password corresponding to the communication ID are managed in association with each other. For example, in the authentication management table 602 shown in Table 1, it is shown that the password of the communication terminal 10 whose communication ID is "01aa" is "aaaa".

通信管理システム５０の記憶部５０００に記憶された端末管理ＤＢ５００３には、例えば、表２に示されているような端末管理テーブル６０３が含まれる。この端末管理テーブル６０３では、通信端末１０の通信ＩＤ毎に、各通信端末１０を宛先とした場合の宛先名、各通信端末１０の稼動状態、後述のログイン要求情報が通信管理システム５０で受信された受信日時、及び通信端末１０のＩＰアドレスが関連付けられて管理される。例えば、表２に示されている端末管理テーブル６０３において、通信ＩＤが「０１ａａ」の通信端末１０は、端末名が「日本 本社」で、稼動状態が「ＯＮライン（通信可能）」であることが示されている。また、通信ＩＤが「０１ａａ」の通信端末１０は、通信管理システム５０でログイン要求情報が受信された日時が「２０ｘｘ年４月１０日の１３時４０分」で、ＩＰアドレスが「１．２．１．３」であることが示されている。

The terminal management DB 5003 stored in the storage unit 5000 of the communication management system 50 includes, for example, a terminal management table 603 as shown in Table 2. In this terminal management table 603, the communication management system 50 receives, for each communication ID of the communication terminal 10, the destination name when each communication terminal 10 is the destination, the operating status of each communication terminal 10, and login request information to be described later. The received date and time and the IP address of the communication terminal 10 are managed in association with each other. For example, in the terminal management table 603 shown in Table 2, the communication terminal 10 with the communication ID "01aa" has the terminal name "Japan Headquarters" and the operating state is "ON line (communication possible)". It is shown. Furthermore, the communication terminal 10 with the communication ID "01aa" has the date and time when the login request information was received by the communication management system 50 as "13:40 on April 10, 20xx" and the IP address as "1.2 .1.3''.

通信管理システム５０の記憶部５０００に記憶された宛先リスト管理ＤＢ５００４には、例えば、表３に示されているような宛先リスト管理テーブル７０１が含まれる。この宛先リスト管理テーブル７０１では、テレビ会議における通信の開始を要求する要求元端末の通信ＩＤに対して、宛先端末の候補として登録されている宛先端末の通信ＩＤが全て関連付けられて管理される。例えば、表３に示されている宛先リスト管理テーブル７０１において、通信ＩＤが「０１ａａ」である要求元端末から通信の開始を要求することができる宛先端末の候補は、通信ＩＤが「０１ｂ１」、「０１ｂ２」、及び「０１ｂ３」の通信端末である。この宛先端末の候補は、任意の要求元端末から通信管理システム５０に対する追加又は削除の要請により、通信管理システム５０によって追加又は削除されることで更新される。

The destination list management DB 5004 stored in the storage unit 5000 of the communication management system 50 includes, for example, a destination list management table 701 as shown in Table 3. In this destination list management table 701, all the communication IDs of destination terminals registered as destination terminal candidates are managed in association with the communication ID of the requesting terminal requesting the start of communication in a video conference. For example, in the destination list management table 701 shown in Table 3, the destination terminal candidates that can request the start of communication from the request source terminal whose communication ID is “01aa” are those whose communication ID is “01b1”, These are communication terminals "01b2" and "01b3". These destination terminal candidates are updated by being added or deleted by the communication management system 50 in response to a request for addition or deletion to the communication management system 50 from any request source terminal.

According to such a configuration, the requesting terminal (for example, 01aa) can only start communication with a destination terminal candidate (for example, 01b1) that has been registered in advance. This destination terminal (for example, 01b1) cannot communicate with the request source terminal unless the request source terminal (for example, 01aa) is registered as the destination terminal in the destination list management table 701. Such a mechanism is preferable in that it can reduce the possibility that unexpected communication terminals 10 will communicate with each other. However, the communication terminals 10 may be able to communicate with each other arbitrarily without the need for registration in the destination list management table 701.

通信管理システム５０の記憶部５０００に記憶されたセッション管理ＤＢ５００５には、例えば、表４に示されているようなセッション管理テーブル７０２が含まれる。このセッション管理テーブル７０２には、セッションの識別情報であるセッションＩＤ毎に、中継に使用される中継装置３０の中継装置ＩＤ、要求元端末の通信ＩＤ、宛先端末の通信ＩＤ、及びセッション参加日時等の情報が管理される。例えば、表４に示されているセッション管理テーブル７０２において、セッションＩＤ「ｓｅ２」のセッションは、要求元端末の通信ＩＤ「０１ａｄ」と宛先端末の通信ＩＤ「０１ｃａ」との間で行われていることが示されている。また、セッションＩＤ「ｓｅ２」のセッションは、中継装置ＩＤ「１１１ｂ」の中継装置３０を介して、「２０ｘｘ／４／１０ １３：１１：１１」に開始されたことが示されている。

The session management DB 5005 stored in the storage unit 5000 of the communication management system 50 includes, for example, a session management table 702 as shown in Table 4. This session management table 702 includes, for each session ID, which is session identification information, the relay device ID of the relay device 30 used for relay, the communication ID of the requesting terminal, the communication ID of the destination terminal, the date and time of session participation, etc. information is managed. For example, in the session management table 702 shown in Table 4, the session with session ID "se2" is conducted between the communication ID "01ad" of the requesting terminal and the communication ID "01ca" of the destination terminal. It has been shown that Further, it is shown that the session with the session ID "se2" was started at "20xx/4/10 13:11:11" via the relay device 30 with the relay device ID "111b".

<Processing flow>
Next, the flow of processing of the communication system 1 will be explained.

(Preparatory stage processing)
FIG. 15 is a sequence diagram illustrating an example of processing in the communication preparation stage of the communication system 1. Here, as an example, processing in a preparation stage before starting a session between the communication terminal 10A and the communication terminal 10B1 will be described. In the following description, it is assumed that the communication ID of the communication terminal 10A is "01aa" and the communication ID of the communication terminal 10B1 is "01b".

First, when the operator of the communication terminal 10A, which is the requesting terminal, performs an operation to turn on the power of the communication terminal 10A, for example, the operation input reception unit 12 accepts the operation to turn on the power, and the operator of the communication terminal 10A turns on the power. Turn on the power (step S21).

Then, the communication control unit 13, triggered by the above-described power ON, transmits login request information requesting login from the transmission/reception unit 11 to the communication management system 50 via the communication network 2 (step S22). Note that the login request information is transmitted by turning on the power of the communication terminal 10A, for example, by the operator's operation on the input unit 108, by starting an application, etc. It may be.

The login request information also includes a communication ID (communication ID of the requesting terminal) for identifying the communication terminal 10A, which is the requesting device itself, and a password. These communication ID and password are, for example, information read from the storage unit 18 via the storage/readout unit 17. Further, when login request information is transmitted from the communication terminal 10A to the communication management system 50, the communication management system 50 on the receiving side can grasp the IP address of the communication terminal 10A on the sending side.

Next, the terminal authentication unit 52 of the communication management system 50 searches the authentication management table 602 described above using the communication ID and password included in the login request information received via the transmitting/receiving unit 51 as a search key. The terminal authentication unit 52 performs authentication depending on whether the combination of communication ID and password included in the login request information received from the communication terminal 10A is included in the authentication management table 602 (step S23).

If the terminal authentication unit 52 determines that the login request is from the communication terminal 10A that has legitimate usage authority, the terminal management unit 53 uses the communication ID “01aa” of the communication terminal 10A recorded in the terminal management table 603. " operation status is changed to "ON line (communication available)". At this time, the terminal management unit 53 updates the reception date and time, and also updates the IP address of the communication terminal 10 as necessary (step S24). As a result, the terminal management table 603 includes the communication ID "01aa" of the communication terminal 10A, the operating status "ON line (communication possible)", the reception date and time "20xx.4.10.13:40", and the communication ID "01aa" of the communication terminal 10A. The IP address "1.2.1.3" is managed in association.

The transmitting/receiving unit 51 of the communication management system 50 transmits authentication result information indicating the authentication result obtained by the terminal authentication unit 52 via the communication network 2 to the requesting terminal that has requested the login. It is transmitted to the communication terminal 10A (step S25). Here, a case where the terminal authentication unit 52 determines that the communication terminal has legitimate usage authority will be described below.

The terminal extraction unit 54 of the communication management system 50 searches the destination list management table 701 using the communication ID "01aa" of the requesting terminal (communication terminal 10A) that made the login request as a search key. Thereby, the terminal management unit 53 extracts the communication ID of the destination terminal candidate that can communicate with the request source terminal (communication terminal 10A) (step S26). Here, as an example, it is assumed that "01b1", "01b2", and "01b3" are extracted as communication IDs of the destination terminal corresponding to the communication ID "01aa" of the request source terminal (communication terminal 10A).

Next, the terminal extraction unit 54 searches the terminal management table 603 using the communication IDs (“01b1”, “01b2”, “01b3”) of the extracted destination terminal candidates as search keys. Thereby, by reading the operating status for each extracted communication ID, the operating status of each communication ID ("01b1", "01b2", "01b3") is acquired (step S27).

Next, the transmitting/receiving unit 51 sends the destination state information including the operating status of each of the communication IDs (“01b1”, “01b2”, “01b3”) of the destination terminal candidates to the requesting terminal (communication terminal 10A). (step S28). As a result, the request source terminal (communication terminal 10A) can check the current operating status of the communication ID ("01b1", "01b2", "01b3") that is a candidate for the destination terminal of this request source terminal (communication terminal 10A). can be grasped. The communication terminal 10A displays the destination selection screen shown in FIG.

Further, the terminal extraction unit 54 of the communication management system 50 searches the destination list management table 701 using the communication ID "01aa" of the request source terminal (communication terminal 10A) that has made the login request as a search key. Thereby, the terminal extracting unit 54 extracts the communication IDs of other requesting terminals that have registered the communication ID "01aa" of the requesting terminal (communication terminal 10A) as a destination terminal candidate (step S29). In the destination list management table 701 shown in Table 3, the communication IDs of other request source terminals that are extracted are "01b1," "01b2," and "01b3."

Next, the terminal management unit 53 of the communication management system 50 searches the terminal management table 603 using the communication ID "01aa" of the request source terminal (communication terminal 10A) that made the login request as a search key. Thereby, the terminal management unit 53 acquires the operating state of the request source terminal (communication terminal 10A) that has made the login request (step S30).

Then, the terminal management unit 53 of the communication management system 50 determines that among the communication IDs ("01b1", "01b2", "01b3") extracted in step S29, the operating state is "ON line" in the terminal management table 603. Extract the communication IDs (“01b1”, “01b2”, “01b3”) that are “Communication possible)”.

Further, the transmitting/receiving unit 51 sends the communication terminal 10B1 corresponding to the extracted communication ID (“01b1”, “01b2”, “01b3”) the communication ID “01aa” of the requesting terminal (communication terminal 10A) and the operating state. Destination status information including "ON line (communication available)" is transmitted (step S31).

On the other hand, the other communication terminal 10B1 also performs the same processes as those in steps S22 to S32, for example, in response to an operation to turn on the power. The communication terminal 10B1 is powered on by, for example, an administrator of the mobile device 20.

<Destination selection screen>
FIG. 16 is a diagram showing an example of a destination selection screen displayed on the communication terminal 10A. A destination selection screen 1201 shown in FIG. 16 displays a message 1202 prompting the operator to select the destination communication terminal 10 and a plurality of buttons 1203 for selecting the destination communication terminal 10. The operator of the communication terminal 10A selects a destination terminal that requests participation in the session, for example, by selecting one button 1203 from among the displayed buttons 1203. In other words, which mobile device 20 is to be controlled is selected. As shown in FIG. 16, by allowing the operator to select a plurality of communication terminals 10B, the operator can grasp the status of different bases from the same location.

<Communication processing>
FIG. 17 is a sequence diagram illustrating an example of communication processing of the communication system 1. Here, an example of a communication management method for starting communication between the communication terminal 10A and the communication terminal 10B1, which is a device control terminal capable of controlling the mobile device 20, will be described.

In step S901, the operation input receiving unit 12 of the communication terminal 10A receives an operation for selecting a destination terminal (communication terminal 10B1) by the operator of the communication terminal 10A.

The transmitting/receiving unit 11 of the communication terminal 10A transmits start request information requesting the start of a session to the communication management system 50 (step S902). This start request information includes, for example, the communication ID of the requesting terminal which is the communication ID of the communication terminal 10A which is the requesting terminal, the communication ID of the destination terminal which is the communication ID of the communication terminal 10B1 which is the destination terminal, etc. . The start request information also includes information such as the IP address of the communication terminal 10A (request source IP address).

In step S903, the terminal management unit 53 of the communication management system 50, which has received the start request information from the communication terminal 10A, uses the communication ID "01aa" of the requesting terminal (communication terminal 10A) included in the start request information to The management DB 5003 is updated. For example, the terminal management unit 53 changes the operating state information corresponding to the communication ID "01aa" of the communication terminal 10A to "ON line (communicating)" and also updates the information on the reception date and time.

In step S904, the session management unit 55 of the communication management system 50 transmits start request information requesting the communication terminal 10B1, which is the destination terminal, to start a session. This start request information includes, for example, the communication ID of the request source terminal of the communication terminal 10A, which is the request source terminal.

In step S905, the communication terminal 10B1 that has received the start request information from the communication management system 50 transmits start response information to the communication management system 50. This start response information includes, for example, the communication ID of the destination terminal of the communication terminal 10B1. In this embodiment, it is assumed that the start response information is transmitted without requiring any operation on the side of the communication terminal 10B1, but the condition may be that the communication terminal 10B1 is operated by the administrator.

In step S906, the terminal management unit 53 of the communication management system 50 that has received the start response information from the communication terminal 10B1 updates the terminal management DB 5003 based on the communication ID "01b1" of the communication terminal 10B1 included in the start response information. . For example, the terminal management unit 53 changes the operating state information corresponding to the communication ID "01b1" of the communication terminal 10B1 to "ON line (communicating)" and updates the information on the reception date and time.

In step S907, the session management unit 55 of the communication management system 50 assigns a session ID, which is identification information for identifying the session. The session management unit 55 also manages the session by associating the created session ID with the communication ID of the requesting terminal (communication ID of communication terminal 10A) and the communication ID of the destination terminal (communication ID of communication terminal 10B1). Store in DB5005.

In step S909, the session management unit 55 of the communication management system 50 transmits session information to the relay device 30. This session information includes, for example, information such as the session ID created in step S907. The relay device 30 can acquire session information from the session management DB 5005 based on the session ID.

In step S910a, the session management unit 55 of the communication management system 50 transmits start instruction information that instructs the communication terminal 10A to start a session. Similarly, in step S910b, the session management unit 55 of the communication management system 50 transmits start instruction information that instructs the communication terminal 10B1 to start a session. The start instruction information includes a session ID, and the communication terminal 10 can acquire session information from the session management DB 5005 based on the session ID.

In step S911a, the communication terminal 10A establishes a session with the relay device 30 based on the received start instruction information. Similarly, in step 911b, communication terminal 10B1 establishes a session with relay device 30 based on the received start instruction information. This allows communication terminals 10A and 10B1 to participate in the same session.

In step S911c, the wide-angle image transmitting unit 31 of the wide-angle imaging device 9 transmits the equirectangular projection image to the communication terminal 10B1. For example, when the communication terminal 10B1 establishes a session, the communication terminal 10B1 instructs the wide-angle imaging device 9 to start capturing and transmitting an equirectangular projection image. Alternatively, the wide-angle imaging device 9 may always transmit the equirectangular projection image to the communication terminal 10B1 while the wide-angle imaging device 9 is powered on. Furthermore, the equirectangular projection video may be transmitted before the session is established.

The communication terminal 10A and the communication terminal 10B1 can, for example, hold a video conference by participating in a session with the same session ID and transmitting and receiving content data such as image data and audio data (S912). This image data includes an equirectangular projection image, a front image, and a rear image.

Furthermore, the communication terminal 10A can send and receive information regarding the operation of a device (for example, the mobile device 20) to and from the communication terminal 10B1 using the established session.

Note that the information regarding the operation of the device may be sent and received using a control session via the communication management system 50 without using a session, or may be sent and received using a control session via the communication management system 50, or via the communication network 2 etc. It may be transmitted and received between 10A and communication terminal 10B1.

Here, the following description will be given assuming that the communication terminal 10A transmits and receives information regarding the operation of a device (for example, the mobile device 20) to and from the communication terminal 10B1 using the established session.

In step S913, when the operator of the communication terminal 10A inputs an input on the operation screen of the device, operation instruction information corresponding to the input operation is transmitted using the session. This operation instruction information includes, for example, the communication ID of the communication terminal 10A and an operation instruction according to the operation content of the operator.

In step S914, the device control unit 19c of the communication terminal 10B1 controls the mobile device 20 via the device-to-device communication unit 19d based on the operation instruction included in the notified operation instruction information.

<Pattern 1>
Next, examples of arrangement of images of patterns 1 to 3 will be explained.
FIG. 18 shows an example of a video display screen 610 that the communication terminal 10A displays on the display unit 109 in pattern 1. The video display screen 610 has a video display field 611, a front video button 612, a back video button 613, and a mobile device operation button 614. The video display column 611 in FIG. 18(a) displays a spherical video. The display control unit 16 of the communication terminal 10A pastes the equirectangular projection image transmitted from the communication terminal 10B onto a three-dimensional sphere to generate a spherical image (see FIG. 10).

The operator can rotate the spherical image horizontally and vertically using a finger, a mouse, or the like. You can also zoom out by pinching your fingers in, and zoom in by pinching them out. In this manner, the operator can display any predetermined area image Q in the video display field 611.

The front image button 612 is a button for switching the image displayed in the image display field 611 to the front image, and the rear image button 613 is a button for switching the image displayed in the image display field 611 to the rear image. The moving device operation button 614 has four buttons, and each of the four buttons corresponds to a button for moving the moving device 20 forward, backward, left, and right. Since the mobile device operation button 614 is displayed overlapping the spherical image, the display area can be made wider and easier to see than when the mobile device operation button 614 is displayed independently.

FIG. 18B shows a video display screen 610 in which a front video is displayed in the video display field 611 when the front video button 612 is pressed. Since the front image button 612 has been pressed, the front image is displayed in the image display field 611. Further, instead of the front image button 612, a spherical image button 615 is displayed. The omnidirectional image button 615 is a button for switching the image displayed in the image display field 611 to an omnidirectional image.

Since the mobile device operation button 614 is displayed overlapping the front image or the rear image, the display area can be made wider and easier to see than when the mobile device operation button 614 is displayed independently.

FIG. 18C shows a video display screen 610 in which a rear video is displayed in the video display field 611 when the rear video button 613 is pressed. In place of the rear image button 613, an omnidirectional image button 615 and a front image button 612 are displayed.

In this way, by pressing the front image button 612, the rear image button 613, and the omnidirectional image button 615, the images displayed in the image display field 611 can be alternately switched.

FIG. 19 is an example of a flowchart showing a procedure for switching images in an example of the arrangement of images of pattern 1. The process in FIG. 19 starts when the communication terminal 10A receives at least one of a spherical image, a front image, and a rear image.

First, the display control unit 16 displays a spherical image on the display unit 109 (S101). In this case, a mobile device operation button 614 is displayed superimposed on the spherical image. When a spherical image is displayed, the operation input reception unit 12 accepts rotation, enlargement, and reduction operations, and the display control unit 16 displays a predetermined area T in accordance with the operation in the image display field 611.

Note that displaying the omnidirectional image first is just an example, and the front image or the rear image may be displayed first. They may be displayed in the order received.

The display control unit 16 displays a front image button 612 and a rear image button 613 on the display unit 109 (S102).

The operation input reception unit 12 determines whether the front image button 612 has been pressed (S103). If the front image button 612 is not pressed, the process advances to step S106.

When the front image button 612 is pressed, the display control unit 16 displays the front image (S104). The display control unit 16 displays the rear image button 613 and the omnidirectional image button 615 on the display unit 109 (S105).

The operation input reception unit 12 determines whether the rear image button 613 has been pressed (S106). If the rear image button 613 is not pressed, the process advances to step S109.

When the rear image button 613 is pressed, the display control unit 16 displays the rear image (S107). The display control unit 16 displays the front image button 612 and the omnidirectional image button 615 on the display unit 109 (S108).

The operation input reception unit 12 determines whether the omnidirectional image button 615 has been pressed (S109). If the omnidirectional image button 615 is pressed, the process advances to step S101. If the omnidirectional image button 615 is not pressed, the process advances to step S103.

As described above, the communication terminal 10A can display the image of pattern 1 by receiving the press of the front image button 612, the rear image button 613, or the omnidirectional image button 615.

In this way, in the image arrangement example of pattern 1, since the omnidirectional image and the front image (rear image) can be switched and displayed on one display unit 109, the sizes of the omnidirectional image and the front image (rear image) can be changed. It is possible to maintain the clearness of the image, and to prevent it from becoming difficult for the operator to see. When the spherical image is displayed, the operator can rotate the spherical image to check the surroundings in 360 degrees. Further, since the frontal image has a high resolution, the frontal image displayed is effective when reading characters on a signboard, etc., or when checking the structure of an object.

<Pattern 2>
FIG. 20 shows an example of a video display screen 610 that the communication terminal 10A displays on the display unit 109 in pattern 2. The video display screen 610 in FIG. 20(a) has an entire display field 621, a front video field 622, and a mobile device operation button 614. In the entire display field 621, an equirectangular cylindrical projection image is displayed. The equirectangular projection image of FIG. 20(a) is an example of a room in which a model room is located, and four walls and some fixtures are shown. In this way, in an equirectangular projection image, a 360-degree subject is represented on a plane.

A front image column 622 displays a front image. Although the shelf 622a is shown in FIG. 20A, the distortion is large in the equirectangular projection image, whereas in the front image there is almost no distortion, although the angle of view is narrow. Further, although the frontal image is displayed larger than the equirectangular projection image, since the frontal image has a higher resolution than the equirectangular projection image, it becomes easier for the operator to obtain detailed information from the frontal image. For example, it is effective when an operator reads text on a signboard or when checking the structure of a product. Further, although the equirectangular projection image is distorted and displayed smaller than the frontal image, since it is always displayed on the image display screen 610, the operator can grasp the surrounding situation of the mobile device 20.

Since the mobile device operation button 614 is displayed overlapping the front image, the display area can be made wider and easier to see than when the mobile device operation button 614 is displayed independently.

The front image field 622 has a rear image button 613, which allows the user to switch the front image to the rear image.

FIG. 20(b) shows an example of the video display screen 610 when the rear video button is pressed. The video display screen 610 in FIG. 20(b) has a rear video field 623 instead of the front video field 622, and displays a rear video. The rear image field 623 has a front image button 612, which allows the user to switch the rear image to the front image.

Further, in both of FIGS. 20(a) and 20(b), the user can place the front image or the rear image on the upper side and the equirectangular projection image on the lower side. For example, when the user swipes (or flicks) the entire display field 621 downward, or swipes the front image field 622 or the rear image field 623 upward, the communication terminal 10A is displayed as shown in FIG. 20(c). A video display screen 610 as shown in (d) can be displayed.

In FIG. 20(c), the front image is displayed on the upper side and the equirectangular projection image is displayed on the lower side, and in FIG. 20(d), the rear image is displayed on the upper side and the equirectangular projection image is displayed on the lower side. In this way, the user can display the front image, rear image, and equirectangular projection image in the arrangement that he or she wants.

FIG. 21 is an example of a flowchart showing a procedure for displaying images in the example of the arrangement of images of pattern 2. The process in FIG. 21 starts when the communication terminal 10A receives at least one of a spherical image, a front image, and a rear image.

The display control unit 16 displays the equirectangular projection image in the entire display field 621 (S201).

The display control unit 16 displays the front image in the front image field 622 (S202). In this case, a mobile device operation button 614 is displayed superimposed on the spherical image. Note that the rear images may be displayed first, or may be displayed in the order in which they are received.

Next, the operation input reception unit 12 determines whether the rear image button 613 has been pressed (S203). When the rear image button 613 is pressed, the display control unit 16 displays the rear image in the rear image field 623 (S204).

Next, the operation input reception unit 12 determines whether a vertical switching operation such as a swipe has been input (S205). When a vertical swapping operation is input, the display control unit 16 swaps the top and bottom of the front image field 622 and the whole display field 621, or the rear image field 623 and the whole display field 621 (S206).

By doing so, the equirectangular projection image and the front image, or the equirectangular projection image and the rear image can be displayed simultaneously on the image display screen 610.

Next, another display example of pattern 2 will be explained using FIG. 22. FIG. 22 shows an example of a video display screen 610 that the communication terminal 10A displays on the display unit 109 in pattern 2. The video display screen 610 in FIG. 22(a) has an omnidirectional display field 631, a front video field 622, and a mobile device operation button 614. The omnidirectional image is displayed in the omnidirectional display column 631. The display control unit 16 of the communication terminal 10A pastes the equirectangular projection image transmitted from the communication terminal 10B onto a three-dimensional sphere to generate a spherical image (see FIG. 10). In FIGS. 22(b) to 22(d), as in FIGS. 20(b) to 20(d), the front image and the rear image are switched, or the spherical image and the front image, or the spherical image and the rear image are switched. Indicates the switched state.

In FIG. 22(a), a copy machine 631a is shown in the spherical display column 631, but this is for convenience of explanation, and the operator can display any predetermined area image Q of the spherical image on a spherical display. It can be displayed in column 631. The front image column 622 is similar to that in FIG. 20(a). Note that the mobile device operation button 614 may be displayed in the omnidirectional display field 631.

In FIG. 22(a), the omnidirectional display field 631 and the front image field 622 are approximately the same size. The operator can enlarge any desired direction in the celestial sphere display field 631. Although the front image field 622 is smaller than that in FIG. 20(a), since the front image has a high resolution, the operator can obtain detailed information from the front image. Therefore, by having the spherical display field 631 and the front image field 622 of the same size, it becomes easier to obtain the respective advantages of the spherical image and the front image.

FIG. 23 is an example of a flowchart showing a procedure for displaying an image in another display example of pattern 2 in FIG. 22. In FIG. 23, the equirectangular projection image in FIG. 21 has been changed to a spherical image, but the overall processing flow may be the same as that in FIG. 21. On the video display screen 610, the omnidirectional image and the front image, or the omnidirectional image and the rear image can be displayed simultaneously.

In patterns 1 and 2, an example is shown in which the communication terminal 10A has a vertically long screen, but the screen of the communication terminal 10A may also be horizontally long. In this case, it is preferable to maintain the aspect ratio of the video, but the aspect ratio may be changed to scale the video vertically or horizontally.

<Pattern 3>
FIG. 24 shows an example of a video display screen 610 that the communication terminal 10A displays on the display unit 109 in pattern 3. The video display screen 610 in FIG. 24 has an omnidirectional display field 631, a sub-front video field 642, a sub-back video field 643, and a mobile device operation button 614. The omnidirectional image is displayed in the omnidirectional display column 631. A front image is displayed in the sub front image column 642, and a rear image is displayed in the sub rear image column 643. The sub-front image column 642 and the sub-rear image column 643 show subjects whose directions are 180 degrees different from each other when viewed from the mobile device 20.

In the video display screen 610 of FIG. 24, the omnidirectional display field 631 is larger than the sub-front video field 642 and the sub-back video field 643. Therefore, the spherical image is the main image, and the front image and rear image are the secondary images. The operator can enlarge any desired direction in the celestial sphere display field 631. Furthermore, since the front image and the rear image are always displayed, the moving direction of the moving device can always be grasped, and the situation on the rear side can also be grasped at all times.

Since the mobile device operation button 614 is displayed overlapping the spherical image, the display area can be made wider and easier to see than when the mobile device operation button 614 is displayed independently.

Note that the spherical display field 631 is larger than the sub-front image field 642 and the sub-rear image field 643 is merely an example, and the front image may be displayed larger than the spherical image and the rear image. However, the rear image may be displayed larger than the spherical image and the front image. Further, it would be better if the operator could arbitrarily set what to display in the three display fields (the omnidirectional display field 631, the sub-front image field 642, and the sub-back image field 643).

FIG. 25 is an example of a flowchart showing a procedure for displaying images in the example of arrangement of images of pattern 3. The process in FIG. 25 starts when the communication terminal 10A receives at least one of the omnidirectional image, the front image, and the rear image.

The display control unit 16 displays the omnidirectional image in the omnidirectional display column 631 (S301). In this case, a mobile device operation button 614 is displayed superimposed on the spherical image.

The display control unit 16 displays the front image in the sub-front image field 642 (S302). The display control unit 16 displays the rear image in the sub-rear image column 643 (S303).

By doing so, the omnidirectional image, the front image, and the rear image can be displayed simultaneously on the image display screen 610.

Next, another display example of pattern 3 will be explained using FIG. 26. FIG. 26 shows an example of a video display screen 610 that the communication terminal 10A displays on the display unit 109 in pattern 3. The video display screen 610 in FIG. 26 has a main display field 651, one or more various video fields 652 (various video fields 652a to 652c), and mobile device operation buttons 614. The main display field 651 displays an omnidirectional image. The display control unit 16 of the communication terminal 10A pastes the equirectangular projection image transmitted from the communication terminal 10B onto a three-dimensional sphere to generate a spherical image (see FIG. 10).

In one or more of the various video fields 652, not only a front video and a rear video can be displayed, but also a spherical video of an arbitrary predetermined area image Q can be displayed.

The main display field 651 occupies almost the entire surface of the display section 109, and one or more various video fields 652 are movably superimposed on the main display field 651. In FIG. 26, three various video columns 652a to 652c are displayed. The various video fields 652 are windows, and the operator can change the position, change the size, delete (make invisible, close window), minimize, and create new ones.

When displaying the video in pattern 3, the display control unit 16 displays the front video and the rear video in the various video fields 652. Furthermore, any one or more predetermined area images Q of the spherical image are duplicated and displayed in the various image column 652. Since the operator can change the predetermined area image Q, the predetermined area image Q displayed in the various video fields 652 in the initial state may be anywhere. However, since the front image and the rear image are displayed in the various image column 652, it is preferable to duplicate the predetermined area image Q other than these. For example, the right side and the left side in the horizontal direction with respect to the front of the moving device 20.

The positions of the various video fields 652 in the initial state may be, for example, the four corners of the main display field 651. The operator can move the various video fields 652 to arbitrary positions.

Due to the sizes of the main display field 651 and the various video fields 652, the main display field 651 is the main image, and the images displayed in the various video field 652 are positioned as sub-images. The operator can view spherical images on a large screen, and can enlarge any direction he or she wishes to view. In addition, since the front image and the rear image are displayed in the various image column 652, the situation in the moving direction and the rear direction of the moving device 20 can be grasped. Furthermore, when a spherical image is displayed in the various image column 652 (the spherical image displayed in the various image column 652 is an example of a duplicate image), the situation around the mobile device 20 in any direction can be grasped. I can do it. When the operator performs an operation to newly generate various video fields 652, the operation input reception unit 12 accepts this, and the display control unit 16 newly displays a spherical video in the various video fields 652.

Since the various video fields 652 can be moved, the operator can move them to a location where they will not get in the way. You can also delete or minimize it. The operation input reception unit 12 receives such an operation, and the display control unit 16 hides or minimizes the various video fields 652.

Conventionally, such a display method would require as many imaging devices as the number of various image columns 652, but in this embodiment, any predetermined area image Q can be duplicated and displayed from a spherical image. It is possible to constantly display images from many directions at low cost.

Since the mobile device operation button 614 is displayed overlapping the spherical image, the display area can be made wider and easier to see than when the mobile device operation button 614 is displayed independently.

FIG. 27 is an example of a flowchart showing a procedure for displaying images in the example of the arrangement of images of pattern 3 in FIG. 26. The process in FIG. 27 starts when the communication terminal 10A receives at least one of a spherical image, a front image, and a rear image.

The display control unit 16 displays the omnidirectional image in the main display field 651 (S401). In this case, a mobile device operation button 614 is displayed superimposed on the spherical image.

The display control unit 16 displays the front image superimposed on the omnidirectional image (S402). The display control unit 16 displays the rear image superimposed on the omnidirectional image (S403). The initial positions of the various video fields 652 showing the front video and rear video are predetermined, or are the positions set by the user when the video display screen 610 of FIG. 26 was displayed last time.

The operation input reception unit 12 determines whether a new various video field 652 has been added (S404). Any operation method may be used, but for example, there is a method of displaying a menu by long-pressing or double-tapping, and then pressing "Add various video fields". Note that the menu may display selection buttons for the front image and rear image, but if the front image or the rear image is already displayed, it is preferable that the selection buttons for the front image and the rear image not be displayed. As a result, one or less front images and rear images can be displayed at the same time as spherical images.

When the various video field 652 is added, the display control unit 16 sets the spherical video of the predetermined area T of the long-pressed or double-tapped location in the new various video field 652, and displays it superimposed on the spherical video. (S405).

Although patterns 1 to 3 have been described above, the operator can arbitrarily switch between patterns 1 to 3 by a predetermined operation such as pressing a button.

<About switching patterns>
Patterns 1 to 3 can be switched arbitrarily, but if the patterns are automatically switched in response to some event, usability for the operator will be improved.

The trigger for pattern switching will be explained using FIG. 28. FIG. 28 is an example of a diagram illustrating the timing at which patterns are automatically switched.

FIG. 28(a) will be explained. FIG. 28(a) starts when the communication terminal 10A establishes a session.

By establishing the session, the display control unit 16 of the communication terminal 10A can display a spherical image, a front image, and a rear image. First, the display control unit 16 displays a spherical image in the image display field 611 in pattern 1 (S1001). That is, a spherical image is displayed on the entire surface of the display unit 109.

Next, the display control unit 16 rotates the spherical image horizontally at a constant speed (S1002). This is achieved by keeping the angle of view α constant, the elevation angle ea constant in the horizontal direction, and changing the azimuth angle aa at a constant speed. Therefore, the predetermined area image Q appears to be rotating to the operator.

The display control unit 16 determines whether or not the image has been rotated 360 degrees in the horizontal direction (S1003). This can be easily determined by memorizing the direction (ea, aa) of the center point CP before starting the rotation. The rotation continues until it rotates 360 degrees in the horizontal direction.

When rotated 360 degrees in the horizontal direction (Yes in S1003), the display control unit 16 switches to pattern 2 or pattern 3 (S1004).

Immediately after the session is established, the operator cannot see the surrounding situation, but immediately after the session is established, the predetermined area image Q is rotated 360 degrees and then displayed in patterns 2 and 3, so the operator can grasp the situation. From there, you can check the situation in front of you using the front-view video and start moving. Note that the front image may be displayed in a large size as pattern 1 is.

FIG. 28(b) will be explained. FIG. 28(b) starts when the communication terminal 10A establishes a session.

By establishing the session, the display control unit 16 of the communication terminal 10A can display a spherical image, a front image, and a rear image. First, the display control unit displays an equirectangular projection image and a frontal image in pattern 2 (S2001). That is, a small equirectangular projection image is displayed above the display unit 109, and a front image is displayed below.

The display control unit 16 determines whether a predetermined period of time has elapsed since the pattern 2 was displayed (S2002). The pattern 2 display continues until the predetermined time period elapses.

If the predetermined time has elapsed (Yes in S2002), the display control unit 16 switches to pattern 1 or pattern 3 (S2003).

Immediately after the session is established, the operator does not know the surrounding situation, but since the equirectangular projection image of pattern 2 is displayed for a predetermined period of time after the session is established, the operator can use patterns 1 and 1 after understanding the situation. is displayed, so you can grasp the situation and then check the situation in front of you using the frontal image and start moving. Note that pattern 2 may continue to be displayed.

<Summary>
As explained above, the communication terminal 10A of this embodiment displays the omnidirectional image and the front image (rear image) in an appropriately arranged manner, so that the operator can see the overall situation around the mobile device 20. You can also check the high-resolution images in front of you or behind you. By displaying the spherical image and the front image (rear image) in the arrangement of patterns 1 to 3, the operator can move the moving device 20 by utilizing the spherical image and the front image (rear image). You can check your surroundings. Furthermore, patterns 1 to 3 can be switched arbitrarily or automatically.

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited to these examples in any way, and various modifications can be made without departing from the gist of the present invention. and substitutions can be added.

For example, in the present embodiment, the wide-angle imaging device 9 transmits an equirectangular cylindrical projection image to the communication terminal 10B, and the communication terminal 10B transmits an equirectangular cylindrical projection image to the communication terminal 10A. A session may be connected with the terminal 10A and the equirectangular projection video may be transmitted. In this case, the wide-angle imaging device 9 becomes one form of the communication terminal 10, and becomes the communication terminal 10 that transmits images but does not receive them.

Furthermore, although it has been described in this embodiment that the rear image is transmitted, the communication terminal 10B does not have to transmit the rear image. Furthermore, the communication terminal 10B may transmit an image with a normal angle of view separately from the front image and the rear image.

Furthermore, patterns 1 to 3 are not independent, but one communication terminal 10 can transition from pattern 1 to pattern 2, from pattern 2 to pattern 3, from pattern 3 to pattern 1 (or vice versa). .

Further, in this embodiment, the mobile device 20 that travels on land has been described as an example, but the mobile device 20 may also fly in the air, such as a drone. Moreover, it may sail on the sea or may move underwater. You may also move underground, through narrow streets, or underground. Furthermore, when moving on land, the animal may move on wheels, move on multiple legs such as two, three, or four legs, or move on caterpillars (registered trademark).

Further, the display method of the present embodiment may be applied to images from a wide-angle imaging device 9 and a flat imaging device mounted not only on a telepresence robot but also on a device moved by a human.

Further, in this embodiment, an example in which one operator operates one mobile device 20 has been mainly described, but it is also possible for one operator to operate a plurality of mobile devices 20. In this case, the communication terminal 10A establishes sessions with a plurality of communication terminals 10B. Conversely, it is also possible for multiple operators to operate one mobile device 20. In this case, for example, the last operation instruction information sent to the communication terminal 10B may be valid, or the concept of operation authority may be introduced so that only the communication terminal 10A with operation authority can transmit operation instructions. . The operating authority can be transferred by an operator or the like.

Further, in this embodiment, it has been described that the operation instruction information is transmitted in a communication session, but it may be transmitted in a content data session.

Further, in this embodiment, the communication terminal 10 communicates via the relay device 30, but the communication terminal 10 may communicate without using the relay device. For example, WebRTC (Web Real-Time Communication) is known as a communication protocol for such communication.

Further, in the configuration example shown in FIG. 14 and the like, in order to facilitate understanding of the processing by the communication management system 50 and the communication terminal 10, the structure is divided according to main functions. The present invention is not limited by the method of dividing the processing units or the names thereof. The processing of the communication management system 50 and the communication terminal 10 can also be divided into more processing units depending on the processing content. Furthermore, one processing unit can be divided to include more processing.

Furthermore, in this embodiment, for convenience of explanation, the communication management system 50 and the relay device 30 have been described as separate devices, but a device in which the functions of both are integrated provides the functions of the communication management system 50 and the relay device 30. It's okay.

Further, the communication system 1 may include a plurality of communication management systems 50, and the functions of the communication management system 50 may be distributed and installed in a plurality of servers.

Furthermore, one or more of the databases included in the storage unit 5000 of the communication management system 50 may exist on the communication network 2.

Note that the transmitting/receiving section 11 is an example of a communication means, the display control section 16 is an example of a display processing means, and the operation input reception section 12 is an example of a reception means.

1 communication system 9 wide-angle imaging device 10 communication terminal 20 mobile device 30 relay device 50 communication management system

Japanese Patent Application Publication No. 2013-112030

Claims (15)

Communicate with another communication terminal installed in the mobile device or a communication unit included in the mobile device via a network, and transmit operation instruction information to the mobile device according to an input operation from an operator to the other communication terminal or the communication unit. A communication terminal for transmitting data to the department,
Communication means for receiving a first image transmitted by the other communication terminal or the communication unit, and a wide-angle second image captured by a wide-angle imaging device connected to or integrated with the other communication terminal or the mobile device. and,
A controller that controls the first video and the second video received by the communication means to be displayed on the display screen, and also controls the movement of the mobile device, which is displayed on the display screen. a display processing means for displaying the first image in an overlapping manner;
The communication terminal is characterized in that the controller receives the input operation from the display screen. The display processing means displays both the first video and the second video on the same display screen,
The first image is a front image or a rear image of the mobile device or the communication terminal, and the second image is a wider-angle image than the first image, and includes a view of the surroundings of the mobile device. It is a captured spherical image or an equirectangular projection image,
The spherical image or the equirectangular projection image is a captured image of an area including the area where the first image was captured,
The communication terminal according to claim 1, wherein the second image displayed on the display screen is a predetermined area of the omnidirectional image or the equirectangular projection image. 3. The display processing means displays the second image above the display screen and displays the first image below the second image. communication terminal. 2. The display processing means displays one of the second images above the display screen, and displays one of the first images below the second image. Or the communication terminal according to 3. 2. The communication terminal according to claim 1, wherein the display processing means displays a display area for the first image larger than a display area for the second image on the display device. 2. The communication terminal according to claim 1, wherein the display processing means displays a display area for the second image larger than a display area for the first image on the display device. 2. The communication terminal according to claim 1, wherein the display processing means displays the second video display area and the first video display area in the same size on the display device. 6. The communication terminal according to claim 5, wherein the second image is an equirectangular cylindrical projection image that captures 360 degrees around the other communication terminal or the mobile device. The communication means receives the two first videos transmitted by the other communication terminal or the communication unit and having different imaging directions,
7. The communication terminal according to claim 6, wherein the display processing means displays two of the first video and the second video on the display device. 2. The communication terminal according to claim 1, wherein the display processing means displays only either the first video or the second video on a display device according to a user's operation. When the communication terminal starts communication with the other communication terminal or the communication unit,
9. The communication terminal according to claim 8, wherein the display processing means displays the second image while rotating it in a horizontal direction, and displays the first image when rotated by a certain angle. Communicate with another communication terminal installed in the mobile device or a communication unit included in the mobile device via a network, and transmit operation instruction information to the mobile device according to an input operation from an operator to the other communication terminal or the communication unit. The communication terminal that sends data to the
Communication means for receiving a first image transmitted by the other communication terminal or the communication unit, and a wide-angle second image captured by a wide-angle imaging device connected to or integrated with the other communication terminal or the mobile device. and,
The first video and the second video received by the communication means are controlled to be displayed on the display screen, and the controller that controls the movement of the mobile device is controlled to display the first video and the second video received by the communication means on the display screen. Function as a display processing means for displaying over the first video ,
A program characterized in that the controller receives the input operation from the display screen. Communicate with another communication terminal installed in the mobile device or a communication unit included in the mobile device via a network, and transmit operation instruction information to the mobile device according to an input operation from an operator to the other communication terminal or the communication unit. A display method carried out by a communication terminal transmitting data to the department,
The communication means transmits a first image transmitted by the other communication terminal or the communication unit, and a wide-angle second image captured by a wide-angle imaging device connected to or integrated with the other communication terminal or the mobile device. a step of receiving;
A display processing means controls the first video and the second video received by the communication means to be displayed on the display screen, and also controls a controller that controls movement of the mobile device to display the first video and the second video on the display screen. displaying the first image in a superimposed manner on the first image displayed on the first image;
A display method, wherein the controller receives the input operation from the display screen. Communicate with another communication terminal installed in the mobile device or a communication unit included in the mobile device via a network, and transmit operation instruction information to the mobile device according to an input operation from an operator to the other communication terminal or the communication unit. The communication terminal that sends data to the
Communication means for receiving a first image transmitted by the other communication terminal or the communication unit, and a wide-angle second image captured by a wide-angle imaging device connected to or integrated with the other communication terminal or the mobile device. and,
The first video and the second video received by the communication means are controlled to be displayed on the display screen, and the controller that controls the movement of the mobile device is controlled to display the first video and the second video received by the communication means on the display screen. Function as a display processing means for displaying over the first video ,
A computer-readable recording medium storing a program, wherein the controller receives the input operation from the display screen. A system including a mobile device equipped with another communication terminal or a communication unit, and a communication terminal that transmits operation instruction information for the mobile device according to an input operation from an operator to the other communication terminal or the communication unit. There it is,
The communication terminal is
Communication means for receiving a first image transmitted by the other communication terminal or the communication unit, and a wide-angle second image captured by a wide-angle imaging device connected to or integrated with the other communication terminal or the mobile device. and,
The first video and the second video received by the communication means are controlled to be displayed on the display screen, and the controller that controls the movement of the mobile device is controlled to display the first video and the second video received by the communication means on the display screen. a display processing means for displaying the first image in an overlapping manner;
The controller receives the input operation from the display screen,
A system characterized in that the mobile device moves according to the operation instruction information transmitted from the communication terminal.

Images