JP7371369B2 - Communication terminals and image communication systems - Google Patents

Description

The present invention relates to a communication terminal and an image communication system.

2. Description of the Related Art Conference systems that conduct remote conferences with remote locations via communication networks such as the Internet have become widespread. In this conference system, in a conference room where one of the parties, such as an attendee, is holding a remote conference, the communication terminal of the remote conference system is used to record and record images and voices of the party, etc. in the conference room. The data is collected, converted into digital data, and sent to the other party's communication terminal. This allows a video call to be made by displaying an image on the display in the conference room of the other party and outputting audio through the speaker, making it possible to conduct a conference between remote locations in conditions similar to an actual conference. (See Patent Document 1).

In addition, a photographing device capable of acquiring spherical panoramic images in real time is connected to the communication terminal, and the spherical panoramic images acquired from this photographing device are transmitted to each communication terminal of the other party. 2. Description of the Related Art There is a known technique for displaying a predetermined region image representing a predetermined region that is a part of a spherical panoramic image on a display or the like (see Patent Document 2). Thereby, the user at each base can independently determine and display a predetermined region image that represents a predetermined region of interest to the user within the entire spherical panoramic image.

In addition, in a relay server that relays video between two or more video communication terminals, when superimposing a figure on the video by the video relay server, the target object in the video specified by the video communication terminal is A technique for superimposing predetermined figures is known (see Patent Document 3). Thereby, even if the position of the object in the image moves, it is possible to provide a composite image in which the figure follows the target object.

However, in the conventional technology, when displaying a plurality of images on the receiver side, the sender side cannot change the display hierarchy of the plurality of images. Therefore, a plurality of images are displayed in a front-back relationship that is not intended by the sender, and the images interfere with each other, resulting in a problem that the images are difficult to see and understand in a conference system or the like.

The invention according to claim 1 is an image communication system having a communication terminal that transmits a plurality of image data, and another communication terminal that displays an image related to each of the plurality of image data transmitted from the communication terminal. The communication terminal includes a reception unit that accepts changes in the display hierarchy of the plurality of image data, and an order in which images are combined with the base image based on the change in the display hierarchy received by the reception unit. a transmitter configured to transmit display layer information for displaying images related to each of the plurality of image data in display layers to the other communication terminal, and the other communication terminal transmits information from the communication terminal The image communication system is characterized by comprising a display control unit that displays the plurality of images in the predetermined display hierarchy based on display hierarchy information.

As explained above, according to the present invention, when multiple images are displayed in a front-to-back relationship that is not intended by the sender, or when the front-to-back relationship between multiple images is displayed during a conference, You can change the display hierarchy if you wish. This allows you to change the display hierarchy and display images in cases where multiple images are displayed in a front/back relationship that was not intended by the sender, or if you want to change the front/back relationship of multiple images. This has the effect of making it easier to see and understand.

(a) is a left side view of the photographing device, (b) is a front view of the photographing device, and (c) is a plan view of the photographing device. It is a usage image diagram of a photographing device. (a) is a hemispherical image (front) photographed by the photographing device, (b) is a hemispherical image (back) photographed by the photographing device, and (c) is a diagram showing an image expressed by the Mercator projection. (a) A conceptual diagram showing a state where a sphere is covered by a Mercator image, and (b) a diagram showing a spherical panoramic image. FIG. 3 is a diagram showing the positions of a virtual camera and a predetermined area when the omnidirectional panoramic image is a three-dimensional solid sphere. (a) is a three-dimensional perspective view of FIG. 5, and (b) is a diagram showing a state in which an image of a predetermined area is displayed on a display of a communication terminal. 3 is a diagram showing a relationship between predetermined area information and an image of a predetermined area T. FIG. FIG. 2 is a diagram showing points in a three-dimensional Euclidean space based on spherical coordinates. 1 is a schematic diagram of an image communication system according to an embodiment of the present invention. FIG. 2 is a hardware configuration diagram of a photographing device. FIG. 2 is a hardware configuration diagram of a video conference terminal. It is a hardware configuration diagram of a communication management system and a PC. It is a hardware configuration diagram of a smartphone. FIG. 2 is a functional block diagram of a part of the image communication system. FIG. 2 is a functional block diagram of a part of the image communication system. FIG. 3 is a conceptual diagram showing an image type management table. FIG. 3 is a conceptual diagram showing a photographing device management table. FIG. 3 is a conceptual diagram showing a predetermined area management table. FIG. 2 is a conceptual diagram showing a display hierarchy (display position) management table. It is an explanatory diagram of a display hierarchy and a display position. FIG. 3 is a conceptual diagram showing a session management table. FIG. 3 is a conceptual diagram showing an image type management table. FIG. 3 is a conceptual diagram showing a predetermined area management table. FIG. 3 is a sequence diagram showing a process for participating in a specific communication session. FIG. 3 is a diagram showing a communication session (virtual conference room) selection screen. FIG. 3 is a sequence diagram showing image type information management processing. It is an image diagram showing the state of a video call. FIG. 3 is a sequence diagram showing communication processing of photographed image data, sound data, and document image data in a video call. A display example of the display at base B is shown, and (a) is displayed as is from the image data sent from the imaging devices 1a and 1b without creating a spherical panoramic image or creating a predetermined area image. FIG. 29(b) shows a case where a spherical panoramic image and a predetermined area image are created from the image data sent from the imaging devices 1a and 1b. ) is a diagram showing a case where the predetermined area image in FIG. 29(b) is changed. FIG. 7 is a sequence diagram showing a process of sharing predetermined area information. FIG. 7 is a sequence diagram showing another process of sharing predetermined area information. FIG. 3 is a sequence diagram showing a process of sharing display hierarchy information. This is an example of a screen at base A. This is an example of a screen for changing the display hierarchy. FIG. 3 is a sequence diagram showing a process of sharing display position information. This is an example of a screen for changing the display position. 7 is a flowchart showing image display processing. It is an example of a display screen of base B after execution of display hierarchy change processing. This is an example of the display screen of base B after executing the display position change process.

Embodiments of the present invention will be described below with reference to FIGS. 1 to 39.

<<Overview of the embodiment>>
<How to generate a spherical panoramic image>
A method for generating a spherical panoramic image will be described using FIGS. 1 to 7.

First, the appearance of the photographing device 1 will be explained using FIG. 1. The photographing device 1 is a digital camera for obtaining photographed images that are the basis of a three-dimensional spherical (360°) panoramic image. Note that FIG. 1(a) is a left side view of the imaging device, FIG. 1(b) is a front view of the imaging device, and FIG. 1(c) is a plan view of the imaging device.

As shown in FIG. 1(a), the photographing device 1 has a size that can be held by a human with one hand. Further, as shown in FIGS. 1(a), 1(b), and 1(c), the upper part of the imaging device 1 includes an image sensor 103a on the front side (front side) and an image sensor 103a on the back side (rear side). ) is provided with an image sensor 103b. These image sensors (image sensors) 103a and 103b are used in conjunction with optical members (for example, fisheye lenses 102a and 102b, which will be described later) that can capture hemispherical images (angle of view of 180° or more). Further, as shown in FIG. 1(b), an operating section 115 such as a shutter button is provided on the surface opposite to the front side of the photographing device 1.

Next, using FIG. 2, the usage status of the photographing device 1 will be explained. Note that FIG. 2 is an image diagram of how the photographing device is used. As shown in FIG. 2, the photographing device 1 is, for example, held by a user and used to photograph objects around the user. In this case, two hemispherical images can be obtained by capturing images of objects around the user using the imaging device 103a and the imaging device 103b shown in FIG. 1, respectively.

Next, using FIGS. 3 and 4, an outline of the process from which a spherical panoramic image is created from an image photographed by the photographing device 1 will be explained. In addition, Fig. 3(a) is a hemispherical image (front side) taken with the imaging device, Fig. 3(b) is a hemispherical image (back side) taken with the imaging device, and Fig. 3(c) is represented by the Mercator projection. FIG. 2 is a diagram showing an image (hereinafter referred to as a "Mercator image"). FIG. 4(a) is a conceptual diagram showing a state where a sphere is covered by a Mercator image, and FIG. 4(b) is a diagram showing a spherical panoramic image.

As shown in FIG. 3A, the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by a fisheye lens 102a, which will be described later. Further, as shown in FIG. 3(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by a fisheye lens 102b, which will be described later. The hemispherical image (front side) and the hemispherical image (rear side) reversed by 180 degrees are combined by the imaging device 1, and a Mercator image is created as shown in FIG. 3(c).

Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is pasted to cover the spherical surface as shown in Figure 4(a), and the Mercator image is pasted to cover the spherical surface as shown in Figure 4(b). A spherical panoramic image as shown is created. In this way, the spherical panoramic image is represented as an image in which the Mercator image faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the spherical panoramic image may be a still image or a moving image.

As described above, since the spherical panoramic image is an image pasted to cover a spherical surface, it feels strange when viewed by humans. Therefore, by displaying a predetermined area (hereinafter referred to as a "predetermined area image") of a part of the spherical panoramic image as a flat image with less curvature, it is possible to perform a display that does not give a sense of discomfort to humans. This will be explained using FIGS. 5 and 6.

Note that FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional solid sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the spherical image CE displayed as a three-dimensional solid sphere. Further, FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a predetermined area image when displayed on a display. Furthermore, in FIG. 6(a), the omnidirectional image CE shown in FIG. 4 is represented by a three-dimensional solid sphere CS. Assuming that the spherical image CE generated in this manner is a three-dimensional sphere CS, the virtual camera IC is located outside the spherical image CE, as shown in FIG. The predetermined area T in the spherical image CE is a shooting area of the virtual camera IC, and is specified by the predetermined area information indicating the shooting direction and angle of view of the virtual camera IC in the three-dimensional virtual space including the spherical image CE. Ru.

The predetermined area image Q shown in FIG. 6(a) is displayed on a predetermined display as an image of the imaging area of the virtual camera IC, as shown in FIG. 6(b). The image shown in FIG. 6(b) is a predetermined area image represented by initially set (default) predetermined area information. Note that the predetermined area information may be indicated by the imaging area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the position coordinates of the virtual camera IC. In the following, description will be given using the photographing direction (rH, rV) and angle of view (α) of the virtual camera IC.

The relationship between the predetermined area information and the image of the predetermined area T will be explained using FIG. 7. Note that FIG. 7 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 7, rH represents horizontal radian, rV represents vertical radian, and α represents angle of view. That is, the attitude of the virtual camera IC is changed so that the point of view of the virtual camera IC indicated by the shooting direction (rH, rV) becomes the center point CP of the predetermined area T that is the shooting area of the virtual camera IC. Become. The predetermined area image Q is an image of a predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP. L is the distance between an arbitrary vertex of the predetermined region T and the center point CP (2L is a diagonal line). In FIG. 7, the trigonometric function expressed by the following (Equation 1) generally holds true.

L/f=tan(α/2)...(Formula 1)
FIG. 8 is a diagram showing points in a three-dimensional Euclidean space using spherical coordinates. Let the position coordinates of the center point CP expressed in a spherical polar coordinate system be (r, θ, φ). (r, θ, φ) are the radius vector, polar angle, and azimuth angle, respectively. The vector radius r is the distance from the origin to the center point CP of the three-dimensional virtual space including the spherical panoramic image, and is therefore equal to f. FIG. 8 is a diagram showing these relationships. Hereinafter, the description will be made using the position coordinates (r, θ, φ) of the virtual camera IC.

<Overview of image communication system>
Next, the outline of the configuration of the image communication system of this embodiment will be described using FIG. 9. FIG. 9 is a schematic diagram of the configuration of the image communication system of this embodiment.

As shown in FIG. 9, the image communication system of this embodiment includes photographing devices 1a and 1b, video conference terminals 3a and 3b, displays 4a and 4b, a communication management system 5, a PC (Personal Computer) 6, and a PC 7. , a photographing device 8, and a smartphone 9, and can communicate via a communication network 100 such as the Internet. The connection form of the communication network 100 may be either wireless or wired.

Of these, the photographing devices 1a and 1b are special digital cameras for photographing objects, scenery, etc. to obtain two hemispherical images that are the basis of a spherical panoramic image, as described above. On the other hand, the photographing device 8 is a general digital camera for photographing objects, scenery, etc. to obtain general two-dimensional images.

Video conference terminals 3a and 3b are terminals dedicated to video conferences, and display images of video calls on displays 4a and 4b, respectively, via wired cables such as USB (Universal Serial Bus) cables. The video conference terminal 3a normally takes pictures with a camera 312 shown in FIG. 11, which will be described later, but when it is connected via a wired cable to the cradle 2a to which the photographing device 1a is attached, the photographing device 1a is prioritized and the source of the spherical panoramic image is taken. Two hemisphere images can be obtained. Further, the video conference terminal 3a is connected to the PC 6 and can display the screen of the PC 6. When using a wired cable, the cradle 2a not only performs communication between the photographing device 1a and the video conference terminal 3a, but also plays a role of supplying power to the photographing device 1a and supporting the photographing device 1a. Note that here, the photographing device 1a, the cradle 2a, the video conference terminal 3a, the display 4a, and the PC 6 are located at the same base A. Furthermore, at base A, four users A1, A2, A3, and A4 are participating in a video call. On the other hand, the video conference terminal 3d and the display 4d are placed at the same base D. Furthermore, at base D, three users D1, D2, and D3 are participating in a video call.

The communication management system 5 manages and controls communication between the video conference terminals 3a, 3b, the PC 7, and the smartphone 9, and manages the types of image data (general images and special images) to be transmitted and received. Therefore, the communication management system is also a communication control system. Here, the general image is a flat image. The special image is a spherical panoramic image. Note that the communication management system 5 is installed at a service company or the like that provides video communication services. Further, the communication management system 5 may be constructed by a single computer, or may be constructed by a plurality of computers in which each section (function, means, or storage section) is divided and arbitrarily assigned.

The PC 6 generates material image data that is image data of a material image displayed in a video conference. Material images include, but are not limited to, images that are displayed, created, or edited with a general-purpose application running on a PC6, or images that are played back on a PC6 from an image taken with a general digital camera, etc. I can't do it.

By attaching the photographing device 8 to the PC 7, video communication becomes possible. Note that here, the PC 7 and the photographing device 8 are placed at the same base, base C. Further, at base C, one user C is participating in a video call.

The smartphone 9 displays an image of the video call on a display 917, which will be described later, provided on the smartphone 9. The smartphone 9 normally takes pictures using a CMOS (Complementary Metal Oxide Semiconductor) sensor 905, which will be described later, provided on its own device, but it also uses wireless communication technology such as WiFi (Wireless Fidelity) or BLUETOOTH (registered trademark) to take pictures. Two hemisphere image data that are the basis of the spherical panoramic image obtained by the photographing device 1b can be acquired. When using wireless communication technology, the cradle 2b serves only to supply power to the photographing device 1b and to support the photographing device 1b. Note that here, the photographing device 1b, the cradle 2b, and the smartphone 9 are placed at the same base, base B. Furthermore, at base B, two users B1 and B2 are participating in a video call.

Further, the video conference terminals 3a, 3d, the PC 7, and the smartphone 9 are examples of communication terminals. OpenGL ES is installed on each communication terminal, and it can create predetermined area information that indicates a part of the spherical panoramic image, and use it to create spherical panoramic images sent from other communication terminals. It is possible to create an image of a predetermined area.

Note that the arrangement of each terminal (communication terminal, display, photographing device), device, and user shown in FIG. 9 is one example, and other examples may be used. For example, at the base C, instead of the photographing device 8, the user may use a photographing device capable of photographing a spherical panoramic image. Communication terminals also include digital televisions, smart watches, car navigation devices, and the like. In addition, hereinafter, when representing an arbitrary photographing device among the photographing devices 1a and 1b, it will be expressed as "photographing device 1." Furthermore, when representing any video conference terminal among the video conference terminals 3a and 3b, it is expressed as "video conference terminal 3." Furthermore, when representing an arbitrary display among the displays 4a and 4b, it is represented as "display 4."

<<Hardware configuration of embodiment>>
Next, the hardware configurations of the photographing device 1, video conference terminal 3, communication management system 5, PC 6, PC 7, and smartphone 9 of this embodiment will be described in detail using FIGS. 10 to 13. Note that since the photographing device 8 is a general camera, a detailed description thereof will be omitted.

<Hardware configuration of imaging device 1>
First, the hardware configuration of the imaging device 1 will be explained using FIG. 10. FIG. 10 is a hardware configuration diagram of the photographing device 1. As shown in FIG. In the following, the photographing device 1 is assumed to be a spherical (omnidirectional) photographing device using two image sensors, but the number of image sensors may be any number greater than or equal to two. Also, it does not necessarily have to be a device exclusively for omnidirectional photography, and it may be possible to have substantially the same functions as the photography device 1 by attaching an aftermarket omnidirectional photography unit to a normal digital camera, smartphone, etc. .

As shown in FIG. 10, the imaging device 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, and a ROM (Read Only Memory). ) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, operation section 115, network I/F 116, communication section 117, and antenna 117a.

Among these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging units provided corresponding to each wide-angle lens. It includes elements 103a and 103b. The image sensors 103a and 103b are image sensors such as CMOS sensors and CCD (Charge Coupled Device) sensors that convert optical images formed by fisheye lenses 102a and 102b into electrical signal image data and output the image data, and horizontal or vertical synchronization signals of the image sensors. It has a timing generation circuit that generates a pixel clock and a pixel clock, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set.

The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I/F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are connected separately to the imaging control unit 105 via a serial I/F bus (such as an I2C bus). The image processing unit 104 and the imaging control unit 105 are connected to the CPU 111 via a bus 110. Furthermore, a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, a network I/F 116, a communication unit 117, an electronic compass 118, and the like are also connected to the bus 110.

The image processing unit 104 takes in image data output from the image sensors 103a and 103b through the parallel I/F bus, performs predetermined processing on each image data, and then synthesizes these image data. , create Mercator image data as shown in FIG. 3(c).

Generally, the imaging control unit 105 uses the I2C bus to set commands and the like in register groups of the imaging devices 103a and 103b, using the imaging control unit 105 as a master device and the imaging devices 103a and 103b as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to take in status data and the like of the register groups of the imaging elements 103a and 103b, and sends it to the CPU 111.

Further, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Depending on the photographing device 1, it may have a preview display function or a function corresponding to moving image display on a display (for example, the display of the video conference terminal 3a). In this case, image data is output continuously from the image sensors 103a and 103b at a predetermined frame rate (frames/minute).

The imaging control unit 105 also functions as a synchronization control unit that synchronizes the output timing of image data of the imaging elements 103a and 103b in cooperation with the CPU 111, as will be described later. Note that in this embodiment, the photographing device 1 is not provided with a display, but may be provided with a display.

The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in sound data output from the microphone 108 through the I/F bus, and performs predetermined processing on the sound data.

The CPU 111 controls the overall operation of the photographing device 1 and executes necessary processing. ROM112 stores various programs for CPU111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111, data being processed, and the like. In particular, the DRAM 114 stores image data that is currently being processed by the image processing unit 104 and data of processed Mercator images.

The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that has both display and operation functions, and the like. The user inputs various shooting modes, shooting conditions, etc. by operating the operation buttons.

The network I/F 116 is a general term for interface circuits (USB I/F, etc.) with external media such as an SD card, a personal computer, and the like. Further, the network I/F 116 may be wireless or wired. The Mercator image data stored in the DRAM 114 is recorded on an external media via this network I/F 116, or is transmitted to an external device such as the video conference terminal 3a via the network I/F 116 as necessary. or

The communication unit 117 communicates with an external device such as the video conference terminal 3a via an antenna 117a provided in the photographing device 1 using short-range wireless technology such as WiFi or the NFC (Near Field Communication) standard. This communication unit 117 can also transmit Mercator image data to an external device of the video conference terminal 3a.

The electronic compass 118 calculates the direction and inclination (roll rotation angle) of the photographing device 1 from the earth's magnetism, and outputs the direction and inclination information. This orientation/tilt information is an example of related information (metadata) in accordance with Exif, and is used for image processing such as image correction of captured images. Note that the related information also includes data such as the date and time when the image was taken and the data capacity of the image data.

<Hardware configuration of video conference terminal>
Next, the hardware configuration of the video conference terminal 3 will be explained using FIG. 11. FIG. 11 is a hardware configuration diagram of the video conference terminal. As shown in FIG. 11, the video conference terminal 3 includes a CPU 301, ROM 302, RAM 303, flash memory 304, SSD 305, media I/F 307, operation button 308, power switch 309, bus line 310, network I/F 311, Equipped with a camera 312, an image sensor I/F 313, a microphone 314, a speaker 315, a sound input/output I/F 316, a display I/F 317, an external device connection I/F 318, a short-range communication circuit 319, and an antenna 319a of the short-range communication circuit 319. ing.

Among these, the CPU 301 controls the entire operation of the video conference terminal 3. The ROM 302 stores programs used to drive the CPU 301, such as IPL (Initial Program Loader). RAM 303 is used as a work area for CPU 301. The flash memory 304 stores various data such as communication programs, image data, and sound data. An SSD (Solid State Drive) 305 controls reading and writing of various data to and from the flash memory 304 under the control of the CPU 301 . Note that an HDD may be used instead of the SSD. The media I/F 307 controls reading or writing (storage) of data to a recording medium 306 such as a flash memory. The operation button 308 is a button that is operated when selecting the destination of the video conference terminal 3, for example. The power switch 309 is a switch for switching the power of the video conference terminal 3 ON/OFF.

Further, the network I/F 311 is an interface for data communication using the communication network 100 such as the Internet. The camera 312 is a type of built-in imaging means that images a subject and obtains image data under the control of the CPU 301. The image sensor I/F 313 is a circuit that controls driving of the camera 312. The microphone 314 is a type of built-in sound collecting means for inputting sound. The sound input/output I/F 316 is a circuit that processes input/output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301 . The display I/F 317 is a circuit that transmits image data to the external display 4 under the control of the CPU 301. External device connection I/F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit based on the NFC standard, BLUETOOTH (registered trademark), or the like.

Further, the bus line 310 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 301 shown in FIG. 11.

The display 4 is a type of display means configured with a liquid crystal or organic EL that displays an image of a subject, operation icons, and the like. Further, the display 4 is connected to the display I/F 317 by a cable 4c. This cable 4c may be a cable for analog RGB (VGA) signals, a cable for component video, HDMI (registered trademark) (High-Definition Multimedia Interface) or DVI (Digital Video). It may also be a cable for interactive) signals.

Note that the camera 312 includes a lens and a solid-state image sensor that converts light into electric charge to digitize an image (video) of a subject, and a CMOS sensor, a CCD sensor, or the like is used as the solid-state image sensor. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I/F 318 using a USB (Universal Serial Bus) cable or the like. When an external camera is connected, the external camera is driven with priority over the built-in camera 312 under the control of the CPU 301. Similarly, when an external microphone or external speaker is connected, the external microphone or speaker is given priority over the built-in microphone 314 and built-in speaker 315, respectively, under the control of the CPU 301. Driven by external speakers.

Furthermore, the recording medium 306 is configured to be detachable from the video conference terminal 3. Further, as long as it is a non-volatile memory that reads or writes data under the control of the CPU 301, it is not limited to the flash memory 304, and an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used.

<Communication management system, PC hardware configuration>
Next, the hardware configurations of the communication management system 5, PC 6, and PC 7 will be explained using FIG. 12. FIG. 12 is a hardware configuration diagram of the communication management system and PC. Note that since the communication management system 5, PC6, and PC7 are all computers and have the same configuration, the configuration of the communication management system 5 will be described below, and the description of the configurations of the PC6 and PC7 will be omitted.

The communication management system 5 includes a CPU 501 that controls the overall operation of the communication management system 5, a ROM 502 that stores programs used to drive the CPU 501 such as IPL, a RAM 503 that is used as a work area for the CPU 501, and programs for the communication management system 5. an HDD (Hard Disk Drive) 505 that controls reading or writing of various data to the HD 504 according to the control of the CPU 501, and reading or writing (storage) of data to a recording medium 506 such as a flash memory. A media drive 507 to control, a display 508 to display various information such as a cursor, menu, window, characters, or images, a network I/F 509 to communicate data using the communication network 100, characters, numbers, various instructions, etc. a keyboard 511 equipped with a plurality of keys for inputting information; a mouse 512 for selecting and executing various instructions, selecting a processing target, moving a cursor, etc.; and a CD-RW (Compact Disc) as an example of a removable recording medium. -ReWritable) 513, and bus lines such as an address bus and a data bus for electrically connecting each of the above components as shown in FIG. 510.

<Hardware configuration of smartphone>
Next, the hardware of the smartphone will be explained using FIG. 13. FIG. 13 is a hardware configuration diagram of the smartphone. As shown in FIG. 13, the smartphone 9 includes a CPU 901, a ROM 902, a RAM 903, an EEPROM 904, a CMOS sensor 905, an acceleration/direction sensor 906, a media I/F 908, and a GPS receiving section 909.

Among these, the CPU 901 controls the entire operation of the smartphone 9. The ROM 902 stores programs used to drive the CPU 901 such as IPL. RAM903 is used as a work area for CPU901. The EEPROM 904 reads or writes various data such as smartphone programs under the control of the CPU 901. The CMOS sensor 905 images a subject (mainly a self-portrait) under the control of the CPU 901 and obtains image data. The acceleration/direction sensor 906 is a variety of sensors such as an electronic magnetic compass, a gyro compass, and an acceleration sensor that detect geomagnetism. A media I/F 908 controls reading or writing (storage) of data to a recording medium 907 such as a flash memory. GPS receiving section 909 receives GPS signals from GPS satellites.

The smartphone 9 also includes a long-distance communication circuit 911, a camera 912, an image sensor I/F 913, a microphone 914, a speaker 915, a sound input/output I/F 916, a display 917, an external device connection I/F 918, a short-distance communication circuit 919, An antenna 919a of a short-range communication circuit 919 and a touch panel 921 are provided.

Among these, the long distance communication circuit 911 is a circuit that communicates with other devices via the communication network 100. The camera 912 is a type of built-in imaging means that images a subject and obtains image data under the control of the CPU 901. The image sensor I/F 913 is a circuit that controls driving of the camera 912. The microphone 914 is a type of built-in sound collecting means for inputting audio. The sound input/output I/F 916 is a circuit that processes input/output of sound signals between the microphone 914 and the speaker 915 under the control of the CPU 901 . The display 917 is a type of display means such as a liquid crystal or an organic EL display that displays an image of a subject, various icons, and the like. External device connection I/F 918 is an interface for connecting various external devices. The short-range communication circuit 919 is a communication circuit based on the NFC standard, BLUETOOTH (registered trademark), or the like. The touch panel 921 is a type of input means by which the user operates the smartphone 9 by pressing the display 917.

The smartphone 9 also includes a bus line 910. The bus line 910 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 901.

Note that recording media such as CD-ROMs in which the above programs are stored and HDs in which these programs are stored can both be provided domestically or internationally as program products.

<<Functional configuration of embodiment>>
Next, the functional configuration of this embodiment will be explained using FIGS. 14 to 21. 14 and 15 are functional block diagrams of a part of the image communication system.

<Functional configuration of photographing device 1a>
As shown in FIG. 14, the photographing device 1a includes a receiving section 12a, an imaging section 13a, a sound collecting section 14a, a communication section 18a, and a storage/reading section 19a. Each of these parts is a function or means that is realized when any of the constituent elements shown in FIG. It is.

Further, the photographing device 1 has a storage unit 1000a constructed by a ROM 112, an SRAM 113, and a DRAM 114 shown in FIG. The storage unit 1000a stores a GUID (Globally Unique Identifier) of the device itself.

Note that the photographing device 1b includes a reception section 12b, an imaging section 13b, a sound collection section 14b, a communication section 18b, a storage/reading section 19b, and a storage section 1000b. , the imaging section 13a, the sound collecting section 14a, the communication section 18a, the storage/reading section 19a, and the storage section 1000a, and therefore the description thereof will be omitted.

(Functional configuration of photographing device 1a)
Next, each functional configuration of the photographing device 1a will be described in more detail using FIGS. 10 and 14.

The reception unit 12a of the photographing device 1 is mainly realized by the processing of the operation unit 115 and the CPU 111 shown in FIG. 10, and accepts operation input from the user.

The imaging unit 13a is mainly realized by the processing of the imaging unit 101, the image processing unit 104, the imaging control unit 105, and the CPU 111 shown in FIG. 10, and images a landscape or the like to obtain photographed image data.

The sound collection unit 14a is realized by the processing of the sound processing unit 108 and the sound processing unit 109 shown in FIG. 10, and the CPU 111, and collects sounds around the photographing device 1a.

The communication unit 18a is mainly realized by the processing of the CPU 111, and can communicate with the communication unit 38a of the video conference terminal 3a using short-range wireless communication technology such as the NFC standard, BLUETOOTH (registered trademark), and WiFi.

The storage/reading unit 19a is mainly realized by the processing of the CPU 111 shown in FIG. Read out.

<Functional configuration of video conference terminal 3a>
As shown in FIG. 14, the video conference terminal 3a includes a transmission/reception section 31a, a reception section 32a, an image/sound processing section 33a, a display control section 34a, a judgment section 35a, a creation section 36a, a calculation section 37a, and a communication section. 38a, and a storage/readout section 39a. Each of these parts is realized by one of the components shown in FIG. 11 operating in response to an instruction from the CPU 301 according to a program for the video conference terminal 3a expanded from the flash memory 304 onto the RAM 303. It is a function or means.

Further, the video conference terminal 3a has a storage section 3000a constructed from a ROM 302, a RAM 303, and a flash memory 304 shown in FIG. This storage unit 3000a has an image type management DB 3001a, a photographing device management DB 3002a, a predetermined area management DB 3003a, and a display hierarchy management DB 3004a. Among these, the image type management DB 3001a is composed of an image type management table shown in FIG. 16. The photographing device management DB 3002a is composed of a photographing device management table shown in FIG. 17. The predetermined area management DB 3003a is configured by a predetermined area management table shown in FIG. The display hierarchy management DB 3004a is composed of a display hierarchy management table shown in FIG. 19.

The video conference terminal 3d includes a transmitting/receiving section 31d, a reception section 32d, an image/sound processing section 33d, a display control section 34d, a judgment section 35d, a creation section 36d, a calculation section 37d, a communication section 38d, a storage/reading section 39d, and a storage section 3000d, respectively, a transmitting/receiving section 31a, a reception section 32a, an image/sound processing section 33a, a display control section 34a, a judgment section 35a, a creation section 36a, a calculation section 37a, and a storage section 3000d in the video conference terminal 3a. Since functions similar to those of the communication section 38a, storage/readout section 39a, and storage section 3000a are realized, descriptions of these sections will be omitted. Furthermore, in the storage unit 3000d of the video conference terminal 3d, an image type management DB 3001d, a photographing device management DB 3002d, a predetermined area management DB 3003d, and a display hierarchy management DB 3004d are constructed. Since the data structure is similar to that of the device management DB 3002a, the predetermined area management DB 3003a, and the display hierarchy management DB 3004a, a description thereof will be omitted.

(Image type management table)
FIG. 16 is a conceptual diagram showing the image type management table. In this image type management table, the image data ID, the IP address, which is an example of the destination of the sending terminal, and the source name are stored and managed in association with each other. Among these, the image data ID is an example of image data identification information for identifying image data when performing video communication. The same image data ID is added to image data transmitted from the same transmission source terminal. This allows the destination terminal (receiving communication terminal) to identify the source terminal of the received image data. The IP address of the source terminal indicates the IP address of the communication terminal that transmits the image data indicated by the associated image data ID. The source name is a name for specifying the photographing device that outputs the image data indicated by the associated image data ID, and is an example of image type information. This source name is a name created by each communication terminal such as the video conference terminal 3a according to a predetermined name naming rule.

For example, a communication terminal with an IP address of "1.2.1.3" sends image data indicated by image data IDs "RS001", "RS002", and "RS003" to a communication terminal with an IP address of "1.2.1.4". ” transmits the image data indicated by the image data ID “RS004”, and the communication terminal whose IP address is “1.3.1.4” transmits the image data indicated by the image data ID “RS005”. It is shown that Furthermore, the image types indicated by the source name of each communication terminal are "Video_Theta", "Video", "Video_Contents", "Video_Theta", and "Video", and these are in order of the image types "special image" and " ``general image'', ``material image'', ``special image'', and ``general image''. Note that the special image here is a spherical panoramic image.

Note that data other than image data related to general images, special images, and material images may also be managed in association with the image data ID. Data other than image data is, for example, sound data.

(Photography equipment management table)
FIG. 17 is a conceptual diagram showing the imaging device management table. In this imaging device management table, vendor IDs and product IDs of the GUIDs of imaging devices that can obtain two hemispherical images that are the basis of a spherical panoramic image are stored and managed. As the GUID, for example, a vendor ID (VID) and a product ID (PID) used in a USB device can be used. The vendor ID and product ID are stored when the communication terminal such as a video conference terminal is shipped from the factory, but may be additionally stored after the communication terminal is shipped from the factory.

(Prescribed area management table)
FIG. 18 is a conceptual diagram showing the predetermined area management table. This predetermined area management table includes the IP address of the communication terminal that is the source of the captured image data, the IP address of the communication terminal that is the destination of the captured image data, and the predetermined area that is being displayed on the communication terminal that is the destination of the captured image data. Predetermined area information indicating images is stored and managed in association with each other. Note that the communication terminal to which the captured image data is transmitted is also the communication terminal to which the predetermined area information is transmitted. The predetermined area information is a conversion parameter for converting a captured image into an image of a predetermined area T (predetermined area image) in this captured image, as shown in FIGS. 6 and 7. Note that the IP address is an example of destination information, and the destination information includes a MAC (Media Access Control) address, a terminal ID (Identification) for identifying a communication terminal, and the like. Further, here, the IP address is expressed as a simplified version of the IPv4 address. The IP address may be IPv6.

For example, if the IP address of the video conference terminal 3a is "1.2.1.3", the first to third rows of the predetermined area management table in FIG. The data is transmitted via the communication management system 5 to the video conference terminal 3d with the IP address "1.2.2.3", the PC 7 with the IP address "1.3.1.3", and the PC 7 with the IP address "1.2.2.3". .3.2.3" was sent to the smartphone 9. Furthermore, the video conference terminal 3d is managed to be the communication terminal that is the source of the predetermined area information (r=10, θ=20, φ=30). Similarly, the PC 7 is managed to be the communication terminal that is the source of the predetermined area information (r=20, θ=30, φ=40). Furthermore, the smartphone 9 is managed to be the communication terminal that is the source of the predetermined area information (r=30, θ=40, φ=50).

In addition, the transmitting/receiving unit 31a sends new predetermined area information that includes the same set of IP addresses as the IP address of the communication terminal that is the source of the photographed image data and the IP address of the communication terminal that is the destination of the photographed image data that has already been managed. If received, the storage/reading unit 39a rewrites the predetermined area information already managed by the newly received predetermined area information.

(Display hierarchy management table)
FIG. 19 is a conceptual diagram showing the display hierarchy management table. This display hierarchy management table stores and manages image data IDs managed in the image type management table shown in FIG. 16 in association with a display hierarchy that is unique for each source terminal IP address. ing.

Further, the display hierarchy management table is managed by storing the image data ID managed in the image type management table shown in FIG. 16 in association with the display coordinates. The display hierarchy management table manages the display coordinates as well as the display hierarchy, so in that case it can also be called a display position management table.

The display hierarchy will be explained using FIG. 20. FIG. 20 shows a Mercator image of a spherical image into which a material image, which is a combination source image, has been combined. The display hierarchy represents the order in which images are synthesized with respect to the base image. When the display hierarchy is "base", it represents the base image that is the source of composition. Note that there must be one line in which "base" is stored for each source terminal IP address. If the display hierarchy is a number, it indicates the order in which images are synthesized on top of the base image, and images with the lowest number are synthesized in order. In other words, the larger the number, the more it will be displayed in the front. Note that the numbers do not necessarily have to be consecutive numbers, but they do need to be unique for each sending terminal IP address. As an example, in FIG. 20, a material image whose display hierarchy is "1" is combined with a Mercator image of a spherical video whose display hierarchy is "base".

Next, display coordinates will be explained using FIG. 20. FIG. 20 shows a Mercator image of a spherical image into which a material image, which is a combination source image, has been combined. The display coordinates consist of starting point coordinates and ending point coordinates. When the starting point is the upper left vertex position of the material image that is the source of synthesis, the starting point coordinates are expressed as (0,0) and the ending point coordinates are (xn,yn). At this time, if you want to synthesize the starting point coordinates (0,0) and the ending point coordinates (xn, yn) to the positions corresponding to the coordinates (x1, y1) and coordinates (x2, y2) in the Mercator image, respectively, the display hierarchy (position ) The starting point coordinates are associated with (x1, x2) and the ending point coordinates are (x2, y2) in the management table. Note that the IP address is an example of destination information, and the destination information includes a MAC (Media Access Control) address, a terminal ID (Identification) for identifying a communication terminal, and the like. Further, here, the IP address is expressed as a simplified version of the IPv4 address. The IP address may be IPv6.

(Functional configuration of video conference terminal 3a)
Next, each functional configuration of the video conference terminal 3a will be described in more detail using FIGS. 11 and 14.

The transmitting/receiving unit 31a of the video conference terminal 3a is mainly realized by the processing of the network I/F 311 and the CPU 301 shown in FIG. Send and receive.

The receiving unit 32a is mainly realized by processing by the operation buttons 308 and the CPU 301, and receives various selections or inputs from the user. Further, in addition to the operation button 308, a touch panel or the like may be used as other input means.

The image/sound processing unit 33a is realized by instructions from the CPU 301 shown in FIG. 11, and performs image processing on image data obtained by capturing an image of a subject with the camera 312. Furthermore, after the user's voice is converted into an audio signal by the microphone 314, the image/sound processing unit 33a performs audio processing on sound data related to this audio signal.

Furthermore, the image/sound processing section 33a performs image processing on the image data received from another communication terminal based on image type information such as the source name in order for the display control section 34 to display the image on the display 4. Let's do it. Specifically, when the image type information indicates that the image is a special image, the image/sound processing unit 33a inputs the image data (for example, each image shown in FIGS. 3(a) and 3(b) A spherical panoramic image is created by converting the hemispherical image data into a spherical panoramic image as shown in FIG. 4(b). Further, if there is a document image to be synthesized with the omnidirectional panoramic image, a composite image that is an image obtained by synthesizing the document image with the omnidirectional panoramic image is generated based on the document image data and display position information. Note that compositing includes superimposing or overwriting a document image, which is a second image, on a part of the spherical panoramic image, which is the first image. Alternatively, a document image may be embedded in place of a portion of the spherical panoramic image. The image/sound processing unit 33a also creates a predetermined area image as shown in FIG. 6(b). Further, the image/sound processing unit 33a outputs an audio signal related to sound data received from another communication terminal via the communication management system 5 to the speaker 315, and causes the speaker 315 to output audio.

The display control unit 34a is mainly realized by the processing of the display I/F 317 and the CPU 301, and performs control to display various images, characters, etc. on the display 4.

The determining unit 35a is mainly realized by the processing of the CPU 301, and, for example, determines the type of image related to the image data received from the photographing device 1a, but is not limited to this, and performs various other types of determinations regarding the image data.

The creation unit 36a is mainly realized by the processing of the CPU 301, and based on the result determined by the determination unit 35a to be a general image, a special image (here, a spherical panoramic image), or a document image, the creation unit 36a creates an image according to the naming rule described above. , create a source name, which is an example of image type information. For example, when the determining unit 35a determines that the image is a general image, the creating unit 36a creates a source name "Video" indicating that the image is a general image. On the other hand, if the determining unit 35a determines that the image is a special image, the creating unit 36a creates a source name "Video_Theta" indicating that it is a special image. Further, when the determining unit 35a determines that the image is a material image, the creating unit 36a creates a source name "Video_Contents" indicating that the image is a material image.

The calculation unit 37a is mainly realized by the processing of the CPU 301, and calculates display hierarchy information.

The communication unit 38a is mainly realized by the short-range communication circuit 319, the antenna 318a, and the processing of the CPU 301, and is implemented by the communication unit 18a of the photographing device 1a and short-range wireless technology using the NFC standard, BLUETOOTH (registered trademark), WiFi, etc. can be communicated by. Although the communication section 38a and the transmitting/receiving section 31a have been described as having separate communication units, they may have a shared structure.

The storage/reading unit 39a is mainly realized by the processing of the CPU 301 shown in FIG. Read out.

<Functional configuration of communication management system>
Next, each functional configuration of the communication management system 5 will be explained in detail using FIGS. 12 and 15. The communication management system 5 includes a transmitting/receiving section 51, a determining section 55, a generating section 56, and a storage/reading section 59. Each of these units has a function or function that is realized when any of the components shown in FIG. It is a means.

Further, the communication management system 5 includes a storage unit 5000 constructed by a RAM 503 and an HD 504 shown in FIG. A session management DB 5001, an image type management DB 5002, and a predetermined area management DB 5003 are constructed in this storage unit 5000. Among these, the session management DB 5001 is configured by the session management table shown in FIG. 21. The image type management DB 5002 is composed of an image type management table shown in FIG. 22. The predetermined area management DB 5003 is composed of a predetermined area management table shown in FIG. 23.

(Session management table)
FIG. 21 is a conceptual diagram showing the session management table. In this session management table, session IDs and IP addresses of participating communication terminals are stored and managed in association with each other. Among these, the session ID is an example of session identification information for identifying a communication session for realizing a video call, and is generated for each virtual conference room. The session ID is also managed by each communication terminal such as the video conference terminal 3a, and is used when selecting a communication session at each communication terminal. The IP address of the participating communication terminal indicates the IP address of the communication terminal that participated in the virtual conference room indicated by the associated session ID.

(Image type management table)
FIG. 22 is a conceptual diagram showing the image type management table. In addition to each piece of information managed in the image type management table shown in FIG. 16, the image type management table shown in FIG. 22 is associated with the same session ID as the session ID managed in the session management table. It is well managed. Here, the virtual conference rooms indicated by the same session ID "se101" have IP addresses "1.2.1.3", "1.2.1.4", "1.3.1. It is shown that three communication terminals "4" are participating. In the communication management system 5, a new communication terminal must manage the same image data ID, source terminal IP address, and image type information that are managed by a communication terminal such as the video conference terminal 3a. This is to send image type information and the like to a communication terminal that is already engaged in a video call and a newly participating communication terminal when entering a virtual conference room. Thereby, there is no need to send and receive image type information and the like between the communication terminal that is already engaged in a video call and the communication terminal that has newly joined.

(Prescribed area management table)
FIG. 23 is a conceptual diagram showing the predetermined area management table. This predetermined area management table basically has the same data structure as the predetermined area management table shown in FIG. 18. However, as will be described later, since the transmitter/receiver 51 transmits the latest predetermined area information to each communication terminal at fixed intervals (for example, 30 seconds), , all the predetermined area information received by the transmitter/receiver 51 is stored without being deleted. In FIG. 21, newer predetermined area information is managed at a higher level.

(Functional configuration of communication management system)
Next, each functional configuration of the communication management system 5 will be explained in detail using FIGS. 12 and 15.

The transmission/reception unit 51 of the communication management system 5 is mainly realized by the processing of the network I/F 509 and the CPU 501 shown in FIG. (or information).

The judgment unit 55 is mainly realized by the processing of the CPU 501, and performs various judgments.

The generation unit 56 is mainly realized by the processing of the CPU 501, and generates an image data ID.

The storage/readout section 59 is mainly realized by the processing of the HDD 505 and the CPU 501 shown in FIG. information).

<Functional configuration of PC>
Next, each functional configuration of the PC 6 will be described in detail using FIGS. 12 and 14.

As shown in FIG. 14, the PC 6 includes a reception section 62, a display control section 64, a communication section 68, and a storage/reading section 69. Each of these units is a function or means realized by one of the components shown in FIG. 12 operating in response to an instruction from the CPU 501 according to a PC 6 program expanded from the HD 504 onto the RAM 503. .

Further, the PC 6 has a storage unit 6000 constructed by a ROM 502, a RAM 503, and an HD 504 shown in FIG.

(Each functional configuration of PC)
The reception section 62 of the PC 6 is mainly realized by the processing of the keyboard 511, mouse 512, and CPU 501, and realizes the same functions as the reception section 32a. The display control unit 64 is mainly realized by the processing of the CPU 501, and performs control to display various images, characters, etc. on the display 508. The communication unit 68 is mainly realized by the processing of the CPU 501, and can communicate with the communication unit 38a of the video conference terminal 3a using short-range wireless communication technology such as the NFC standard, BLUETOOTH (registered trademark), and WiFi. The storage/readout unit 69a is realized by the processing of the CPU 501, and stores various data (or information) in the storage unit 6000 and reads out various data (or information) from the storage unit 6000.

<Functional configuration of a PC as a communication terminal>
Next, the functional configuration of the PC 7 will be described in detail using FIGS. 12 and 15. The PC 7 basically has the same functions as the video conference terminal 3a. That is, as shown in FIG. 14, the PC 7 includes a transmitter/receiver section 71, a reception section 72, an image/sound processing section 73, a display control section 74, a judgment section 75, a creation section 76, a calculation section 77, and a communication section 78. , and a storage/readout section 79. Each of these units is a function or means realized by one of the components shown in FIG. 12 operating in response to an instruction from the CPU 501 according to a PC 7 program expanded from the HD 504 onto the RAM 503. .

Furthermore, the PC 7 has a storage unit 7000 constructed from a ROM 502, a RAM 503, and an HD 504 shown in FIG. In this storage unit 7000, an image type management DB 7001, a photographing device management DB 7002, a predetermined area management DB 7003, and a display hierarchy management DB 7004 are constructed. Note that the image type management DB 7001, imaging device management DB 7002, predetermined area management DB 7003, and display hierarchy management DB 7004 have the same data structure as the image type management DB 3001a, imaging device management DB 3002a, predetermined area management DB 3003a, and display hierarchy management DB 3004a, respectively. Therefore, their explanations will be omitted.

(Each functional configuration of a PC as a communication terminal)
The transmitting/receiving section 71 of the PC 7 is mainly realized by the processing of the network I/F 509 and the CPU 501 shown in FIG. 12, and realizes the same functions as the transmitting/receiving section 31a.

The reception section 72 is mainly realized by the processing of the keyboard 511, the mouse 512, and the CPU 501, and realizes the same functions as the reception section 32a. The image/sound processing section 73 is mainly realized by commands from the CPU 501, and realizes the same functions as the image/sound processing section 33a. The display control unit 74 is mainly realized by processing of the CPU 501, and realizes the same functions as the display control unit 34a. The determining unit 75 is mainly realized by the processing of the CPU 501, and realizes the same function as the determining unit 35a. The creation unit 76 is mainly realized by the processing of the CPU 501, and realizes the same functions as the creation unit 36a. The calculation unit 77 is mainly realized by the processing of the CPU 501, and realizes the same function as the calculation unit 37a. The communication unit 78 is mainly realized by the processing of the CPU 501, and realizes the same functions as the communication unit 38a. The storage/readout unit 79a is realized by the processing of the CPU 501, and stores various data (or information) in the storage unit 7000 and reads out various data (or information) from the storage unit 7000.

<Functional configuration of smartphone>
Next, the functional configuration of the smartphone 9 will be described in detail using FIGS. 13 and 14. The smartphone 9 basically has the same functions as the video conference terminal 3a. That is, as shown in FIG. 14, the smartphone 9 includes a transmitting/receiving section 91, a receiving section 92, an image/sound processing section 93, a display control section 94, a determining section 95, a creating section 96, a calculating section 97, and a communication section. 98 and a storage/readout section 99. Each of these parts is a function or means that is realized when any of the components shown in FIG. be.

Furthermore, the smartphone 9 has a storage unit 9000 constructed from a ROM 902, a RAM 903, and an EEPROM 904 shown in FIG. This storage unit 9000 is constructed with an image type management DB 9001, a photographing device management DB 9002, a predetermined area management DB 9003, and a display hierarchy management DB 9004. The image type management DB 9001, imaging device management DB 9002, predetermined area management DB 9003, and display hierarchy management DB 9004 have the same data structure as the image type management DB 3001a, imaging device management DB 3002a, predetermined area management DB 3003a, and display hierarchy management DB 3004a, respectively. Therefore, the explanation of these will be omitted.

(Each function configuration of smartphone)
The transmitting/receiving unit 91 of the smartphone 9 is mainly realized by the processing of the long distance communication circuit 911 and the CPU 901 shown in FIG. 13, and realizes the same functions as the transmitting/receiving unit 31a.

The reception unit 92 is mainly realized by the processing of the touch panel 921 and the CPU 901, and realizes the same functions as the reception unit 32a.

The image/sound processing section 93 is mainly realized by commands from the CPU 901, and realizes the same functions as the image/sound processing section 33a. The display control unit 94 is mainly realized by the processing of the CPU 901, and realizes the same functions as the display control unit 34a. The determining unit 95 is mainly realized by the processing of the CPU 901, and realizes the same function as the determining unit 35a. The creation unit 96 is mainly realized by the processing of the CPU 901, and realizes the same functions as the creation unit 36a. The calculation unit 97 is mainly realized by the processing of the CPU 901, and realizes the same function as the calculation unit 37a. The communication unit 98 is mainly realized by the processing of the CPU 901, and realizes the same functions as the communication unit 38a. The storage/readout unit 99 is realized by the processing of the CPU 901 and stores various data (or information) in the storage unit 9000 and reads various data (or information) from the storage unit 9000.

<<Processing or operation of embodiment>>
Next, the processing or operation of this embodiment will be described using FIGS. 24 to 35.

<Participation processing>
First, the process of participating in a specific communication session will be described using FIGS. 24 and 25. FIG. 24 is a sequence diagram showing a process for participating in a specific communication session. FIG. 25 is a diagram showing a communication session (virtual conference room) selection screen.

First, when a user at base A (for example, user A1) performs an operation to display a selection screen for a communication session (virtual conference room) on the video conference terminal 3a, the reception unit 32a displays the selection screen. Upon receiving the operation, the display control unit 34a displays a selection screen as shown in FIG. 25 on the display 4a (step S21). This selection screen displays selection buttons b1, b2, b3, etc. indicating virtual conference rooms R1, R2, R3, etc. to be selected. Further, each session ID is associated with each selection button b1 and the like.

Here, when the user A1 selects a desired selection button (here, selection button b1) for the virtual conference room, the reception unit 32a accepts the selection of a communication session (step S22). Then, the transmitter/receiver 31a transmits a request to participate in the virtual conference room to the communication management system 5 (step S23). This participation request includes a session ID indicating the communication session whose selection was accepted in step S22, and the IP address of the video conference terminal 3a that is the requesting terminal. Thereby, the transmitting/receiving unit 51 of the communication management system 5 receives the participation request.

Next, in the session management DB 5001 (see FIG. 21), the storage/reading unit 59 enters the IP address received in step S23 in the participating terminal IP address field in the record with the same session ID as the session ID received in step S23. By adding the address, participation in the communication session is performed (step S24). Then, the transmitting/receiving unit 51 transmits a participation request response to the video conference terminal 3a (step S25). This participation request response includes the session ID received in step S23 and the participation processing result. Thereby, the transmitting/receiving unit 31a of the video conference terminal 3a receives the participation request response. Hereinafter, a case where the participation process is successful will be explained.

<Management processing of image type information>
Next, image type information management processing will be described using FIG. 26. FIG. 26 is a sequence diagram showing image type information management processing.

First, when a user at base A (for example, user A1) connects the USB cable of the cradle 2a with the photographing device 1a and the PC 6 attached to the video conference terminal 3a, the storage/reading section of the photographing device 1a is connected to the video conference terminal 3a. 19a reads the GUID of its own device (photographing device 1a) stored in the storage section 1000a, and the communication section 18a transmits the GUID of its own device to the communication section 38a of the video conference terminal 3a (step S51). Thereby, the communication unit 38a of the video conference terminal 3a receives the GUID of the photographing device 1a.

Then, the storage/reading section 69 of the PC 6 reads out the PC information of the own device (PC6) stored in the storage section 6000, and the communication section 68 sends the PC information of the own device to the communication section 38a of the video conference terminal 3a. Transmit (step S52). Thereby, the communication unit 38a of the video conference terminal 3a receives the PC information of the PC 6. The PC information may be any information that allows it to be determined that the PC 6 is the PC that generates the material image. For example, information that can be determined by the video conference terminal 3a may be set in advance, such as product information of the PC 6 itself or product information of an application by which the PC 6 generates document images.

Further, as described in FIG. 11, when the image sensor 1a is attached, the image sensor 1a is used preferentially, but the camera 312 of the video conference terminal 3a may also be used.

Next, the determination unit 35a of the video conference terminal 3a determines the image type (step S53). First, a method for determining the image type of the photographing device 1a will be explained. In the imaging device management DB 3002a (see FIG. 17), the image type is determined by determining whether the same vendor ID and product ID as the vendor ID and product ID in the GUID received in step S51 are managed. (Step S53). Specifically, if the same vendor ID and product ID are managed in the imaging device management DB 3002a, the determination unit 35a determines whether the imaging device 1a captures a special image (here, a spherical panoramic image). It is determined that it is a photographic device. On the other hand, if the same vendor ID and product ID are not managed in the imaging device management DB 3002a, the determining unit 35a determines that the imaging device 1a is an imaging device that takes general images. Regarding the camera 312 of the video conference terminal 3a, since the same vendor ID and product ID are not managed in the photographing device management DB 3002a, the determination unit 35a of the video conference terminal 3a determines that the camera 312 is a photographing device that normally photographs images. I judge that.

Next, a method for determining the type of image on the PC 6 will be explained. The determining unit 35a of the video conference terminal 3a determines the image type by determining whether the PC information received in step S52 is information set in advance to determine the terminal that transmits the material image. If the information is preset, it is determined that the PC 6 is a device that generates document images.

Next, the storage/reading unit 39a sends the image type management DB 3001a (see FIG. 16) the IP address of its own terminal (video conference terminal 3a), which is the source terminal, and the determination result determined in step S53. It is stored in association with certain image type information (step S54). In this state, the image data ID is not associated. The image type information is, for example, a source name determined according to a predetermined naming rule or an image type (general image, special image, document image).

Next, the transmitting/receiving unit 31a transmits a request to add image type information to the communication management system 5 (step S55). This image type information addition request includes the IP address of the own terminal, which is the transmission source terminal, and the image type information stored in step S55. In this embodiment, as an example, the IP address of the own terminal, which is the transmission source terminal, and the image type information include three types: a general image, a special image, and a document image. Thereby, the transmitting/receiving unit 51 of the communication management system 5 receives the request for adding image type information.

Next, the storage/reading unit 59 of the communication management system 5 searches the session management DB 5001 (see FIG. 21) using the IP address of the source terminal received in step S54 as a search key, and retrieves the corresponding session ID. is read out (step S56).

Next, the generation unit 56 generates a unique image data ID (step S57). In this embodiment, this image data ID is generated for each of the general image, special image, and material image.

Then, the storage/reading unit 59 stores the session ID read in step S56, the image data ID generated in step S57, and the image data ID generated in step S54 as new records in the image type management DB 5002 (see FIG. 22). The received IP address of the source terminal and image type information are stored in association with each other (step S58). In this embodiment, as an example, a combination of this image data ID, the IP address of the sending terminal received in step S55, and the image type information is stored for each of the general image, special image, and material image.

Then, the transmitting/receiving unit 51 transmits the image data ID generated in step S56 to the video conference terminal 3a. Thereby, the transmitting/receiving unit 31a of the video conference terminal 3a receives the image data ID (step S59).

Next, the storage/reading unit 39a of the video conference terminal 3a stores the IP address of the own terminal (video conference terminal 3a), which is the transmission source terminal, stored in the image type management DB 3001a (see FIG. 16) in step S53 above. The image data ID received in step S58 is stored in association with the address and image type information (step S60).

On the other hand, the transmitting/receiving unit 51 of the communication management system 5 transmits an additional notification of image type information to another communication terminal (here, the video conference terminal 3d) (step S61). This notification of addition of image type information includes the image data ID generated in step S56, the IP address of the own terminal (video conference terminal 3a) that is the source terminal stored in step S55, and image type information. It is. Thereby, the transmitting/receiving unit 31d of the video conference terminal 3d receives the notification of addition of image type information. Note that the transmission destination of the transmitter/receiver 51 is another IP address associated with the same session ID as the IP address of the video conference terminal 3a in the session management DB 5001 (see FIG. 21). That is, the destination is another communication terminal in the same virtual conference room as the video conference terminal 3a.

Next, the storage/reading unit 39d of the video conference terminal 3d stores the image data ID, the IP address of the source terminal, and the information received in step S60 as a new record in the image type management DB 3001d (see FIG. 16). and image type information are stored in association with each other (step S62). Similarly, additional communication of image type information is also sent to the other communication terminals PC7 and smartphone 9, and is stored in the image type management DBs 7001 and 9001 in the video conference terminal PC7 and smartphone 9, respectively. As described above, each communication terminal can share the same information in each image type management DB 3001a, 3001d, 7001, and 9001.

<Communication processing of photographed image data>
Next, communication processing of photographed image data and material image data in a video call will be explained using FIGS. 27 to 35. FIG. 27 is an image diagram showing the state of a video call using base A as an example.

As shown in FIG. 27, the video conference terminal 3a synthesizes the image taken by the camera 312 and the document image generated by the PC 6 onto the spherical panoramic image taken by the photographing device 1a, and displays the composite image. 4. Note that a whiteboard W is installed on the right side of FIG. 27 on which the user A1 and others can write letters, pictures, and the like.

Next, using FIG. 28, the captured image data, sound data, and material image data obtained at base A shown in FIG. The process of transmitting to the PC 7 and video conference terminal 3d) will be explained. FIG. 28 is a sequence diagram showing communication processing of captured image data and material image data in a video call.

First, the communication unit 18a of the photographing device 1a transmits photographed image data obtained by photographing a subject, scenery, etc. and sound data obtained by collecting sounds to the communication unit 38a of the video conference terminal 3a (step S101 ). In this case, since the photographing device 1a is a device that can obtain two hemisphere images that are the basis of a spherical panoramic image, the photographed image data is is composed of data of two hemisphere images. Thereby, the communication unit 38a of the video conference terminal 3a receives the photographed image data and the sound data.

Next, the communication unit 68 of the PC 6 transmits the document image data displayed by the display control unit 64 to the communication unit 38a of the video conference terminal 3a (step S102).

Next, the transmitting/receiving unit 31a of the video conference terminal 3a transmits to the communication management system 5 the photographed image data and sound data sent from the photographing device 1a, the material image data sent from the PC 6, and the video conference terminal. 3a transmits the photographed image data photographed by the camera 312 (step S103). Thereby, the transmitting/receiving unit 51 of the communication management system 5 receives the captured image data, sound data, and material image data. Here, the captured image data includes an image data ID for identifying the captured image data to be transmitted.

Next, the transmitting/receiving unit 51 of the communication management system 5 sends photographed image data, sound data, and Material image data is transmitted (steps S104, S105, S106). Regarding the transmission of captured image data in each of these transmissions, an image data ID for identifying the captured image data to be transmitted is included. As a result, the transmitting/receiving unit 91 of the smartphone 9, the transmitting/receiving unit 71 of the PC 7, and the transmitting/receiving unit 31d of the video conference terminal 3d receive the image data ID associated with the captured image data and the image data ID associated with the document image data, respectively. and also receives sound data.

Next, a display example of the display 917 at base B will be described using FIG. 29. FIG. 29 is an example of a display at base B. Of these, FIG. 29(a) is based on the photographed image data sent from the photographing device 1a at base A via the video conference terminal 3a and the photographed image data sent from the photographing device 1b at base B. , a case where the image is displayed as it is without creating a spherical panoramic image or creating a predetermined area image. On the other hand, FIG. 29(b) shows a case where a spherical panoramic image and a predetermined area image are created from the photographed image data sent from the photographing devices 1a and 1b. Note that the image of base A is displayed in the left display area (layout number "1") of the display 917, and the image of base B (own base) is displayed in the upper right display area (layout number "2"). has been done. Furthermore, the image of base C is displayed in the middle right display area (layout number "3") of the display 917, and the image of base D is displayed in the lower right display area (layout number "4"). . The display area with layout number "1" is the main display area, and the display areas with layout numbers "2", "3", and "4" are sub display areas. The image in the main display area and the image in the sub display area can be changed at each communication terminal. Normally, at each base, an image of the base where the person at the center of the video call is located is displayed in the main display area.

Here, if the photographed image data sent from the photographing devices 1a and 1b, which are capable of photographing spherical panoramic images, are displayed as they are, as shown in FIG. 29(a), base A and base B are displayed. images are displayed as a front hemisphere image and a rear hemisphere image, as shown in FIGS. 3(a) and 3(b), respectively.

On the other hand, the image/sound processing unit 93 generates a spherical panoramic image from the photographed image data output by the photographing devices 1a and 1b, which can obtain two hemispherical images that are the basis of the spherical panoramic image. When the predetermined area image is created and the predetermined area image is further created, the predetermined area image, which is a planar image, is displayed as shown in FIG. 29(b). Note that at base C, the photographing device 8 that obtains general images is photographing, and at base D, the video conference terminal 3d that obtains general images is also photographing, so both FIGS. 29(a) and 29(b) are An image (here, a flat image) is displayed.

Further, the user at each base can change the predetermined area related to the predetermined area image in the same spherical panoramic image. For example, when the user B1 operates the touch panel 921, the reception unit 92 accepts movement of the predetermined area image, and the display control unit 94 shifts, rotates, reduces, or enlarges the predetermined area image. can do. As a result, as shown in FIG. 29(b), the predetermined area image in which some users A1 and A2 of base A are displayed in the initial setting (default) is changed to the image shown in FIG. 29(c). The predetermined area image can be displayed as shown in FIG. Specifically, FIG. 29(c) shows a state in which the image taken at base A shown in FIG. 27 is changed from a predetermined area image including users A1 and A2 to a predetermined area image including whiteboard W. It shows.

Note that the spherical icons 191 and 192 shown in FIGS. 29(b) and 29(c) are special icons used to indicate that these are predetermined area images that indicate a predetermined area T that is part of a spherical panoramic image. This is an example of an image identification icon. Note that the display positions of the celestial sphere icons 191 and 192 may be anywhere, such as the upper left, lower left, or lower right, instead of the upper right. Furthermore, the types of omnidirectional sphere icons 191 and 192 are not limited to those shown in FIGS. 29(b) and 29(c). In addition, instead of the omnidirectional icons 191 and 192, it may be text such as "encyclopedia image" or a combination of an icon and text.

Next, using FIG. 30, image communication when displaying the predetermined area image as shown in FIG. 29(b) and when changing the predetermined area image as shown in FIG. 29(b) to FIG. 29(c). Processing in the system will be explained. FIG. 30 is a sequence diagram showing a process of sharing predetermined area information. In FIG. 30, the video conference terminal 3a at base A is the third communication terminal, the video conference terminal 3d at base D is another communication terminal, and the smartphone 9 at base B is the communication terminal (own terminal).

First, as shown in FIG. 29(b), when a user D1 or the like at base D uses the video conference terminal 3d to display a predetermined area image of base A, the transmitter/receiver of the video conference terminal 3d 31d transmits predetermined area information indicating the displayed predetermined area image to the communication management system 5 (step S111). This predetermined area information includes the IP address of the video conference terminal 3a that is the source of the captured image data, and the IP address of the video conference terminal 3d that is the destination of the captured image data (the source of the predetermined area information). ing. Thereby, the transmitting/receiving unit 51 of the communication management system 5 receives the predetermined area information.

Next, the storage/reading unit 59 of the communication management system 5 stores the predetermined area information received in step S111 and the IP addresses of the transmission source and destination in association with each other in the predetermined area management DB 5003 (step S112). Note that the processes in steps S111 and 112 are performed every time the predetermined area image is changed on the video conference terminal 3d, as shown in FIGS. 29(b) to 29(c).

Next, the storage/reading unit 59 of the communication management system 5 selects the latest set (the latest one) of the sets of predetermined area information and each IP address stored in the predetermined area management DB 5003. A set of predetermined area information and each IP address is read out at regular intervals (for example, 30 seconds) (step S113). Then, in step S113, the transmitting/receiving unit 51 reads information from other communication terminals (video conference terminal 3a, smartphone 9, PC 7) that are conducting the same video call as the video conference terminal 3d that is the transmission source of the predetermined area information. The predetermined area information including each of the issued IP addresses is distributed (transmitted) (steps S114, S116, S118). Thereby, the transmitting/receiving unit 31a of the video conference terminal 3a receives the predetermined area information. Then, the storage/reading unit 39a stores the predetermined area information and each IP address received in step S114 in association with each other in the predetermined area management DB 3003a (step S115). Similarly, in the smartphone 9, after the transmitter/receiver 91 receives the predetermined area information, the storage/reader 99 stores it in the predetermined area management DB 9003 in association with the predetermined area information and each IP address received in step S116. (Step S117). Furthermore, in the PC 7 as well, after the transmitter/receiver 71 receives the predetermined area information, the storage/reader 79 associates the predetermined area information received in step S118 with each IP address similarly received in the predetermined area management DB 7003. It is stored (step S119).

Next, other processing for sharing predetermined area information will be described using FIG. 31. FIG. 31 is a sequence diagram showing another example of the process shown in FIG. 30 and showing another process of sharing predetermined area information. In FIG. 31, the video conference terminal 3a at base A is the communication terminal (own terminal), and the video conference terminal 3d at base D is another communication terminal.

In the process shown in FIG. 30 above, the communication management system 5 once manages each predetermined area information transmitted from each communication terminal (see S112), and periodically sends each communication terminal other than the transmission source to each communication terminal. The predetermined area information has been transmitted (see S114 to S119). On the other hand, in the present embodiment, as shown in FIG. The information is once managed (see S213), and each predetermined area information is transmitted to each communication terminal other than the own terminal at regular intervals (see S215 to S221). That is, in the present embodiment, the communication terminal that is the transmission source of the photographed image data determines whether the photographed image data transmitted by the own terminal (in this case, the video conference terminal 3a) is received by another communication terminal in a predetermined area of the predetermined area T1. It manages whether it is displayed as an image.

Note that the processing shown in FIG. 31 is realized by the same hardware configuration as the processing shown in FIG. 30. Therefore, a description of the hardware configuration for realizing the processing in FIG. 31 will be omitted.

In the process shown in FIG. 31, first, when a user D1 or the like uses the video conference terminal 3d at base D to display an image of a predetermined area of base A, the transmitter/receiver 31d of the video conference terminal 3d performs communication management. Predetermined area information indicating the displayed predetermined area image is transmitted to the system 5 (step S211). This predetermined area information includes the IP address of the video conference terminal 3a that is the source of the captured image data, and the IP address of the video conference terminal 3d that is the destination of the captured image data (the source of the predetermined area information). ing. Thereby, the transmitting/receiving unit 51 of the communication management system 5 receives the predetermined area information.

Next, the transmitting/receiving unit 51 of the communication management system 5 transmits the predetermined area information including each IP address received in step S211 to the video conference terminal 3a that is the source of the captured image data (step S212). Thereby, the transmitting/receiving unit 31a of the video conference terminal 3a receives the predetermined area information.

Next, the storage/reading unit 39a of the video conference terminal 3a stores the predetermined area information received in step S212 and the IP addresses of the source and destination in association with each other in the predetermined area management DB 3003a (step S213). The process in step S213 is a process for managing how the captured image data transmitted by the own terminal (here, the video conference terminal 3a) is displayed on the other communication terminal. Note that the processing in steps S211 to S213 is performed every time the predetermined area image is changed on the video conference terminal 3d.

Next, the storage/reading unit 39a of the video conference terminal 3a selects the latest set (the latest one) out of the sets of predetermined area information and each IP address stored in the predetermined area management DB 3003a. The predetermined area information and each IP address are read out at regular intervals (for example, 30 seconds) (step S214). Then, the transmitting/receiving unit 31a transmits predetermined area information including each IP address read in step S214 to the communication management system 5 (step S215). Thereby, the transmitting/receiving unit 51 of the communication management system 5 receives the predetermined area information.

Next, the transmitting/receiving unit 51 of the communication management system 5 transmits (distributes) the predetermined area information including each IP address received in step S215 to each communication terminal (video conference terminal 3d, smartphone 9, PC 7) (step S216, S218, S220). Thereby, the transmitting/receiving unit 31d of the video conference terminal 3d receives the predetermined area information. Then, the storage/reading unit 39d stores the predetermined area information received in step S216 in the predetermined area management DB 3003d in association with each of the similarly received IP addresses (step S217). Similarly, in the smartphone 9, after the transmitting/receiving unit 91 receives the predetermined area information, the storage/reading unit 99 stores the predetermined area information received in step S218 in the predetermined area management DB 9003, and stores the predetermined area information received in step S218 in each IP address also received. It is stored in association (step S219). Furthermore, in the PC 7 as well, after the transmitter/receiver 71 receives the predetermined area information, the storage/reader 79 associates the predetermined area information received in step S220 with each IP address similarly received in the predetermined area management DB 7003. It is stored (step S221).

As described above, the predetermined area information indicating the predetermined area image changed at base A is transmitted to each communication terminal of other bases B, C, and D during the same video call. The predetermined area information indicating the image is also shared among other bases B, C, and D. This process is performed in the same way when the predetermined area image is changed at bases B, C, and D, and the predetermined area information indicating the predetermined area image being displayed at other bases is common among all bases during the same video call. be done.

<Changing the display hierarchy>
Next, display hierarchy settings for document images will be explained using FIGS. 32 to 34. FIG. 32 is a sequence diagram illustrating processing in the image communication system when changing the display layer in which document images are combined into a spherical panoramic image at base A, which is an image distribution source.

First, the storage/reading unit 39a of the video conference terminal 3a reads out the image data ID associated with the IP address of the own terminal stored in the image type information DB 5002 (step S71). The creation unit 36a then sets the initial value of the display hierarchy of each image data ID associated with each read image data ID (step S72). As an example of how to set the initial value, set the display hierarchy of the first image data ID stored in the image type information DB 5002 to "base", and then set the display hierarchy as "1", "2" and so on. I will do it. Note that this display hierarchy is set to be unique for each sending terminal IP. Furthermore, initial values of display coordinates may be set. As an example, the display hierarchy "base" is the full screen size, the image size is fixed from the display hierarchy "1" onwards, and the display hierarchy "1" is the left corner of the whole screen, and from then on, the image size is set to the previous one in the order of the display hierarchy. The image may be set as appropriate so that it is displayed to the right of the image in the display hierarchy.

The display control unit 34a then displays the composite image according to the initial value set in step S72 (step S73). If display coordinates are not set, the display position may be arbitrary and the composite image may be displayed according to the display hierarchy.

Next, changing the display hierarchy will be explained. The receiving unit 32a receives the selection of a document image in response to an operation by the user A1 or the like of the base A (step S74). As an example of how to select the selected state, on the preview screen of the spherical panoramic image distributed by user A1 of base A, etc., click "right click", "double click", "press the corresponding key" on the document image part. There are ways to perform operations such as "input". In the case of a touch-enabled terminal, the normal image in the preview area may be realized by operations such as "long tap" or "double tap". Next, the receiving unit 32a receives a change in the display hierarchy of the document images in response to an operation by the user A1 or the like (step S75).

Next, the accepting unit 32a accepts the determination of the display hierarchy of the material image according to the operation of the user A1, etc., and the calculating unit 37a generates display hierarchy information (step S76). As an example of a method for determining, the display hierarchy may be determined automatically after the operation in step S75 is completed, or by pressing a separate determination button on the screen, or by performing a predetermined key operation. It is also possible to do this.

Here, an example of a screen on which the user A or the like changes the display hierarchy of document images will be described with reference to FIGS. 33 and 34. FIG. 33 is a preview screen at base A of the spherical panoramic image. The preview screen here is a screen for checking the video being distributed by a certain base at the own base without going through the communication network 100. As shown in FIG. 33, on the preview screen, a predetermined area image including users A2, A4, and the whiteboard W out of the spherical panoramic image is displayed as a base image in front of the preview screen (area C0). Letters and pie charts are written on the whiteboard W. Further, in FIG. 33, in area C1 and area C2, a document image generated by the PC 6 and a general image taken by the camera 312 of the video conference terminal 3a are displayed, display hierarchy information and display position information associated with the image data ID. Displayed based on.

In other words, in FIG. 33, at that point, the spherical panoramic image follows the display hierarchy ("base") associated with the image data ID of the spherical panoramic image, and also follows the display coordinates associated with the image data ID. It is displayed in the corresponding area C0. The material image is displayed in the area C1 corresponding to the display coordinates similarly associated with the image data ID, according to the display hierarchy associated with the image data ID of the material image, eg "1". Furthermore, according to the display hierarchy associated with the image data ID of the general image, eg, "2", it is displayed in the area C2 corresponding to the display coordinates similarly associated with the image data ID.

With reference to FIG. 34, operations by the user A1 and the like will be explained in correspondence with steps S74 to S76 described above. First, when the user A1 etc. selects the document image displayed in the area C1 (step S74), a pop-up P is displayed, and a message saying "use this image as a base" and "change the coordinates of this image" is displayed. be done. Among the messages, "use this image as a base" is selected (step S75). In other words, instead of the omnidirectional panoramic image displayed in area C0 at this point, the document image currently displayed in area C1 is changed to become the base image, that is, the display hierarchy "base." The message is not limited to this, and may be displayed with appropriate settings such as "move to the front" or "move to the back". Then, an operation for determining the changed display hierarchy is executed (step S76).

Continuing the description of FIG. 32, the storage/reading unit 39a of the video conference terminal 3a stores the determined display hierarchy in the display hierarchy management DB 3004d (step S77). Next, the communication unit 38a transmits the source IP address and the changed display hierarchy information to the transmission/reception unit 51 of the communication system 5 (step S78). This display hierarchy information includes the display hierarchy of each image data ID after the change. Thereby, the transmitting/receiving unit 51 of the communication management system 5 receives the display hierarchy information.

Next, the transmitting/receiving unit 51 of the communication management system 5 communicates with other communication terminals (video conference terminal 3d, smartphone 9, PC 7) that are conducting the same video call with the video conference terminal 3a that is the source of the display layer information. Then, the source IP address and display layer information are distributed (transmitted) (steps S79, S81, S83). Thereby, the transmitting/receiving unit 31d of the video conference terminal 3d receives the source IP address and display layer information. Then, the storage/reading unit 39a stores the display hierarchy information received in step S79 in association with the image data ID also received in the display hierarchy management DB 3004d (step S80). Similarly, in the smartphone 9, after the transmitter/receiver 91 receives the source IP address and the display hierarchy information, the storage/reader 99 sends the display hierarchy information received in step S81 to the display hierarchy management DB 9004. The image data ID is stored in association with the image data ID (step S82). Further, in the PC 7 as well, after the transmitter/receiver 71 receives the source IP address and the display layer information, the storage/reader 79 stores the display layer information received in step S83 in the display layer management DB 7004, and stores the display layer information received in step S83 as well as the received image. It is stored in association with the data ID (step S84).

Note that steps S71 to S73 are initial settings that are executed when no association is set in the display hierarchy management table. Therefore, it only needs to be executed when displaying the composite image for the first time at the start of a video conference, and if the display hierarchy after step S73 is changed more than once, it can be changed without executing steps S71 to S72 from the second time onwards. .
<Change display position>
Next, the display position setting of the material image will be explained using 34, FIGS. 35 and 36. Here, FIG. 35 is a sequence diagram showing the processing in the image communication system when changing the display position at which the document image is combined into the spherical panoramic image at base A, which is the image distribution source.

In FIG. 35, first, the storage/reading unit 39a of the video conference terminal 3a reads out the image data ID associated with the IP address of the own terminal stored in the image type information DB 5002 (step S271). The creation unit 36a then sets the initial value of the display hierarchy of the image data associated with each read image data ID (step S272). As an example of how to set the initial value, set the display hierarchy of the first image data ID stored in the image type information DB 5002 to "base", and then set the display hierarchy as "1", "2" and so on. I will do it. Note that this display hierarchy is set to be unique for each sending terminal IP. Furthermore, initial values of display coordinates may be set. As an example, the display hierarchy "base" is the full screen size, the image size is fixed from the display hierarchy "1" onwards, and the display hierarchy "1" is the left corner of the whole screen, and from then on, the image size is set to the previous one in the order of the display hierarchy. Set it to be displayed to the right of the image in the display hierarchy.

The display control unit 34a then displays the composite image according to the initial values set in step S272 (step S273). If display coordinates are not set, the display position may be arbitrary and the composite image may be displayed according to the display hierarchy.

Next, changing the display position will be explained. The receiving unit 32a receives the selection of a document image in response to an operation by the user A1 or the like of the base A (step S274). As an example of how to select the selected state, on the preview screen of the spherical panoramic image distributed by user A1 of base A, etc., click "right click", "double click", "press the corresponding key" on the document image part. There are ways to perform operations such as "input". In the case of a touch-enabled terminal, the normal image in the preview area may be realized by operations such as "long tap" or "double tap". Next, the receiving unit 32a receives a change in the display position of the document image in response to an operation by the user A1 or the like (step S275).

Next, the receiving unit 32a receives the determination of the display position of the document image in accordance with the operation of the user A1, etc., and the calculating unit 37a generates display position information (step S276). As an example of a method for determining the display position, the display position may be determined automatically after the operation in step S275 is completed, or by pressing a separate determination button on the screen, or by performing a predetermined key operation. It is also possible to do this.

Here, using FIGS. 34 and 36, the operations of the user A1 etc. will be explained in correspondence with steps S274 to S276 described above. In FIG. 34, when the user A1 or the like selects the document image displayed in the area C1 (step S274), a pop-up P is displayed with instructions to ``use this image as a base'' and ``change the coordinates of this image.'' A message is displayed. Among the messages, "Change the coordinates of this image" is selected (step S275). Then, as an example, as shown in FIG. 5, by dragging and dropping the document image in area C1 to area C3, which is the changed display position, an operation for determining the changed display position is executed (step S276).

Continuing the description of FIG. 35, the storage/reading unit 39a of the video conference terminal 3a stores the determined display position information in the display hierarchy management DB 3004d (step S277). Next, the communication unit 38a transmits the changed display position information to the transmission/reception unit 51 of the communication system 5 (step S278). This display position information includes display coordinates for each changed image data ID. Thereby, the transmitting/receiving unit 51 of the communication management system 5 receives the display position information.

Next, the transmitting/receiving unit 51 of the communication management system 5 communicates with other communication terminals (video conference terminal 3d, smartphone 9, PC 7) that are conducting the same video call with the video conference terminal 3a that is the source of the display layer information. Then, the source IP address and display position information are distributed (transmitted) (steps S279, S281, S283). Thereby, the transmitting/receiving unit 31d of the video conference terminal 3d receives the source IP address and display position information. Then, the storage/reading unit 39a stores the display position information received in step S279 in association with the image data ID also received in the display hierarchy management DB 3004d (step S280). Similarly, in the smartphone 9, after the transmitting/receiving unit 91 receives the source IP address and the display position information, the storage/read unit 99 sends the display position information received in step S281 to the display hierarchy management DB 9004. The image data ID is stored in association with the image data ID (step S282). Furthermore, in the PC 7 as well, after the transmitting/receiving unit 71 receives the source IP address and the display position information, the storage/read unit 79 stores the display position information received in step S283 in the display hierarchy management DB 7004, as well as the received image. It is stored in association with the data ID (step S284).

Note that steps S271 to S273 are initial settings that are executed when no association is set in the display hierarchy management table. Therefore, it is only necessary to execute this when displaying a composite image for the first time at the start of a video conference, and if the display hierarchy after step S273 is to be changed more than once, it can be changed without executing steps S271 to S272.
<Display processing by receiving terminal>
FIG. 37 is a flowchart showing the process of displaying document images on the receiving terminal. Note that since the processing of each receiving terminal is the same, the processing of the smartphone 9 at base B will be described here.

First, in the smartphone 9, from the received source IP address and display layer information (image data ID, display layer), the creation unit 96 creates a composite list on the RAM 903 consisting of image data IDs with the same source IP address. (Step S131). Note that the source IP address and display layer information (image data ID, display layer) are received from the conference terminal 3a, as described in step S81 shown in FIG. It is received sequentially from each communication terminal participating in the same video conference.

Next, the storage/reading unit 99a extracts the image data ID associated with the display hierarchy "base" from the display hierarchy management DB 9004 from among the image data IDs of the created composite list (step S132). The storage/reading unit 99a acquires the image type of the image data ID extracted in step S132 from the image type management DB 9001 (step S133). Then, the display control unit 94 displays the image of the image data ID associated with "base" as a base image on the preview screen in a method according to the image type (step S134). For example, when the image type is "Video_Theta", that is, a spherical image, the method according to the image type is displayed using the above-mentioned method for generating a spherical panoramic image.

Next, the storage/reading unit 99a extracts image data IDs with a smaller display hierarchy from the display hierarchy management DB 9004 among the image data IDs that are not yet displayed in the created combination list (step S135). The storage/reading unit 99a acquires the image type of the image data ID extracted in step S135 from the image type management DB 9001 (step S136). Then, the display control unit 94 displays the image of the extracted image data ID superimposed on the base image using a method according to the image type (step S137). At this time, in the area where the display positions overlap with each other, the image having the largest display hierarchy among the superimposed images is displayed.Then, the judgment unit 95 displays all the image data IDs in the image data ID list. (step S138), and steps S135 to S137 are repeated until the display of all image data IDs is completed. When the display of all the image data IDs is completed, the determining unit 95 determines whether the processing of steps S132 to 138 has been completed for all the source IP addresses (step S139), and the determination unit 95 determines whether the processing of steps S132 to S138 has been completed for all the source IP addresses (step S139). Steps S132 to S138 are repeated until completion. When processing is completed for all source IP addresses, this flow ends.

FIG. 38 is an example of the display screen of base B after executing the display hierarchy change process described in FIG. 34. As shown in FIG. 38, the composite image is displayed at base B in the display hierarchy changed at base A as described in FIG. 34. The material image that was displayed in area C1 before the display hierarchy change (FIG. 34) has been changed to the base image, so it is placed at the backmost position and is not displayed.

As a result, the display hierarchy information changed at base A is sent to each communication terminal at other bases B, C, and D during the same video call, so that the display hierarchy information changed at base A is displayed in the same display hierarchy as the document image being displayed at base A. It is also displayed and shared at other bases B, C, and D.

FIG. 39 is an example of a display screen of base B that has executed the display position change process described in FIG. 36. As shown in FIG. 39, the composite image is displayed at base B at the display position changed at base A as explained in FIG. 36. The document image that was displayed in area C1 before the display position change (FIG. 34) is now displayed in area C3.

As described above, the display position information changed at base A is sent to each communication terminal at other bases B, C, and D during the same video call, so that the display position information changed at base A is displayed at the same display position as the document image being displayed at base A. It is also displayed and shared at other bases B, C, and D.

<<Main effects of this embodiment>>
As explained above, according to the present embodiment, the communication terminal such as the video conference terminal 3a creates a spherical panoramic image based on the image data ID sent with the image data and the corresponding image type information. , it is also possible to create a predetermined area image. Thereby, as shown in FIG. 27(a), it is possible to prevent the front hemisphere image and the rear hemisphere image from being displayed.

Further, according to the present embodiment, in a conference system or the like in which a plurality of images received by a receiver are displayed, the plurality of images may be displayed in a front-back relationship that is not intended by the sender. , If you want to change the front/back relationship of multiple images during a meeting, you can change the display hierarchy. This allows you to change the display hierarchy and display images in cases where multiple images are displayed in a front/back relationship that was not intended by the sender, or if you want to change the front/back relationship of multiple images. This has the effect of making images easier to see and understand.

〔supplement〕
In the above embodiment, as an example of the panoramic image to be combined, a three-dimensional spherical panoramic image has been described as a photographed image (the entire image), but it may be a two-dimensional panoramic image.

Further, in the above embodiment, the communication management system 5 relays the predetermined area information transmitted from each communication terminal, but the present invention is not limited to this. good.

Furthermore, each function of the above embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" as used herein refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or a processor designed to execute each function explained above. It includes devices such as ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array), SOC (System on a chip), GPU (Graphics Processing Unit), and conventional circuit modules.

1a Photographing device 1b Photographing device 3a Video conference terminal (an example of a communication terminal)
3d video conference terminal (an example of other communication terminal)
4a Display 4d Display (an example of display means)
5 Communication management system 6 PC
7 PC (an example of other communication terminal)
8 Photographing device 9 Smartphone (an example of other communication terminal)
31d Transmitting/receiving unit (an example of a transmitting means, an example of a receiving means)
37d Calculation unit (an example of changing means)
3004a Display hierarchy management DB (an example of display hierarchy management means)
3004d Display hierarchy management DB (an example of display hierarchy management means)
7004 Display hierarchy management DB (an example of display hierarchy management means)
9004 Display hierarchy management DB (an example of display hierarchy management means)

Japanese Patent Application Publication No. 2012-178135 Japanese Patent Application Publication No. 2011-223076 Japanese Patent Application Publication No. 2012-156820

Claims (10)

a communication terminal that transmits multiple image data;
another communication terminal that displays images related to each of the plurality of image data transmitted from the communication terminal;
An image communication system comprising:
The communication terminal is
a reception unit that accepts a change in the display hierarchy of the plurality of image data;
Based on the change in the display hierarchy received by the reception unit, the display hierarchy information for displaying the image related to each of the plurality of image data in the display hierarchy representing the order in which images are combined with the base image is provided. It has a transmitter that transmits to other communication terminals,
The other communication terminal is
An image communication system comprising: a display control unit that displays the plurality of images in the predetermined display hierarchy based on display hierarchy information transmitted from the communication terminal. The image communication system according to claim 1,
The image communication system is characterized in that the other communication terminal has an image processing unit that performs image processing to display the plurality of images based on the display hierarchy information. The image communication system according to claim 1 or 2 ,
An image communication system characterized in that the plurality of image data includes captured image data and document image data. The image communication system according to claim 3 ,
An image communication system characterized in that the image related to the captured image data is a spherical panoramic image. A communication terminal that transmits multiple image data,
a reception unit that accepts a change in the display hierarchy of the plurality of image data;
Based on the change in the display hierarchy received by the reception unit, display hierarchy information for displaying images related to each of the plurality of image data in a display hierarchy representing an order in which images are combined with a base image ; a transmitting unit configured to transmit images related to each of the plurality of image data to another communication terminal displaying the images;
A communication terminal characterized by: The communication terminal according to claim 5 ,
A communication terminal characterized in that the plurality of image data includes photographed image data and document image data. The communication terminal according to claim 6 ,
A communication terminal characterized in that the image related to the photographed image data is a spherical panoramic image. A communication terminal according to any one of claims 5 to 7 ,
the other communication terminal;
An image communication system comprising: a communication management system that manages communication of image data between the communication terminal and the other communication terminal. The image communication system according to claim 8 ,
The other communication terminal is
receiving means for receiving the plurality of image data and the display layer information from the communication terminal;
image processing means for superimposing the plurality of images in a hierarchy indicated by the display hierarchy information;
An image communication system comprising: display control means for displaying an image in which the plurality of images are superimposed. A communication method executed by a communication terminal that transmits a plurality of image data,
The communication terminal is
a reception step of accepting a change in the display hierarchy of the plurality of image data;
Based on the change in the display hierarchy received in the receiving step, display hierarchy information for displaying images related to each of the plurality of image data in a display hierarchy representing the order in which images are combined with the base image; a transmitting step of transmitting images related to each of the plurality of image data to another communication terminal displaying the images;
A communication method characterized by performing processing including .