JP7210892B2 - Communication terminal, image communication system, display method, and program - Google Patents

Description

The present invention relates to communication terminals, image communication systems, display methods, and programs.

2. Description of the Related Art A conferencing system for holding a remote conference with a remote location via a communication network such as the Internet has become widespread. In this conference system, in a conference room where one of the parties, such as attendees, who conducts a remote conference, is present, the communication terminal of the remote conference system is used to shoot and collect images of the conference room and voices such as remarks of the parties of the conference. Then, they are converted into digital data and transmitted to the other party's communication terminal. As a result, images can be displayed on the display of the other party's conference room, and audio can be output from the speakers, allowing video calls to be made, allowing remote conferences to be held in a manner similar to that of an actual conference. (See Patent Document 1).

In addition, the communication terminal is connected to a photographing device capable of acquiring a omnidirectional panorama image in real time, the omnidirectional panoramic image obtained from this photographing device is transmitted to each communication terminal of the other party, and at each communication terminal of the other party , a technique for displaying, on a display or the like, a predetermined area image representing a predetermined area that is a part of the omnidirectional panorama image (see Patent Document 2). As a result, the user at each site can independently determine and display a predetermined area image showing a predetermined area of interest to the user in the entire omnidirectional panorama image.

In addition, the communication terminal on the transmitting side transmits the original image and the object image of interest to the server, and the communication terminal on the receiving side detects the coordinates of the object of interest in the current image. (see Patent Document 3).

However, when both omnidirectional (panorama) image data and superimposed image data are transmitted and received between multiple communication terminals via a communication network such as the Internet, the communication load increases and image data delays occur. There is a problem of stowing.

The invention according to claim 1 is a communication terminal capable of displaying a predetermined area image, which is an image of a predetermined area in a whole image, on a display means, wherein the data of the whole image and the superimposed image transmitted by another communication terminal are displayed. receiving means for receiving data and superimposed position information indicating a position where the superimposed image is to be superimposed on the overall image; image processing means for superimposing the superimposed image on the overall image based on the superimposed position information; and display control means for causing a display means to display a predetermined area image in the entire image, wherein the display control means is changed according to a ratio of the predetermined area image and the superimposed image within the display area of the display means. The communication terminal displays the predetermined area image and the superimposed image with the image quality of the predetermined area on the display means.

As described above, according to the present invention, it is possible to suppress the delay of the image data even when the data of the omnidirectional image and the superimposed image are transmitted and received.

(a) is a left side view of the imaging device, (b) is a front view of the imaging device, and (c) is a plan view of the imaging device. It is a use image figure of an imaging device. (a) is a hemispherical image (front) captured by the imaging device, (b) is a hemispherical image (back) captured by the imaging device, and (c) is a diagram showing an image represented by the Mercator projection. (a) A conceptual diagram showing a state in which a sphere is covered with a Mercator image, (b) a diagram showing an omnidirectional panorama image. FIG. 4 is a diagram showing the positions of a virtual camera and a predetermined area when the omnidirectional panorama image is a three-dimensional sphere; (a) is a three-dimensional perspective view of FIG. 5, and (b) is a diagram showing a state in which an image of a predetermined area is displayed on the display of the communication terminal. 4 is a diagram showing a relationship between predetermined area information and an image of a predetermined area T; FIG. Fig. 3 shows a point in three-dimensional Euclidean space with spherical coordinates; 1 is a schematic diagram of an image communication system according to an embodiment of the present invention; FIG. It is a hardware block diagram of an imaging device. 2 is a hardware configuration diagram of a video conference terminal; FIG. 2 is a hardware configuration diagram of a communication management system and a PC; FIG. It is a hardware block diagram of a smart phone. 1 is a functional block diagram of part of an image communication system; FIG. 1 is a functional block diagram of part of an image communication system; FIG. 4 is a conceptual diagram showing an image type management table; FIG. FIG. 3 is a conceptual diagram showing an imaging device management table; 4 is a conceptual diagram showing a predetermined area management table; FIG. 4 is a conceptual diagram showing a superimposition position management table; FIG. It is explanatory drawing of a superimposition position. 4 is a conceptual diagram showing a session management table; FIG. 4 is a conceptual diagram showing an image type management table; FIG. 4 is a conceptual diagram showing a predetermined area information management table; FIG. FIG. 4 is a conceptual diagram showing a ratio management table; FIG. 10 is a sequence diagram showing a process of joining a specific communication session; FIG. 11 is a diagram showing a screen for selecting a communication session (virtual conference room); FIG. 11 is a sequence diagram showing management processing of image type information; FIG. 4 is an image diagram showing a state of a video call; FIG. 4 is a sequence diagram showing communication processing of photographed image data, sound data, and material image data in a video call; Shown are display examples of the display at the site B, (a) is displayed as it is without creating an omnidirectional panorama image and a predetermined area image from the image data sent from the imaging devices 1a and 1b. FIG. 30(b) shows a case in which an omnidirectional panorama image and a predetermined area image are created from the image data sent from the imaging devices 1a and 1b, and FIG. 30(c). ) is a diagram showing a case where the predetermined area image of FIG. 30(b) is changed. FIG. 11 is a sequence diagram showing processing for sharing predetermined area information; FIG. 10 is a sequence diagram showing another process of sharing predetermined area information; FIG. 10 is a sequence diagram showing a process of sharing superimposition position information; It is an example of a screen for changing the superimposition position. 4 is a flowchart showing display processing of a superimposed image; A display example of a superimposed image (material image) at a site B is shown. FIG. 10 is a sequence diagram showing processing for changing image quality; FIG. 10 is a diagram when the predetermined area image including the material image is changed; It is a figure which shows the predetermined area|region after a change.

An embodiment of the present invention will be described below with reference to FIGS. 1 to 36. FIG.

<<Outline of Embodiment>>
<Method for Generating Spherical Panorama Image>
A method of generating an omnidirectional panorama image will be described with reference to FIGS. 1 to 7. FIG.

First, the appearance of the imaging device 1 will be described with reference to FIG. The photographing device 1 is a digital camera for obtaining a photographed image that is the basis of a three-dimensional omnidirectional (360°) panoramic image. 1(a) is a left side view of the photographing device, FIG. 1(b) is a front view of the photographing device, and FIG. 1(c) is a plan view of the photographing device.

As shown in FIG. 1A, the photographing device 1 is of a size that can be held by a person with one hand. Further, as shown in FIGS. 1A, 1B, and 1C, an image sensor 103a is provided on the front side (front side) and an image sensor 103a on the back side (rear side) on the upper portion of the photographing apparatus 1. ) is provided with an image sensor 103b. These imaging devices (image sensors) 103a and 103b are used together with optical members (for example, fisheye lenses 102a and 102b to be described later) capable of capturing hemispherical images (angle of view of 180° or more). Further, as shown in FIG. 1B, an operation unit 115 such as a shutter button is provided on the side opposite to the front side of the photographing apparatus 1 .

Next, with reference to FIG. 2, the use condition of the photographing device 1 will be described. In addition, FIG. 2 is a use image figure of an imaging device. As shown in FIG. 2, the photographing device 1 is used by a user, for example, to photograph a subject around the user. In this case, the imaging elements 103a and 103b shown in FIG. 1 capture images of subjects around the user, respectively, so that two hemispherical images can be obtained.

Next, with reference to FIGS. 3 and 4, the outline of the process until the omnidirectional panorama image is created from the image captured by the imaging device 1 will be described. In addition, FIG. 3(a) is a hemispherical image (front side) photographed by the photographing device, FIG. 3(b) is a hemispherical image (rear side) photographed by the photographing device, and FIG. 3(c) is represented by the Mercator projection. 1 is a diagram showing an image (hereinafter referred to as a "Mercator image"). FIG. 4(a) is a conceptual diagram showing a state in which a sphere is covered with a Mercator image, and FIG. 4(b) is a diagram showing a omnidirectional panorama image.

As shown in FIG. 3A, the image obtained by the imaging device 103a is a hemispherical image (front side) curved by a fisheye lens 102a, which will be described later. Also, as shown in FIG. 3B, the image obtained by the imaging device 103b becomes a curved hemispherical image (rear side) by a fisheye lens 102b, which will be described later. Then, the hemispherical image (front side) and the 180-degree inverted hemispherical image (rear side) are superimposed by the photographing device 1 to create a Mercator image as shown in FIG. 3(c).

Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is pasted so as to cover the spherical surface as shown in FIG. A spherical panoramic image as shown is created. Thus, the omnidirectional panorama image is represented as an image in which the Mercator image faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the omnidirectional panorama image may be a still image or a moving image.

As described above, since the omnidirectional panoramic image is an image pasted so as to cover the spherical surface, it gives a sense of incongruity when viewed by humans. Therefore, by displaying a predetermined area (hereinafter referred to as a "predetermined area image") of a part of the omnidirectional panorama image as a flat image with little curvature, it is possible to display the image without causing discomfort to the human. This will be described with reference to FIGS. 5 and 6. FIG.

FIG. 5 is a diagram showing positions of a virtual camera and a predetermined area when the omnidirectional image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of the user viewing the omnidirectional image CE displayed as a three-dimensional sphere. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a predetermined area image displayed on the display. Also, in FIG. 6A, the omnidirectional image CE shown in FIG. 4 is represented by a three-dimensional solid sphere CS. Assuming that the omnidirectional image CE generated in this way is the stereosphere CS, the virtual camera IC is positioned inside the omnidirectional image CE as shown in FIG. A predetermined region T in the omnidirectional image CE is a photographing region of the virtual camera IC, and is specified by predetermined region information indicating the photographing direction and angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE. be.

Then, the predetermined area image Q shown in FIG. 6(a) is displayed as an image of the imaging area of the virtual camera IC on a predetermined display, as shown in FIG. 6(b). The image shown in FIG. 6B is a predetermined area image represented by default predetermined area information. Note that the photographing area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, may be used instead of the predetermined area information and the position coordinates of the virtual camera IC. In the following, the shooting direction (rH, rV) and the angle of view (α) of the virtual camera IC will be used.

The relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. FIG. 7 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. As shown in FIG. 7, rH is the Horizontal Radian, rV is the Vertical Radian, and α is the angle of view. That is, the orientation of the virtual camera IC is changed so that the point of gaze of the virtual camera IC indicated by the photographing direction (rH, rV) becomes the central point CP of the predetermined area T which is the photographing area of the virtual camera IC. Become. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP. L is the distance between an arbitrary vertex of the predetermined area T and the center point CP (2L is a diagonal line). In FIG. 7, a trigonometric function generally represented by the following (Equation 1) holds.

Lf=tan(α/2) (Formula 1)
FIG. 8 is a diagram showing points in three-dimensional Euclidean space in spherical coordinates. Let (r, θ, φ) be the position coordinates when the center point CP is expressed in a spherical polar coordinate system. (r, θ, φ) are the radial, polar and azimuthal angles, respectively. The radius r is equal to f because it is the distance from the origin of the three-dimensional virtual space containing the omnidirectional panoramic image to the center point CP. FIG. 8 is a diagram showing these relationships. Hereinafter, the position coordinates (r, θ, φ) of the virtual camera IC will be used.

<Overview of image communication system>
Next, with reference to FIG. 9, an outline of the configuration of the image communication system according to this embodiment will be described. FIG. 9 is a schematic diagram of the configuration of the image communication system of this embodiment.

As shown in FIG. 9, the image communication system of this embodiment includes photographing devices 1a and 1b, video conference terminals 3a and 3b, displays 4a and 4b, a communication management system 5, a PC (Personal Computer) 6, and a PC 7. , a photographing device 8, and a smartphone 9, and can communicate via a communication network 100 such as the Internet. The connection form of the communication network 100 may be either wireless or wired.

Of these, the photographing devices 1a and 1b are special digital cameras for photographing subjects, landscapes, etc., and obtaining two hemispherical images that are the basis of an omnidirectional panorama image, as described above. On the other hand, the photographing device 8 is a general digital camera for photographing subjects, landscapes, etc. to obtain general planar images.

The video conference terminals 3a and 3b are terminals dedicated to video conferences, and display images of video calls on displays 4a and 4b, respectively, via wired cables such as USB (Universal Serial Bus) cables. The videoconference terminal 3a normally takes pictures with a camera 312 shown in FIG. 11, which will be described later. We can obtain two hemispherical images such that Furthermore, the video conference terminal 3a is connected to the PC 6 and can obtain the screen of the PC 6. FIG. When a wired cable is used, the cradle 2a not only performs communication between the imaging device 1a and the video conference terminal 3a, but also serves to supply power to the imaging device 1a and support the imaging device 1a. Here, the imaging device 1a, the cradle 2a, the video conference terminal 3a, the display 4a, and the PC 6 are located at the same base A. At base A, four users A1, A2, A3, and A4 participate in a video call. On the other hand, the video conference terminal 3d and the display 4d are located at the same base D. Also, at base D, three users D1, D2, and D3 are participating in a video call.

The communication management system 5 manages and controls communications of the video conference terminals 3a and 3b, the PC 7, and the smartphone 9, and manages the types of image data transmitted and received (general image and special image type). Therefore, the communication management system is also a communication control system. Here, the special image is a spherical panorama image. The communication management system 5 is installed in a service company or the like that provides video communication services. Further, the communication management system 5 may be constructed by a single computer, or may be constructed by a plurality of computers in which each part (function, means, or storage part) is divided and arbitrarily assigned.

The PC 6 generates material image data (material image data) to be displayed in the video conference. Document images include, but are not limited to, images displayed, created, or edited by a general-purpose application running on the PC 6, or images reproduced on the PC 6 from images taken with a general digital camera or the like. can't

The PC 7 becomes capable of video communication by attaching the photographing device 8 . It should be noted that the PC 7 and the imaging device 8 are located at the base C, which is the same base. In addition, at base C, one user C participates in a video call.

The smartphone 9 displays an image of the video call on a display 917 (described later) provided on the smartphone 9 . The smartphone 9 normally captures images with a CMOS (Complementary Metal Oxide Semiconductor) sensor 905 or the like, which is provided in the device itself. It is possible to acquire two hemispherical image data that are the basis of the omnidirectional panorama image obtained by the photographing device 1b. When wireless communication technology is used, the cradle 2b serves only to supply power to the photographing device 1b and to support the photographing device 1b. Here, the imaging device 1b, the cradle 2b, and the smart phone 9 are located at the same base B, which is the same base. At base B, two users B1 and B2 are participating in a video call.

Also, the video conference terminals 3a and 3d, the PC 7, and the smart phone 9 are examples of communication terminals. OpenGL ES is installed on each communication terminal, and it is possible to create predetermined area information that indicates a part of the spherical panoramic image, and to display panoramic panoramic images sent from other communication terminals. A predetermined area image can be created from the

Note that the arrangement of each terminal (communication terminal, display, photographing device), devices, and users shown in FIG. 9 is an example, and other examples may be used. For example, at the base C, instead of the imaging device 8, the user may use an imaging device capable of capturing an omnidirectional panorama image. Communication terminals also include digital televisions, smart watches, car navigation devices, and the like. Further, hereinafter, when an arbitrary image capturing device out of the image capturing devices 1a and 1b is indicated, it will be referred to as "image capturing device 1". Also, when representing an arbitrary video conference terminal among the video conference terminals 3a and 3b, it is expressed as "video conference terminal 3". Furthermore, when representing an arbitrary display out of the displays 4a and 4b, it is represented as "display 4".

<<Hardware configuration of the embodiment>>
Next, the hardware configurations of the photographing device 1, the video conference terminal 3, the communication management system 5, the PC 6, the PC 7, and the smart phone 9 of this embodiment will be described in detail with reference to FIGS. 10 to 13. FIG.
In addition, since the imaging device 8 is a general camera, detailed description thereof will be omitted.

<Hardware configuration of imaging device 1>
First, the hardware configuration of the photographing device 1 will be described with reference to FIG. FIG. 10 is a hardware configuration diagram of the imaging device 1. As shown in FIG. In the following description, the photographing apparatus 1 is assumed to be an omnidirectional (omnidirectional) photographing apparatus using two imaging elements, but the number of imaging elements may be two or more. Further, the device does not necessarily need to be dedicated to omnidirectional photography, and may have substantially the same functions as the imaging device 1 by attaching an omnidirectional photography unit to an ordinary digital camera, smartphone, or the like. .

As shown in FIG. 10, the imaging apparatus 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory). ) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network I/F 116, a communication unit 117, and an antenna 117a.

Among them, the imaging unit 101 includes wide-angle lenses (so-called fish-eye lenses) 102a and 102b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging units provided corresponding to each wide-angle lens. It has elements 103a and 103b. The imaging elements 103a and 103b are image sensors such as CMOS sensors and CCD (Charge Coupled Device) sensors that convert optical images by the fisheye lenses 102a and 102b into image data of electric signals and output them, and horizontal or vertical synchronization signals of the image sensors. , a timing generation circuit that generates a pixel clock, etc., and a group of registers in which various commands and parameters required for the operation of the image sensor are set.

The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I/F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are connected separately from the imaging control unit 105 by a serial I/F bus (I2C bus or the like). The image processing unit 104 and imaging control unit 105 are connected to the CPU 111 via the bus 110 . Furthermore, ROM 112, SRAM 113, DRAM 114, operation unit 115, network I/F 116, communication unit 117, electronic compass 118, and the like are also connected to bus 110. FIG.

The image processing unit 104 fetches the image data output from the imaging elements 103a and 103b through the parallel I/F bus, performs predetermined processing on each image data, and superimposes these image data. , to create Mercator image data as shown in FIG. 3(c).

The imaging control unit 105 generally uses the I2C bus with the imaging control unit 105 as a master device and the imaging elements 103a and 103b as slave devices to set commands and the like in registers of the imaging elements 103a and 103b. Necessary commands and the like are received from the CPU 111 . The imaging control unit 105 also uses the I2C bus to capture status data and the like of the registers of the imaging elements 103 a and 103 b and sends them to the CPU 111 .

Further, the imaging control unit 105 instructs the imaging devices 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Depending on the photographing device 1, the display (for example, the display of the video conference terminal 3a) may have a preview display function or a function corresponding to moving image display. In this case, the image data is output continuously from the imaging devices 103a and 103b at a predetermined frame rate (frames/minute).

In addition, as will be described later, the imaging control unit 105 also functions as synchronization control means for synchronizing the output timing of the image data of the imaging devices 103a and 103b in cooperation with the CPU 111. FIG. Note that in the present embodiment, the imaging device 1 is not provided with a display unit (display), but a display unit may be provided.

The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in sound data output from the microphone 108 through the I/F bus and performs predetermined processing on the sound data.

The CPU 111 controls the overall operation of the photographing device 1 and executes necessary processing. ROM 112 stores various programs for CPU 111 . The SRAM 113 and DRAM 114 are work memories, and store programs to be executed by the CPU 111, data during processing, and the like. In particular, the DRAM 114 stores image data being processed in the image processing unit 104 and processed Mercator image data.

The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, and the like. A user inputs various shooting modes, shooting conditions, and the like by operating operation buttons.

The network I/F 116 is a general term for interface circuits (such as USB I/F) with external media such as SD cards and personal computers. Also, the network I/F 116 may be wireless or wired. Mercator image data stored in the DRAM 114 is recorded on an external medium via the network I/F 116, or is transmitted to an external device such as the video conference terminal 3a via the network I/F 116 as necessary. or

The communication unit 117 communicates with an external device such as the video conference terminal 3a via an antenna 117a provided in the imaging device 1, using short-range wireless technology such as WiFi or NFC (Near Field Communication). This communication unit 117 can also transmit Mercator image data to an external device of the video conference terminal 3a.

The electronic compass 118 calculates the azimuth and tilt (Roll rotation angle) of the imaging device 1 from the magnetism of the earth, and outputs the azimuth/tilt information. This azimuth/tilt information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of a captured image. The related information also includes data such as the date and time when the image was taken and the data volume of the image data.

<Hardware configuration of the video conference terminal>
Next, the hardware configuration of the video conference terminal 3 will be described with reference to FIG. FIG. 11 is a hardware configuration diagram of a video conference terminal. As shown in FIG. 11, the video conference terminal 3 includes a CPU 301, a ROM 302, a RAM 303, a flash memory 304, an SSD 305, a media I/F 307, an operation button 308, a power switch 309, a bus line 310, a network I/F 311, Equipped with a camera 312, an image sensor I/F 313, a microphone 314, a speaker 315, a sound input/output I/F 316, a display I/F 317, an external device connection I/F 318, a short-range communication circuit 319, and an antenna 319a of the short-range communication circuit 319. ing.

Among these, the CPU 301 controls the operation of the video conference terminal 3 as a whole. A ROM 302 stores a program such as an IPL (Initial Program Loader) that is used to drive the CPU 301 . A RAM 303 is used as a work area for the CPU 301 . The flash memory 304 stores various data such as communication programs, image data, and sound data. An SSD (Solid State Drive) 305 controls reading or writing of various data to/from a flash memory 304 under the control of the CPU 301 . Note that an HDD may be used in place of the SSD. A media I/F 307 controls reading or writing (storage) of data to a recording medium 306 such as a flash memory. The operation button 308 is a button that is operated when selecting a destination of the video conference terminal 3 or the like. A power switch 309 is a switch for switching ON/OFF of the power of the video conference terminal 3 .

A network I/F 311 is an interface for data communication using the communication network 100 such as the Internet. The camera 312 is a type of built-in image capturing means that captures an image of a subject under the control of the CPU 301 and obtains image data. An imaging device I/F 313 is a circuit that controls driving of the camera 312 . The microphone 314 is a type of built-in sound collecting means for inputting sound. A sound input/output I/F 316 is a circuit that processes input/output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301 . A display I/F 317 is a circuit that transmits image data to the external display 4 under the control of the CPU 301 . The external device connection I/F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit such as NFC (registered trademark) and Bluetooth (registered trademark).

A bus line 310 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 301 shown in FIG.

The display 4 is a kind of display means configured by liquid crystal or organic EL for displaying an image of a subject, icons for operation, and the like. Also, the display 4 is connected to the display I/F 317 by a cable 4c. This cable 4c may be a cable for analog RGB (VGA) signals, a cable for component video, HDMI (High-Definition Multimedia Interface) (registered trademark) or DVI (Digital Video Interface). Interactive) signal cable.

Note that the camera 312 includes a lens and a solid-state imaging device that converts light into electric charge and digitizes an image (video) of a subject. As the solid-state imaging device, a CMOS sensor, a CCD sensor, or the like is used. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I/F 318 via a USB (Universal Serial Bus) cable or the like. When an external camera is connected, the external camera is driven according to the control of the CPU 301 prior to the built-in camera 312 . Similarly, when an external microphone is connected or when an external speaker is connected, according to the control of the CPU 301, the external microphone and speaker are prioritized over the built-in microphone 314 and built-in speaker 315, respectively. Powered by an external speaker.

Also, the recording medium 306 is detachably attached to the video conference terminal 3 . In addition, as long as it is a non-volatile memory from which data is read or written under the control of the CPU 301, not only the flash memory 304 but also an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used.

<Communication management system, PC hardware configuration>
Next, the hardware configuration of the communication management system 5, PC6 and PC7 will be described with reference to FIG. FIG. 12 is a hardware configuration diagram of the communication management system and PC. Since the communication management system 5, the PC6 and the PC7 are all computers and have the same configuration, the configuration of the communication management system 5 will be described below, and the description of the configurations of the PC6 and PC7 will be omitted.

The communication management system 5 includes a CPU 501 that controls the overall operation of the communication management system 5, a ROM 502 that stores programs used to drive the CPU 501 such as IPL, a RAM 503 that is used as a work area for the CPU 501, and programs for the communication management system 5. HD 504 for storing various data such as; HDD (Hard Disk Drive) 505 for controlling reading or writing of various data to the HD 504 under the control of the CPU 501; A media drive 507 to be controlled, a display 508 for displaying various information such as cursors, menus, windows, characters or images, a network I/F 509 for data communication using the communication network 100, characters, numerical values, various instructions, etc. , a mouse 512 for selecting and executing various instructions, selecting an object to be processed, moving a cursor, etc., and a CD-RW (Compact Disc) as an example of a removable recording medium. -ReWritable) 513, a CD-RW drive 514 for controlling reading of various data, and bus lines such as an address bus and a data bus for electrically connecting the above components as shown in FIG. 510 is provided.

<Hardware configuration of smartphone>
Next, the hardware of the smart phone will be described with reference to FIG. 13 . FIG. 13 is a hardware configuration diagram of a smartphone. As shown in FIG. 13 , the smartphone 9 includes a CPU 901 , ROM 902 , RAM 903 , EEPROM 904 , CMOS sensor 905 , acceleration/direction sensor 906 , media I/F 908 and GPS receiver 909 .

Among these, the CPU 901 controls the operation of the smartphone 9 as a whole. The ROM 902 stores programs such as IPL, which are used to drive the CPU 901 . A RAM 903 is used as a work area for the CPU 901 . The EEPROM 904 reads or writes various data such as smartphone programs under the control of the CPU 901 . A CMOS sensor 905 captures an image of a subject (mainly a self-portrait) and obtains image data under the control of the CPU 901 . The acceleration/direction sensor 906 is various sensors such as an electronic magnetic compass, a gyro compass, and an acceleration sensor for detecting geomagnetism. A media I/F 908 controls reading or writing (storage) of data to a recording medium 907 such as a flash memory. A GPS receiver 909 receives GPS signals from GPS satellites.

In addition, the smartphone 9 includes a long-distance communication circuit 911, a camera 912, an image sensor I/F 913, a microphone 914, a speaker 915, a sound input/output I/F 916, a display 917, an external device connection I/F 918, a short-distance communication circuit 919, An antenna 919 a of a short-range communication circuit 919 and a touch panel 921 are provided.

Among these, the telecommunications circuit 911 is a circuit that communicates with other devices via the communication network 100 . The camera 912 is a type of built-in imaging means for capturing an image of a subject and obtaining image data under the control of the CPU 901 . An imaging device I/F 913 is a circuit that controls driving of the camera 912 . The microphone 914 is a type of built-in sound collecting means for inputting sound. A sound input/output I/F 916 is a circuit that processes input/output of sound signals between the microphone 914 and the speaker 915 under the control of the CPU 901 . A display 917 is a kind of display means such as a liquid crystal or an organic EL that displays an image of a subject, various icons, and the like. The external device connection I/F 918 is an interface for connecting various external devices. A short-range communication circuit 919 is a communication circuit such as NFC or Bluetooth. The touch panel 921 is a type of input means for operating the smartphone 9 by the user pressing the display 917 .

The smartphone 9 also includes a bus line 910 . A bus line 910 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 901 .

Recording media such as CD-ROMs storing the above programs and HDs storing these programs can both be provided domestically or internationally as program products.

<<Function Configuration of Embodiment>>
Next, the functional configuration of this embodiment will be described with reference to FIGS. 14 to 21. FIG. 14 and 15 are functional block diagrams of part of the image communication system.

<Functional configuration of imaging device 1a>
As shown in FIG. 14, the photographing device 1a has a reception section 12a, an imaging section 13a, a sound collection section 14a, a communication section 18a, and a storage/readout section 19a. Each of these units is a function or means implemented by any one of the components shown in FIG. is.

The photographing apparatus 1 also has a storage unit 1000a constructed by the ROM 112, SRAM 113, and DRAM 114 shown in FIG. The storage unit 1000a stores a GUID (Globally Unique Identifier) of its own device.

Note that the imaging device 1b has a reception unit 12b, an imaging unit 13b, a sound collection unit 14b, a communication unit 18b, a storage/readout unit 19b, and a storage unit 1000b. , the imaging unit 13a, the sound collecting unit 14a, the communication unit 18a, the storage/reading unit 19a, and the storage unit 1000a.

(Each functional configuration of the imaging device 1a)
Next, with reference to FIGS. 10 and 14, each functional configuration of the photographing device 1a will be described in further detail.

The reception unit 12a of the photographing device 1 is realized mainly by processing of the operation unit 115 and the CPU 111 shown in FIG. 10, and receives operation input from the user.

The imaging unit 13a is realized mainly by the processing of the imaging unit 101, the image processing unit 104, the imaging control unit 105, and the CPU 111 shown in FIG. image data).

The sound collecting unit 14a is realized by the processing of 108, the sound processing unit 109, and the CPU 111 shown in FIG. 10, and collects sounds around the imaging device 1a.

The communication unit 18a is mainly realized by the processing of the CPU 111, and can communicate with the communication unit 38a of the video conference terminal 3a by short-range wireless communication technology such as the NFC standard, BlueTooth, and WiFi.

The storage/readout unit 19a is mainly realized by the processing of the CPU 111 shown in FIG. read out.

<Functional Configuration of Video Conference Terminal 3a>
As shown in FIG. 14, the videoconference terminal 3a includes a transmission/reception unit 31a, a reception unit 32a, an image/sound processing unit 33a, a display control unit 34a, a determination unit 35a, a creation unit 36a, a calculation unit 37a, a communication unit 38a, and a storage/readout unit 39a. Each of these units is realized by any one of the components shown in FIG. It is a function or a means.

The videoconference terminal 3a also has a storage unit 3000a constructed by the ROM 302, RAM 303, and flash memory 304 shown in FIG. An image type management DB 3001a, an imaging device management DB 3002a, a predetermined area management DB 3003a, and a superimposition position management DB 3004a are constructed in the storage unit 3000a. Of these, the image type management DB 3001a is composed of the image type management table shown in FIG. The imaging device management DB 3002a is composed of the imaging device management table shown in FIG. The predetermined area management DB 3003a is composed of a predetermined area management table shown in FIG. The superimposed position management DB 3004a is composed of a superimposed position management table shown in FIG.

The video conference terminal 3d includes a transmission/reception unit 31d, a reception unit 32d, an image/sound processing unit 33d, a display control unit 34d, a determination unit 35d, a creation unit 36d, a calculation unit 37d, a communication unit 38d, a storage/readout unit 39d, and a storage unit 3000d, which respectively include a transmission/reception unit 31a, a reception unit 32a, an image/sound processing unit 33a, a display control unit 34a, a judgment unit 35a, a creation unit 36a, a calculation unit 37a, and a storage unit 3000d. Since the same functions as those of the communication unit 38a, the storage/readout unit 39a, and the storage unit 3000a are realized, descriptions thereof will be omitted. An image type management DB 3001d, a photographing device management DB 3002d, a predetermined area management DB 3003d, and a superimposition position management DB 3004d are constructed in the storage unit 3000d of the video conference terminal 3d. Since the data structure is the same as that of the device management DB 3002a, the predetermined area management DB 3003a, and the superimposition position management DB 3004a, the description thereof will be omitted.

(Image type management table)
FIG. 16 is a conceptual diagram showing an image type management table. In this image type management table, an image data ID, an IP address as an example of a destination of a transmission source terminal, and a source name are stored and managed in association with each other. Among these, the image data ID is an example of image data identification information for identifying image data when performing video communication. The same image data ID is added to the image data transmitted from the same source terminal. As a result, the destination terminal (communication terminal on the receiving side) can identify the source terminal of the received image data. The IP address of the transmission source terminal indicates the IP address of the communication terminal that transmits the image data indicated by the associated image data ID. A source name is a name for specifying an imaging device that outputs image data indicated by an associated image data ID, and is an example of image type information. This source name is a name created by each communication terminal such as the video conference terminal 3a according to a predetermined naming rule.

For example, four communication terminals whose IP addresses are "1.2.1.3", "1.2.2.3", "1.3.1.3", and "1.3.2.3" are transmitting image data indicated by image data IDs "RS001", "RS002", "RS003", and "RS004", respectively. Further, the types of images indicated by the source names of the respective communication terminals are "Video_Theta", "Video_Theta", "Video", and "Video", and these image types are "special image", "special image", It indicates that it is a "general image" or a "general image". Note that the special image is a spherical panorama image here.

Data other than image data may also be managed in association with the image data ID. Data other than image data is, for example, sound data (imaging device management table)
FIG. 17 is a conceptual diagram showing an imaging device management table. In this photographing device management table, vendor IDs and product IDs of the GUIDs of photographing devices that can obtain two hemispherical images that are the basis of the omnidirectional panorama image are stored and managed. As the GUID, for example, the vendor ID (VID) and product ID (PID) used in the USB device can be used. The vendor ID and product ID are stored when the communication terminal such as the video conference terminal is shipped from the factory, but they may be additionally stored after the shipment from the factory.

(predetermined area management table)
FIG. 18 is a conceptual diagram showing a predetermined area management table. In this predetermined area management table, the IP address of the communication terminal that is the transmission source of the captured image data, the IP address of the communication terminal that is the transmission destination of the captured image data, and the predetermined area being displayed on the communication terminal that is the transmission destination of the captured image data. Predetermined area information indicating an image is associated and stored and managed. The communication terminal that is the transmission destination of the captured image data is also the communication terminal that is the transmission source of the predetermined area information. The predetermined area information is, as shown in FIGS. 6 and 7, a transformation parameter for transforming a captured image into an image of a predetermined area T (predetermined area image) in the captured image. Note that the IP address is an example of destination information, and the destination information includes a MAC (Media Access Control) address, a terminal ID (Identification) for specifying a communication terminal, and the like. Also, here, the IP address is represented by simplifying the IPv4 address. The IP address may be IPv6.

For example, the 1st to 3rd rows of the predetermined area management table in FIG. The data is transferred via the communication management system 5 to the video conference terminal 3d with the IP address "1.2.2.3", the PC 7 with the IP address "1.3.1.3", and the IP address "1". .3.2.3” is managed. Furthermore, it is managed that the video conference terminal 3d is the communication terminal that has transmitted the predetermined area information (r=10, θ=20, φ=30). Similarly, it is managed that the PC 7 is the communication terminal that has transmitted the predetermined area information (r=10, θ=30, φ=40). Further, it is managed that the smartphone 9 is the communication terminal that has transmitted the predetermined area information (r=30, θ=40, φ=50).

Further, the transmitting/receiving unit 31a newly updates the predetermined area information including the IP address of the same group as the already managed IP address of the transmission source communication terminal of the captured image data and the IP address of the transmission destination communication terminal of the captured image data. When the information is received at 10:00 a.m., the storage/reading unit 39a rewrites the predetermined area information already managed by the newly received predetermined area information.

(Superposition position management table)
FIG. 19 is a conceptual diagram showing a superimposition position management table. In this superimposition position management table, the IP address of the communication terminal that is the transmission source of the captured image data, and the superimposition position information indicating where the material image that is the superimposition source image is to be superimposed on the omnidirectional panorama image that is the superimposition destination image. are stored and managed in association with each other.

Superimposed coordinates as an example of superimposed position information will be described with reference to FIG. 20 . FIG. 20 shows a Mercator image of an omnidirectional video on which a material image, which is a superimposition source image, is superimposed. The superimposed coordinates consist of start point coordinates and end point coordinates. Assuming that the top left vertex position of the material image that is the superimposition source and the starting point, the starting point coordinates are expressed as (0, 0) and the ending point coordinates are expressed as (xn, yn). At this time, when superimposing the start point coordinates (0, 0) and end point coordinates (xn, yn) on the positions corresponding to the coordinates (x1, y1) and coordinates (x2, y2) in the Mercator image, the superimposition position management table , with (x1,x2) as the start coordinates and (x2,y2) as the end coordinates. Note that the IP address is an example of destination information, and the destination information includes a MAC (Media Access Control) address, a terminal ID (Identification) for specifying a communication terminal, and the like. Also, here, the IP address is represented by simplifying the IPv4 address. The IP address may be IPv6.

(Each functional configuration of the video conference terminal 3a)
Next, each functional configuration of the video conference terminal 3a will be described in more detail with reference to FIGS. 11 and 14. FIG.

The transmission/reception unit 31a of the videoconference terminal 3a is mainly realized by processing of the network I/F 311 and the CPU 301 shown in FIG. send and receive

The reception unit 32a is realized mainly by processing by the operation buttons 308 and the CPU 301, and receives various selections or inputs from the user. In addition to the operation buttons 308, a touch panel or the like may be used as other input means.

The image/sound processing unit 33a is implemented by a command from the CPU 301 shown in FIG. 11, and performs image processing on image data obtained by imaging a subject with the camera 312. FIG. Further, after the user's voice is converted into an audio signal by the microphone 314, the image/sound processing unit 33a performs audio processing on the audio data related to this audio signal.

Further, the image/sound processing unit 33a performs image processing on image data received from another communication terminal based on image type information such as a source name so that the display control unit 34 can display an image on the display 4. do Specifically, when the image type information indicates that the image is a special image, the image/sound processing unit 33a processes the image data (for example, each of the images shown in FIGS. 3(a) and 3(b)). Based on the hemispherical image data), the omnidirectional panorama image is created by converting into an omnidirectional panorama image as shown in FIG. 4(b). Furthermore, if there is a material image to be superimposed on the omnidirectional panoramic image, a superimposed image is generated by superimposing the material image on the omnidirectional panoramic image based on the material image data and the superimposition position information. The image/sound processing unit 33a also creates a predetermined area image as shown in FIG. 6(b). In addition, the image/sound processing unit 33a outputs to the speaker 315 an audio signal related to sound data received from another communication terminal via the communication management system 5, and causes the speaker 315 to output sound.

The display control unit 34a is mainly realized by processing of the display I/F 317 and the CPU 301, and performs control for displaying various images, characters, etc. on the display 4. FIG.

The determination unit 35a is mainly realized by the processing of the CPU 301, and, for example, determines the image type related to the image data received from the photographing device 1a, but is not limited to this and makes various other determinations related to the image data.

The creating unit 36a is mainly implemented by the processing of the CPU 301, and based on the result of the determination by the determination unit 35a that the image is a general image or a special image (here, a spherical panorama image), determines the image type according to the above naming rule. Create a source name, which is an example of information. For example, when the determination unit 35a determines that the image is a general image, the creation unit 36a creates a source name "Video" indicating that it is a general image. On the other hand, when the determination unit 35a determines that the image is a special image, the creation unit 36a creates a source name "Video_Theta" indicating that it is a special image.

The calculation unit 37a is mainly realized by processing of the CPU 301, and calculates superimposition position information.

The communication unit 38a is mainly implemented by the short-range communication circuit 319, the antenna 318a, and the processing of the CPU 301, and can communicate with the communication unit 18a of the imaging device 1a by short-range wireless technology such as NFC, BlueTooth, WiFi, or the like. can. Although the communication unit 38a and the transmission/reception unit 31a have separate communication units, they may be shared.

The storage/readout unit 39a is mainly implemented by the processing of the CPU 301 shown in FIG. read out.

<Functional configuration of communication management system>
Next, each functional configuration of the communication management system 5 will be described in detail with reference to FIGS. 12 and 15. FIG. The communication management system 5 has a transmission/reception section 51 , a judgment section 55 , a generation section 56 and a storage/read section 59 . 12 operates according to commands from the CPU 501 according to the program for the communication management system 5 developed on the RAM 503 from the HD 504. It is a means.

The communication management system 5 also has a storage unit 5000 constructed by the RAM 503 and the HD 504 shown in FIG. A session management DB 5001 , an image type management DB 5002 , a predetermined area management DB 5003 , and a ratio management DB 5004 are constructed in the storage unit 5000 . Of these, the session management DB 5001 is composed of the session management table shown in FIG. The image type management DB 5002 is composed of an image type management table shown in FIG. The predetermined area management DB 5003 is composed of a predetermined area information management table shown in FIG. The ratio management DB 5004 is composed of the ratio management table shown in FIG.

(session management table)
FIG. 21 is a conceptual diagram showing a session management table. In this session management table, session IDs and IP addresses of participating communication terminals are stored and managed in association with each other. Among these, the session ID is an example of session identification information for identifying a communication session that implements a video call, and is generated for each virtual conference room. The session ID is also managed by each communication terminal such as the video conference terminal 3a, and is used when selecting a communication session in each communication terminal. The IP address of the participating communication terminal indicates the IP address of the communication terminal that has participated in the virtual conference room indicated by the associated session ID.

(Image type management table)
FIG. 22 is a conceptual diagram showing an image type management table. In the image type management table shown in FIG. 22, in addition to each information managed in the image type management table shown in FIG. 16, the same session IDs as those managed in the session management table are associated. managed by Here, the virtual conference rooms indicated by the same session ID "se101" have IP addresses "1.2.1.3", "1.2.2.3", and "1.3.1. 3" are shown participating. In the communication management system 5, the same image data ID, source terminal IP address, and image type information managed by communication terminals such as the video conference terminal 3a are managed by a new communication terminal. This is for transmitting image type information and the like to a communication terminal that is already engaged in a video call and a communication terminal that newly participates, such as when entering a virtual conference room. As a result, there is no need to transmit and receive image type information and the like between a communication terminal that is already engaged in a video call and a communication terminal that has newly joined.

(Predetermined area information management table)
FIG. 23 is a conceptual diagram showing a predetermined area information management table. This predetermined area information management table basically has the same data structure as the predetermined area information management table shown in FIG. However, as will be described later, the transmitting/receiving unit 51 transmits the latest predetermined area information to each communication terminal at regular intervals (for example, 30 seconds). , all the predetermined area information received by the transmitting/receiving unit 51 are stored without being deleted. In FIG. 21, newer predetermined area information is managed at a higher level.

(Proportion management table)
FIG. 24 is a conceptual diagram showing a ratio management table. In this ratio management table, the IP address of the communication terminal that is the transmission source of the captured image data, the IP address of the communication terminal that is the transmission destination of the captured image data, and the ratio information are stored and managed in association with each other. The ratio information is a predetermined area image in the omnidirectional panorama image data sent from the transmission source communication terminal to display means such as a display at the communication terminal of the transmission destination (receiving side) of the captured image data, and When displaying a material image (an example of a superimposed image) related to the material image data, the display area ratios of the predetermined area image and the material image are shown.

(Each functional configuration of the communication management system)
Next, each functional configuration of the communication management system 5 will be described in detail with reference to FIGS. 12 and 15. FIG.

The transmission/reception unit 51 of the communication management system 5 is mainly realized by processing of the network I/F 509 and CPU 501 shown in FIG. Send and receive (or information).

The determination unit 55 is mainly implemented by the processing of the CPU 501 and makes various determinations.

The generation unit 56 is mainly realized by processing of the CPU 501 and generates an image data ID.

The storage/readout unit 59 is mainly realized by processing of the HDD 505 and the CPU 501 shown in FIG. information).

<PC functional configuration>
Next, each functional structure of PC6 is demonstrated in detail using FIG.12 and FIG.14.

The PC 6 has a reception section 62, a display control section 64, a communication section 68, and a storage/read section 69, as shown in FIG. Each of these units is a function or means realized by any one of the components shown in FIG. .

The PC 6 also has a storage unit 6000 constructed by the ROM 502, RAM 503, and HD 504 shown in FIG.

(Each functional configuration of the PC)
The reception unit 62 of the PC 6 is realized mainly by the processing of the keyboard 511, the mouse 512 and the CPU 501, and realizes the same functions as the reception unit 32a. The display control unit 64 is mainly realized by the processing of the CPU 501 and performs control for displaying various images, characters, etc. on the display 508 . The communication unit 68 is mainly realized by the processing of the CPU 501, and can communicate with the communication unit 38a of the video conference terminal 3a by short-range wireless communication technology such as the NFC standard, BlueTooth, and WiFi. The storage/readout unit 69a is implemented by the processing of the CPU 501, stores various data (or information) in the storage unit 6000, and reads out various data (or information) from the storage unit 6000. FIG.

<Functional Configuration of PC as Communication Terminal>
Next, the functional configuration of the PC 7 will be described in detail with reference to FIGS. 12 and 15. FIG. The PC 7 basically has the same functions as the video conference terminal 3a. That is, as shown in FIG. 14, the PC 7 includes a transmission/reception unit 71, a reception unit 72, an image/sound processing unit 73, a display control unit 74, a determination unit 75, a creation unit 76, a calculation unit 77, and a communication unit 78. , and a storage/readout unit 79 . Each of these units is a function or means realized by any one of the components shown in FIG. .

The PC 7 also has a storage unit 7000 constructed by the ROM 502, RAM 503, and HD 504 shown in FIG. In this storage unit 7000, an image type management DB 7001, an imaging device management DB 7002, a predetermined area management DB 7003, and a superimposition position management DB 7004 are constructed. The image type management DB 7001, imaging device management DB 7002, predetermined area management DB 7003, and superimposition position management DB 7004 have the same data structures as the image type management DB 3001a, imaging device management DB 3002a, predetermined region management DB 3003a, and superimposition position management DB 3004a, respectively. Therefore, descriptions thereof are omitted.

(Each functional configuration of a PC as a communication terminal)
The transmitting/receiving section 71 of the PC 7 is mainly realized by the processing of the network I/F 509 and the CPU 501 shown in FIG. 12, and realizes the same function as the transmitting/receiving section 31a.

The reception unit 72 is realized mainly by the keyboard 511, the mouse 512, and the processing of the CPU 501, and realizes the same functions as the reception unit 32a. The image/sound processing unit 73 is mainly implemented by commands from the CPU 501, and implements the same functions as the image/sound processing unit 33a. The display control unit 74 is mainly implemented by the processing of the CPU 501, and implements the same function as the display control unit 34a. The determination unit 75 is mainly implemented by the processing of the CPU 501 and implements the same function as the determination unit 35a. The creation unit 76 is mainly implemented by the processing of the CPU 501 and implements the same function as the creation unit 36a. The calculation unit 77 is mainly realized by the processing of the CPU 501, and realizes the same function as the calculation unit 37a. The communication unit 78 is mainly implemented by the processing of the CPU 501 and implements the same functions as the communication unit 38a. The storage/readout unit 79a is realized by the processing of the CPU 501, stores various data (or information) in the storage unit 7000, and reads out various data (or information) from the storage unit 7000. FIG.

<Smartphone functional configuration>
Next, the functional configuration of the smartphone 9 will be described in detail with reference to FIGS. 13 and 14. FIG. The smart phone 9 basically has the same functions as the video conference terminal 3a. That is, as shown in FIG. 14, the smartphone 9 includes a transmission/reception unit 91, a reception unit 92, an image/sound processing unit 93, a display control unit 94, a determination unit 95, a creation unit 96, a calculation unit 97, a communication unit 98 and a storage/readout unit 99 . Each of these units is a function or means implemented by any one of the components shown in FIG. be.

The smartphone 9 also has a storage unit 9000 constructed by the ROM 902, RAM 903, and EEPROM 904 shown in FIG. In this storage unit 9000, an image type management DB 9001, an imaging device management DB 9002, a predetermined area management DB 9003, and an overlay position management DB 9004 are constructed. The image type management DB 9001, the photographing device management DB 9002, the predetermined area management DB 9003, and the superimposition position management DB 9004 have the same data configurations as the image type management DB 3001a, the photographing device management DB 3002a, the predetermined region management DB 3003a, and the superimposition position management DB 3004a. Therefore, the description of these is omitted.

(Each function configuration of the smartphone)
The transmitting/receiving unit 91 of the smartphone 9 is mainly implemented by the processing of the long-distance communication circuit 911 and the CPU 901 shown in FIG. 13, and implements the same function as the transmitting/receiving unit 31a.

The reception unit 92 is mainly realized by processing of the touch panel 921 and the CPU 901, and realizes the same function as the reception unit 32a.

The image/sound processing unit 93 is implemented mainly by commands from the CPU 901, and implements the same functions as the image/sound processing unit 33a. The display control unit 94 is mainly implemented by the processing of the CPU 901, and implements the same functions as the display control unit 34a. The determination unit 95 is mainly implemented by the processing of the CPU 901, and implements the same function as the determination unit 35a. The creation unit 96 is mainly implemented by the processing of the CPU 901, and implements the same function as the creation unit 36a. The calculation unit 97 is mainly realized by the processing of the CPU 901, and realizes the same function as the calculation unit 37a. The communication unit 98 is mainly implemented by the processing of the CPU 901 and implements the same functions as the communication unit 38a. A storage/readout unit 99 is realized by processing of the CPU 901 , stores various data (or information) in the storage unit 9000 , and reads out various data (or information) from the storage unit 9000 .

<<Processing or operation of the embodiment>>
Next, the processing or operation of this embodiment will be described with reference to FIGS. 25 to 36. FIG.

<Participation processing>
First, processing for joining a specific communication session will be described with reference to FIGS. 25 and 26. FIG. FIG. 25 is a sequence diagram showing the process of joining a specific communication session. FIG. 26 is a diagram showing a screen for selecting a communication session (virtual conference room).

First, when a user of site A (for example, user A1) performs an operation to display a communication session (virtual conference room) selection screen on video conference terminal 3a, reception unit 32a displays the selection screen. Upon receiving the operation, the display control unit 34a displays a selection screen as shown in FIG. 26 on the display 4a (step S21). On this selection screen, selection buttons b1, b2, b3, etc., indicating virtual conference rooms R1, R2, R3, etc., to be selected are displayed. Each session ID is associated with each selection button b1 and the like.

Here, when the user A1 selects a desired virtual conference room selection button (here, the selection button b1), the reception unit 32a receives selection of a communication session (step S22). The transmitter/receiver 31a then transmits a request to participate in the virtual conference room to the communication management system 5 (step S23). This participation request includes the session ID indicating the communication session whose selection was accepted in step S22, and the IP address of the videoconference terminal 3a, which is the request source terminal. Thereby, the transmission/reception unit 51 of the communication management system 5 receives the participation request.

Next, the storage/readout unit 59 stores the IP address received in step S23 in the field of the participating terminal IP address in the record of the same session ID as the session ID received in step S23 in the session management DB 5001 (see FIG. 21). By adding the address, processing for joining the communication session is performed (step S24). Then, the transmission/reception unit 51 transmits a participation request response to the video conference terminal 3a (step S25). This participation request response includes the session ID received in step S23 and the result of the participation processing. As a result, the transmission/reception unit 31a of the videoconference terminal 3a receives the participation request response. Hereinafter, a case where the participation processing is successful will be described.

<Management processing of image type information>
Next, image type information management processing will be described with reference to FIG. FIG. 27 is a sequence diagram showing management processing of image type information.

First, when a user at site A (for example, user A1) connects the USB cable of cradle 2a with photographing device 1a attached to video conference terminal 3a, storage/reading unit 19a of photographing device 1a The GUID of its own device (photographing device 1a) stored in the storage unit 1000a is read, and the communication unit 18a transmits the GUID of its own device to the communication unit 38a of the video conference terminal 3a (step S51). As a result, the communication unit 38a of the video conference terminal 3a receives the GUID of the imaging device 1a.

Next, the determination unit 35a of the video conference terminal 3a determines whether the same vendor ID and product ID as those in the GUID received in step S51 are managed in the photographing device management DB 3002a (see FIG. 17). The type of image is determined by determining whether or not it exists (step S52). Specifically, when the same vendor ID and product ID are managed in the imaging device management DB 3002a, the determination unit 35a determines that the imaging device 1a captures a special image (here, a spherical panorama image). It is determined that it is an imaging device. On the other hand, if the same vendor ID and product ID are not managed in the image capturing apparatus management DB 3002a, the determination unit 35a determines that the image capturing apparatus 1a is an image capturing apparatus that captures general images.

Next, the storage/readout unit 39a stores the IP address of its own terminal (videoconference terminal 3a), which is the transmission source terminal, and the determination result determined in step S52 in the image type management DB 3001a (see FIG. 16). It is stored in association with certain image type information (step S53). In this state, no image data ID is associated. The image type information is, for example, a source name determined according to a predetermined naming rule and an image type (general image, special image).

Next, the transmission/reception unit 31a transmits a request for adding image type information to the communication management system 5 (step S54). This image type information addition request includes the IP address of the terminal itself, which is the transmission source terminal, stored in step S53, and the image type information. Accordingly, the transmission/reception unit 51 of the communication management system 5 receives the request for adding the image type information.

Next, the storage/readout unit 59 of the communication management system 5 retrieves the corresponding session ID by searching the session management DB 5001 (see FIG. 21) using the IP address of the transmission source terminal received in step S54 as a search key. is read out (step S55).

Next, the generator 56 generates a unique image data ID (step S56). Then, the storage/readout unit 59 stores the session ID read in step S55, the image data ID generated in step S56, and the image data ID generated in step S54 as new records in the image type management DB 5002 (see FIG. 22). The received IP address of the transmission source terminal and the image type information are associated and stored (step S57). Then, the transmission/reception unit 51 transmits the image data ID generated in step S56 to the video conference terminal 3a. As a result, the transmission/reception unit 31a of the video conference terminal 3a receives the image data ID (step S58).

Next, the storage/reading unit 39a of the video conference terminal 3a stores the IP of the own terminal (video conference terminal 3a), which is the transmission source terminal, stored in the image type management DB 3001a (see FIG. 16) in step S53. The image data ID received in step S58 is stored in association with the address and image type information (step S59).

On the other hand, the transmission/reception unit 51 of the communication management system 5 transmits an additional notification of image type information to another communication terminal (here, the smart phone 9) (step S60). This additional notification of image type information includes the image data ID generated in step S56, and the IP address and image type information of the own terminal (videoconference terminal 3a) that is the transmission source terminal stored in step S53. is Thereby, the transmitting/receiving unit 91 of the smartphone 9 receives the additional notification of the image type information. The transmission destination of the transmission/reception unit 51 is another IP address associated with the same session ID as the IP address of the video conference terminal 3a in the session management DB 5001 (see FIG. 21). That is, the destination is another communication terminal in the same virtual conference room as the video conference terminal 3a.

Next, the storage/readout unit 99 of the smartphone 9 stores the image data ID, the IP address of the transmission source terminal, and the image type received in step S60 as a new record in the image type management DB 9001 (see FIG. 16). The information is associated and stored (step S61). Similarly, the additional communication of the image type information is also transmitted to the video conference terminals 3d and PC7, which are other communication terminals, and stored in the image type management DBs 3001d and 7001 of the video conference terminals 3d and PC7, respectively. As described above, the image type management DBs 3001a, 3001d, 7001, and 9001 can share the same information in each communication terminal.

<Communication processing of photographed image data>
Next, communication processing of photographed image data and material image data in a video call will be described with reference to FIGS. 28 to 36. FIG. FIG. 28 is an image diagram showing the state of a video call using base A as an example.

As shown in FIG. 28, the video conference terminal 3a causes the display 4 to superimpose the material image generated by the PC 6 on the omnidirectional panorama image captured by the imaging device 1a. In addition, on the right side of FIG. 28, a whiteboard W is installed on which the user A1 or the like can write characters, pictures, and the like.

Subsequently, using FIG. 29, the photographed image data, sound data, and document image data obtained at base A shown in FIG. Processing for transmission to the PC 7 and the videoconference terminal 3d) will be described. FIG. 29 is a sequence diagram showing communication processing of photographed image data, sound data and document image data in a video call.

First, the communication unit 18a of the photographing device 1a transmits the captured image data obtained by capturing an object or scenery and the sound data obtained by collecting sound to the communication unit 38a of the video conference terminal 3a (step S101). ). In this case, since the photographing device 1a is a device capable of obtaining two hemispherical images that are the basis of the omnidirectional panorama image, the photographed image data is composed of data of two hemispherical images. Thereby, the communication unit 38a of the video conference terminal 3a receives the captured image data and the sound data.

Next, the communication section 68 of the PC 6 transmits the material image data displayed by the display control section 64 to the communication section 38a of the video conference terminal 3a (step S102).

Next, the transmission/reception unit 31a of the videoconference terminal 3a transmits the photographed image data, sound data, and material image data sent from the photographing device 1a to the communication management system 5 (step S103). As a result, the transmission/reception unit 51 of the communication management system 5 receives the photographed image data, the sound data, and the material image data. Here, the captured image data includes an image data ID for identifying the captured image data to be transmitted.

Next, the transmission/reception unit 51 of the communication management system 5 sends the captured image data, sound data, and The material image data is transmitted (steps S104, S105, S106). In each of these transmissions, an image data ID for identifying the captured image data to be transmitted is included in the transmission of captured image data. As a result, the transmitting/receiving unit 91 of the smartphone 9, the transmitting/receiving unit 71 of the PC 7, and the transmitting/receiving unit 31d of the video conference terminal 3d receive the captured image data and the image data ID, respectively, and further receive the sound data and the document image data.

Next, a display example of the display 917 at the site B will be described with reference to FIG. 30 . FIG. 30 is an example of a display at base D. FIG. Among them, FIG. 30(a) shows the photographed image data sent from the photographing device 1a at the base A via the video conference terminal 3a and the photographed image data sent from the photographing device 1b at the base B. , the case where the omnidirectional panorama image and the predetermined area image are not created and displayed as they are. On the other hand, FIG. 30(b) shows a case where an omnidirectional panorama image and a predetermined area image are created from the captured image data sent from the imaging devices 1a and 1b. An image of base A is displayed in the left display area (layout number "1") of the display 4d, and an image of base B is displayed in the upper right display area (layout number "2"). Further, the image of base C is displayed in the middle display area (layout number "3") on the right side of the display 4d, and the image of base D (own base) is displayed in the lower right display area (layout number "4"). is displayed. The display area with layout number "1" is the main display area, and the display areas with layout numbers "2", "3" and "4" are sub-display areas. The image in the main display area and the image in the sub display area can be changed by each communication terminal. Normally, at each site, an image of the site where the central person of the video call is present is displayed in the main display area.

Here, if the photographed image data sent from the photographing devices 1a and 1b capable of photographing an omnidirectional panorama image is displayed as it is, base A and base B are displayed as shown in FIG. are displayed as a front hemispherical image and a rear hemispherical image as shown in FIGS. 3(a) and 3(b), respectively.

On the other hand, the image/sound processing unit 93 generates a omnidirectional panoramic image from the photographed image data output by the imaging devices 1a and 1b capable of obtaining two hemispherical images that are the basis of the omnidirectional panoramic image. When the predetermined area image is created and the predetermined area image is created, the predetermined area image, which is a plane image, is displayed as shown in FIG. 30(b). At base C, the photographing device 8 that obtains the general image takes the photograph, and at the base D, the video conference terminal 3d that obtains the general image takes the photograph. An image (here, a planar image) is displayed.

Further, the user at each site can change the predetermined area related to the predetermined area image in the same omnidirectional panorama image. For example, the user B1 operates the touch panel 921 so that the reception unit 92 receives movement of the predetermined area image, and the display control unit 94 shifts, rotates, reduces, or enlarges the predetermined area image. can do. As a result, as shown in FIG. 30(b), from the predetermined area image in which some of the users A1 and A2 of the site A are displayed in the initial setting (default), the As shown, the predetermined area image can be displayed. Specifically, FIG. 30(c) shows a state in which the predetermined area image including the users A1 and A2 is changed to the predetermined area image including the whiteboard W in the photographed image of the base A shown in FIG. is shown.

Note that the omnidirectional icons 191 and 192 shown in FIGS. 30(b) and 30(c) are special icons for indicating that they are predetermined area images showing a predetermined area T that is part of the omnidirectional panorama image. It is an example of an image identification icon. Note that the display positions of the omnidirectional icons 191 and 192 may be anywhere such as the upper left, lower left, and lower right, instead of the upper right. Also, the types of omnidirectional icons 191 and 192 are not limited to those shown in FIGS. Further, instead of the omnidirectional icons 191 and 192, characters such as "omnidirectional image" may be used, or a combination of an icon and characters may be used.

Subsequently, referring to FIG. 31, image communication when the predetermined area image is displayed as shown in FIG. 30(b) and when the predetermined area image is changed from FIG. 30(b) to FIG. 30(c) Processing in the system will be explained. FIG. 31 is a sequence diagram showing processing for sharing predetermined area information. In FIG. 31, the video conference terminal 3a at the site A is the third communication terminal, the video conference terminal 3d at the site D is another communication terminal, and the smart phone 9 at the site B is the communication terminal (own terminal).

First, as shown in FIG. 30(b), when the user D1 or the like at the site D uses the video conference terminal 3d to display the predetermined area image of the site A, the transmission/reception unit of the video conference terminal 3d 31d transmits predetermined area information indicating the displayed predetermined area image to the communication management system 5 (step S111). The predetermined area information includes the IP address of the video conference terminal 3a, which is the transmission source of the captured image data, and the IP address of the video conference terminal 3d, which is the transmission destination of the captured image data (the transmission source of the predetermined area information). ing. Thereby, the transmitting/receiving section 51 of the communication management system 5 receives the predetermined area information.

Next, the storage/reading unit 59 of the communication management system 5 associates and stores the predetermined area information received in step S111 and the source and destination IP addresses in the predetermined area management DB 5003 (step S112). The processing of steps S111 and S112 is performed each time the predetermined area image is changed in the video conference terminal 3d, as shown in FIGS. 30(b) to 30(c).

Next, the storage/readout unit 59 of the communication management system 5 selects the latest pair at that time from among the pairs of the predetermined area information and each IP address stored in the predetermined area management DB 5003 (the latest stored pair). A set of predetermined area information and each IP address (which will be a set of data) is read at regular intervals (for example, 30 seconds) (step S113). Then, in step S113, the transmitting/receiving unit 51 transmits the read data to the other communication terminals (video conference terminal 3a, smartphone 9, PC 7) that are performing the same video call as the video conference terminal 3d that is the transmission source of the predetermined area information. Predetermined area information including each IP address issued is distributed (transmitted) (steps S114, S116, S118). As a result, the transmission/reception unit 31a of the video conference terminal 3a receives the predetermined area information. Then, the storage/reading unit 39a stores the predetermined area information received in step S114 and each IP address in association with each other in the predetermined area management DB 3003a (step S115). Similarly, in the smartphone 9, after the transmitting/receiving unit 91 receives the predetermined area information, the storage/reading unit 99 stores the predetermined area information received in step S116 in the predetermined area management DB 9003 while being associated with each IP address. (step S117). Furthermore, in the PC 7, after the transmitting/receiving unit 71 receives the predetermined area information, the storage/reading unit 79 stores the predetermined area management DB 7003 in association with each IP address received in step S118. Store (step S119).

Next, another process for sharing the predetermined area information will be described with reference to FIG. 32 . FIG. 32 is a sequence diagram showing another example of the processing shown in FIG. 31 and showing another processing for sharing the predetermined area information. In FIG. 32, the videoconference terminal 3a at the base A is the communication terminal (own terminal), and the videoconference terminal 3d at the base D is the other communication terminal.

In the process shown in FIG. 31, the communication management system 5 temporarily manages each predetermined area information transmitted from each communication terminal (see S112), and sends each information to each communication terminal other than the transmission source at regular intervals. The predetermined area information has been transmitted (see S114 to S119). On the other hand, in this embodiment, as shown in FIG. Information is temporarily managed (see S213), and each predetermined area information is transmitted to each communication terminal other than the own terminal at regular intervals (see S215 to S221). That is, in the present embodiment, the communication terminal that is the transmission source of the captured image data determines the specific area of the predetermined area T1 in which the captured image data transmitted by its own terminal (here, the video conference terminal 3a) is transmitted by another communication terminal. It manages whether it is displayed as an image.

Note that the processing shown in FIG. 32 is realized by a hardware configuration common to the processing shown in FIG. Therefore, the description of the hardware configuration for realizing the processing of FIG. 32 is omitted.

In the process shown in FIG. 32, first, when the user D1 or the like at the base D uses the video conference terminal 3d to display the predetermined area image of the base A, the transmission/reception unit 31d of the video conference terminal 3d performs communication management. Predetermined area information indicating the displayed predetermined area image is transmitted to the system 5 (step S211). The predetermined area information includes the IP address of the video conference terminal 3a, which is the transmission source of the captured image data, and the IP address of the video conference terminal 3d, which is the transmission destination of the captured image data (the transmission source of the predetermined area information). ing. Thereby, the transmitting/receiving section 51 of the communication management system 5 receives the predetermined area information.

Next, the transmission/reception unit 51 of the communication management system 5 transmits the predetermined area information including each IP address received in step S211 to the video conference terminal 3a, which is the source of the captured image data (step S212). As a result, the transmission/reception unit 31a of the video conference terminal 3a receives the predetermined area information.

Next, the storage/readout unit 39a of the video conference terminal 3a associates and stores the predetermined area information received in step S212 and the IP addresses of the source and destination in the predetermined area management DB 3003a (step S213). The process of step S213 is a process of managing how the photographed image data transmitted by the own terminal (here, the video conference terminal 3a) is displayed on other communication terminals. Note that the processing of steps S211 to S213 is performed each time the predetermined area image is changed in the video conference terminal 3d.

Next, the storage/readout unit 39a of the videoconference terminal 3a selects the latest pair at that point in time among the sets of the predetermined area information and each IP address stored in the predetermined area management DB 3003a (the latest set). The predetermined area information and each IP address that form a set of data are read at regular intervals (for example, 30 seconds) (step S214). Then, the transmitting/receiving section 31a transmits the predetermined area information including each IP address read in step S214 to the communication management system 5 (step S215). Thereby, the transmitting/receiving section 51 of the communication management system 5 receives the predetermined area information.

Next, the transmission/reception unit 51 of the communication management system 5 transmits (distributes) the predetermined area information including each IP address received in step S215 to each communication terminal (video conference terminal 3d, smartphone 9, PC 7) (step S216, S218, S220). As a result, the transmission/reception unit 31d of the video conference terminal 3d receives the predetermined area information. Then, the storage/reading unit 39d stores the predetermined area information received in step S216 in association with each IP address also received in the predetermined area management DB 3003d (step S217). Similarly, in the smartphone 9, after the transmitting/receiving unit 91 receives the predetermined area information, the storage/reading unit 99 stores the predetermined area information received in step S218 in the predetermined area management DB 9003 to each IP address that was similarly received. They are associated and stored (step S219). Furthermore, in the PC 7 as well, after the transmitting/receiving unit 71 receives the predetermined area information, the storage/reading unit 79 stores the predetermined area management DB 7003 in association with each IP address received in step S220. Store (step S221).

As described above, the predetermined area information indicating the predetermined area image changed at the site A is transmitted to each of the communication terminals of the other sites B, C, and D during the same video call. The predetermined area information indicating the image is shared among the other bases B, C, and D as well. This processing is also performed in the same manner when the predetermined area image is changed at sites B, C, and D, and predetermined area information indicating the predetermined area image being displayed at each other site is shared among all sites during the same video call. be done.

Subsequently, the superimposition position setting of the material image will be described using FIGS. 33 to 35. FIG. FIG. 33 is a sequence diagram showing the processing in the image communication system when changing the superimposition position of superimposing the document image on the omnidirectional panorama image at the site A that is the image distribution source.

First, the accepting unit 32a accepts selection of a document image according to an operation by the user A1 or the like of the site A (step S71). As an example of how to make it selected, on the preview screen of the omnidirectional panorama image distributed by user A1, etc. of site A, "right-click", "double-click", "press the corresponding key There is a method to perform operations such as "input". In the case of a touch-capable terminal, the normal image in the preview area may be realized by operations such as "long tap" and "double tap". Next, the reception unit 32a receives a change in the superimposed position of the material image according to the operation of the user A1 or the like (step S72).

Next, the reception unit 32a receives determination of the superimposition position of the material image according to the operation of the user A1 or the like, and the calculation unit 37a generates superimposition position information (step S73). As an example of a method of determination, determination of the superimposition position may be automatically determined after the operation of step S72 is completed, or may be performed by pressing a determination button separately on the screen or performing a predetermined key operation. It can be a method to

Here, an example of a screen on which the user A etc. changes the superimposed position of the material image will be described with reference to FIG. FIG. 34 is a preview screen at base A of the omnidirectional panorama image. The preview screen referred to here is a screen for confirming a video distributed by a certain base at its own base without going through the communication network 100 . As shown in FIG. 34, the preview screen shows a predetermined area image including users A2, A4 and whiteboard W in the omnidirectional panorama image. Characters and pie charts are written on the whiteboard W.

FIG. 34 further shows a region R1 indicated within a dotted line frame and a region R2 indicated within a solid line frame. Regions R1 and R2 respectively indicate the superimposition position before change and the superimposition configuration after change. By changing the superimposed position of the material image to area 2, which is the area within the whiteboard W, the bar graph, which is the material image displayed by the PC 6, can be combined with other characters and pie charts written within the whiteboard W. They can be displayed side by side to make it easier for conference participants to see them.

Using FIG. 34, the operation of the user A1 etc. will be explained in correspondence with the above-described steps 71 to 73. First, the user A1 etc. selects the region R1 (step S71), then the direction of the arrow to change the superimposed position of the material image (step 72). Then, the operation for determining the post-change superimposition position is executed in a state where the object is moved to the desired region R2 (step 73).

Note that the method of changing the superimposition position is not limited to the "drag and drop" method shown in FIG. As another example, the coordinates of the start point and the end point of the superimposed image may be changed by predetermined key operations, or may be changed by pinch-in and pinch-out operations in the case of a device capable of touch input. Also, if the size of the material image before and after the change, that is, the size of the area R1 and the area R2 are different, enlargement/reduction may be performed so as to fit within the size after the change. Also, the dotted frame and arrow indicating the region R1 may or may not be displayed to assist the user's operation.

Continuing the description of FIG. 33, the communication unit 68 then transmits the changed superimposition position information to the transmission/reception unit 51 of the communication system 5 (step S74). The superimposed position information includes the superimposed position coordinates, the IP address of the video conference terminal 3a that is the transmission source of the captured image data, and the video conference terminal 3d that is the transmission destination of the captured image data (the transmission source of the superimposed position information). Contains an IP address. Thereby, the transmission/reception unit 51 of the communication management system 5 receives the superimposed position information.

Next, the transmission/reception unit 51 transmits the superimposed position information to other communication terminals (video conference terminal 3d, smartphone 9, PC 7) that are performing the same video call as the video conference terminal 3d that is the transmission source of the predetermined area information. is distributed (transmitted) (steps S75, S77, S79). As a result, the transmission/reception unit 31d of the video conference terminal 3d receives the superimposition position information. Then, the storage/readout unit 39a stores the superimposed position information received in step S75 in association with the transmission source address in the superimposed position management DB 3004d (step S76). Similarly, in the smartphone 9, after the transmitting/receiving unit 91 receives the superimposed position information, the storage/reading unit 99 stores the superimposed position information received in step S77 in the superimposed position management DB 9004 in association with the source IP address. (step S78). Furthermore, in the PC 7 as well, after the transmitting/receiving unit 71 receives the superimposed position information, the storage/reading unit 79 stores the predetermined area management DB 7004 in association with the predetermined area information received in step S79 with each IP address also received. Store (step S80).

FIG. 35 is a flow chart showing display processing of a document image in the receiving terminal. Since the processing of each receiving terminal is the same, the processing of the smart phone 9 at base B will be described here.

First, in the smartphone 9, the storage/readout unit 99 searches the image type management DB 9001 (see FIG. 16) using the image data ID received in step S106 shown in FIG. Image type information (source name) is read out (step S131).

Next, the determination unit 95 determines whether or not the image type information read out in step S131 indicates a "special image" (step S132). Further, if the result of determination in step 132 is that the image is a special image, the determination unit 95 confirms whether document image data has been received (step S133). For confirmation in step S133, for example, if there is image data that is not managed by image data ID, it is determined that the image data is material image data. If the judgment result in step S133 has not been received, that is, if No, the image/sound processing unit 93 displays the omnidirectional image and terminates (step S137). On the other hand, if the determination result of step 133 has been received, that is, if Yes, the determination unit 95 determines whether the source IP address of the image information is stored in the superimposition position management DB 9004d (step 135). If the determination result of step 135 is not stored, that is, if No, the image/sound processing unit 93 places the material at the initial superimposition position (default superimposition position), which is the predetermined superimposition position of the omnidirectional image. Image data is superimposed (step S136), and an omnidirectional image is displayed (step S137). If the determination result of step 135 is stored, that is, if Yes, the storage/readout unit 99 acquires the superimposed position from the superimposed position management DB 9004d (step S138). Next, the image/sound processing unit 93 superimposes the material image data on the obtained superimposition position on the omnidirectional image (step 139), and displays the omnidirectional image (S137).

FIG. 36 is an example of the display screen of base B on which the display processing described in FIG. 35 is executed. As shown in FIG. 36, at base B, the material image (superimposed image) is displayed in region R2, which is the superimposed position where the superimposed image is changed at base A in FIG.

As described above, the superimposed position information changed at the site A is transmitted to the communication terminals of the other sites B, C, and D during the same video call, so that the material screen being displayed at the site A is displayed at the same position as the other site A. It is also displayed and shared at bases B, C, and D.

<Image quality control of photographed images and document images>
Image quality control of photographed images and document images executed by the communication management system 5 will be described with reference to FIGS. 36 to 39. FIG. FIG. 37 is a sequence diagram showing processing for changing image quality. FIG. 38 is a diagram when the predetermined area image including the material image is changed. FIG. 39 is a diagram showing the changed predetermined area.

First, the calculation unit 97 of the smartphone 9 calculates the display ratio in the display area of the site A with respect to the predetermined area image and the superimposed image (material image) R2 of the site A shown in FIG. 36 (step S151). In FIG. 36, all superimposed images R2 are displayed.

Next, the transmission/reception unit 91 of the smartphone 9 transmits ratio information indicating the ratio calculated by the calculation unit 97 to the communication management system 5 (step S152). At this time, the IP address of the image transmission source communication terminal (here, video conference terminal 3a) and the IP address of the image transmission destination communication terminal (here, smartphone 9) are also transmitted. Thereby, the transmission/reception unit 51 of the communication management system 5 receives the ratio information and each IP address.

Next, in the communication management system 5, the storage/readout unit 59 associates the ratio information and each IP address received by the transmission/reception unit 51, and stores and manages them in the ratio management DB 5004 (step S153).

29, the transmitting/receiving unit 51 of the communication management system 5 receives the photographed image data, sound data, And the material image data is received (step S154).

Next, the storage/reading unit 59 uses the IP address of the image transmission source communication terminal (here, the video conference terminal 3a) and the image transmission destination communication terminal (here, the smartphone 9) as search keys. By searching the ratio management DB 5004, the corresponding ratio information is read, and the changing unit 53 changes the photographed image related to the photographed image data received in step S154 and the document image data according to the ratio indicated by the ratio information. Change the image quality of the document image. In this case, if the ratio indicated by the ratio information is large, the image quality is raised, and if the ratio is small, the image quality is lowered.

Next, the transmission/reception unit 51 transmits the photographed image data, the material image data, and the sound data according to the ratio to the smartphone 9 (step S156). In this case, the image data ID is also attached to the captured image data. Thereby, the transmission/reception unit 91 of the smartphone 9 receives the captured image data, the document image data, and the sound data.

next. In the smartphone, the image/sound processing unit 93 performs image processing to superimpose a superimposed image (document image) related to the document image data on the entire image related to the photographed image data received by the transmission/reception unit 91 (step S157). Then, the display control unit 94 causes the display 917 to display an image as shown in FIG.

By the way, when users B1 and B2 at site B change the predetermined area image of site A as shown in FIG. change. In FIG. 38, the proportion of the superimposed image R2 is smaller than in FIG. FIG. 39 shows the range of the display area T at the site A in the case of FIG. In such a case, the processing from step S151 is started again. This is because the ratio of the superimposed image R2 is small here. The image quality of the photographed image related to the photographed image data transmitted in step S156 is improved, and the image quality of the document image related to the document image data similarly transmitted is decreased.

The users B1 and B2 at the site B can use the touch panel 921 to change the predetermined position on which the superimposed screen R3 is superimposed. In this case, the receiving unit 92 receives the change of the predetermined position, and the image/sound processing unit 93 superimposes the material image on the predetermined position.

<<main effects of the present embodiment>>
As described above, according to this embodiment, the process of steps S151 to S154 solves the problem that the user on the receiving side overlooks the document image that the user on the transmitting side wants to pay attention to. It has the effect of being able to

Further, the communication terminal such as the video conference terminal 3a can create an omnidirectional panorama image and further create a predetermined area image based on the image data ID sent together with the image data and the corresponding image type information. can. As a result, as shown in FIG. 30A, it is possible to prevent the hemispherical image on the front side and the hemispherical image on the rear side from being displayed.

Further, according to the present embodiment, in a conference system or the like in which another video (superimposition source image) is superimposed on a part of a certain video (superimposition destination image) of a receiver and displayed, the sender uses the superimposition destination image. , the display position of the superimposed original video can be changed. Therefore, if the original video is superimposed in a place not intended by the sender, or if you want to superimpose the video in a different position during the meeting, changing the superimposing position makes it easier to see and understand. The effect of being able to make it into a picture is produced.

〔supplement〕
In the above embodiment, as an example of a panoramic image to be superimposed, a photographed image (whole image) has been described as a three-dimensional omnidirectional panoramic image, but it may be a two-dimensional panoramic image.

In the above embodiment, the communication management system 5 relays the predetermined area information transmitted from each communication terminal. good.

Also, each function of the above embodiments can be realized by one or a plurality of processing circuits. Here, the "processing circuit" in this specification means a processor programmed by software to perform each function, such as a processor implemented by an electronic circuit, or a processor designed to perform each function described above. devices such as ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array), SOC (System on a chip), GPU (Graphics Processing Unit) and conventional circuit modules.

1a photographing device 1b photographing device 3a video conference terminal (an example of another communication terminal)
3d video conference terminal (an example of a communication terminal)
4a display 4d display (an example of display means)
5 Communication management system 6 PC
7 PCs
8 Photographing device 9 Smartphone (an example of another communication terminal)
91 transmission/reception unit (an example of transmission means, an example of reception means)
92 reception unit (an example of reception means)
93 Image/sound processing unit (an example of image processing means)
94 Display control unit (an example of display control means)
97 calculation unit (an example of calculation means)
53 Modification part (an example of modification means)
3004a Superposition position management DB (an example of superposition position management means)
3004d Superposition position management DB (an example of superposition position management means)
5004 Ratio management DB (an example of ratio management means)
7004 Superposition position management DB (an example of superposition position management means)
9004 Superposition position management DB (an example of superposition position management means)

JP 2012-178135 A JP 2011-223076 A JP 2012-156820 A

Claims (8)

A communication terminal capable of displaying a predetermined area image, which is an image of a predetermined area in an entire image, on a display means,
receiving means for receiving data of the overall image, data of the superimposed image, and superimposed position information indicating a position where the superimposed image is superimposed on the overall image, which are transmitted from another communication terminal;
image processing means for superimposing the superimposed image on the overall image based on the superimposed position information;
display control means for causing a display means to display a predetermined area image in the entire image;
has
The display control means causes the display means to display the predetermined area image and the superimposed image with image quality changed according to the ratio of the predetermined area image and the superimposed image in the display area of the display means. A communication terminal characterized by: A communication terminal according to claim 1,
receiving means for receiving a change of the predetermined area image displayed on the display means within the entire image;
When the ratio of the predetermined area image and the superimposed image in the display area of the display means is changed due to the change of the predetermined area image, the display control means controls the predetermined area image and the superimposed image on the display means. and causing the display means to display the predetermined area image and the superimposed image with image quality changed in accordance with the changed ratio of . The communication terminal according to claim 1 or 2, further comprising:
a calculation means for calculating a ratio of the predetermined area image and the superimposed image on the display means;
transmitting means for transmitting ratio information indicating the ratio calculated by the calculating means to a communication management system that relays communication with the other communication terminal;
The communication management system, which received the ratio information transmitted by the transmission means, changed the image quality of the whole image and the superimposed image according to the ratio indicated in the ratio information, and transmitted the change. receiving means for receiving the data of the subsequent whole image and the data of the superimposed image after the change;
has
The display control means causes the display means to display the predetermined area image and the superimposed image with image quality changed according to the ratio of the changed predetermined area image and the superimposed image received by the receiving means. A communication terminal characterized by: a communication terminal according to claim 3;
the communication management system;
An image processing system comprising: 4. The communication terminal according to any one of claims 1 to 3, wherein the overall image is an omnidirectional panorama image. 4. The communication terminal according to any one of claims 1 to 3, wherein the communication terminal is a video conference terminal, personal computer, smart phone, digital television, smart watch, or car navigation device. A display method executed by a communication terminal capable of displaying a predetermined area image, which is an image of a predetermined area in an entire image, on display means,
The communication terminal is
a receiving step of receiving data of the overall image, data of the superimposed image, and superimposition position information indicating a position where the superimposed image is superimposed on the overall image, which are transmitted from another communication terminal;
an image processing step of superimposing the superimposed image on the overall image based on the superimposed position information;
a display control step of causing a display means to display a predetermined area image in the entire image;
and run
The display control step performs processing for displaying on the display means the predetermined area image and the superimposed image with image quality changed according to the ratio of the predetermined area image and the superimposed image in the display area of the display means. A display method , comprising: A program for causing a computer to execute the display method according to claim 7.

Images