JP7151316B2 - Communication terminal, image communication system, display method, and program - Google Patents

Description

The present invention relates to communication terminals, image communication systems, display methods, and programs.

2. Description of the Related Art A conferencing system for holding a remote conference with a remote location via a communication network such as the Internet has become widespread. In this conference system, in a conference room where one of the parties, such as attendees, who conducts a remote conference, is present, the communication terminal of the remote conference system is used to shoot and collect images of the conference room and voices such as remarks of the parties of the conference. Then, they are converted into digital data and transmitted to the other party's communication terminal. As a result, images can be displayed on the display of the other party's conference room, and audio can be output from the speakers, allowing video calls to be made, allowing remote conferences to be held in a manner similar to that of an actual conference. (See Patent Document 1).

In addition, the communication terminal is connected to a photographing device capable of acquiring a omnidirectional panorama image in real time, the omnidirectional panoramic image obtained from this photographing device is transmitted to each communication terminal of the other party, and at each communication terminal of the other party , a technique for displaying, on a display or the like, a predetermined area image representing a predetermined area that is a part of the omnidirectional panorama image (see Patent Document 2). As a result, the user at each site can independently determine and display a predetermined area image showing a predetermined area of interest to the user in the entire omnidirectional panorama image.

However, when displaying a predetermined area image independently determined by each site for the same overall image, the user at any site can determine which portion of the predetermined area image of the overall image is displayed at another site. cannot be grasped. Therefore, if a predetermined area image irrelevant to the topic of the meeting, etc. is displayed during the meeting, etc., the user at this site will have a problem that it is difficult to keep up with the topic of the meeting, etc. .

The invention according to claim 1 is a communication terminal capable of displaying a first predetermined area image, which is an image of a first predetermined area in a whole image transmitted from another communication terminal, and changing the first predetermined area. a second predetermined area transmitted from another communication terminal displaying a second predetermined area image, which is an image of a second predetermined area in the overall image, for specifying the second predetermined area; receiving means for receiving predetermined information; and based on the first predetermined information and the second predetermined information for specifying a first predetermined area image that is an image of a first predetermined area in the entire image, calculating means for calculating the position of the second predetermined area with respect to the first predetermined area in the image; and corresponding position information indicating the position based on the position calculated by the calculating means and corresponding position information for the first predetermined area and display control means for displaying at least one of the direction information indicating the direction of the second predetermined area in the first predetermined area image.

As described above, according to the present invention, a user at an arbitrary site can grasp which part of the entire image of the predetermined area image is displayed at another site. Therefore, for example, there is an effect that it becomes easier to keep up with the topic of a meeting or the like compared to the conventional technology.

(a) is a left side view of the imaging device, (b) is a front view of the imaging device, and (c) is a plan view of the imaging device. It is a use image figure of an imaging device. (a) is a hemispherical image (front) captured by the imaging device, (b) is a hemispherical image (back) captured by the imaging device, and (c) is a diagram showing an image represented by the Mercator projection. (a) A conceptual diagram showing a state in which a sphere is covered with a Mercator image, (b) a diagram showing an omnidirectional panorama image. FIG. 4 is a diagram showing the positions of a virtual camera and a predetermined area when the omnidirectional panorama image is a three-dimensional sphere; (a) is a three-dimensional perspective view of FIG. 5, and (b) is a diagram showing a state in which an image of a predetermined area is displayed on the display of the communication terminal. 4 is a diagram showing a relationship between predetermined area information and an image of a predetermined area T; FIG. Fig. 3 shows a point in three-dimensional Euclidean space with spherical coordinates; 1 is a schematic diagram of an image communication system according to an embodiment of the present invention; FIG. It is a hardware block diagram of an imaging device. 2 is a hardware configuration diagram of a video conference terminal; FIG. 2 is a hardware configuration diagram of a communication management system and a PC; FIG. It is a hardware block diagram of a smart phone. 1 is a functional block diagram of part of an image communication system; FIG. 1 is a functional block diagram of part of an image communication system; FIG. 4 is a conceptual diagram showing an image type management table; FIG. FIG. 3 is a conceptual diagram showing an imaging device management table; 4 is a conceptual diagram showing a predetermined area management table; FIG. 4 is a conceptual diagram showing a session management table; FIG. 4 is a conceptual diagram showing an image type management table; FIG. 4 is a conceptual diagram showing a predetermined area management table; FIG. FIG. 10 is a sequence diagram showing a process of joining a specific communication session; FIG. 11 is a diagram showing a screen for selecting a communication session (virtual conference room); FIG. 11 is a sequence diagram showing management processing of image type information; It is an image diagram showing the state of a video call, (a) shows the case where the photographing device 1a is not used, and (b) shows the case where the photographing device 1a is used. FIG. 4 is a sequence diagram showing communication processing of captured image data and sound data in a video call; Shown are display examples of the display at the site B, (a) is displayed as it is without creating an omnidirectional panorama image and a predetermined area image from the image data sent from the imaging devices 1a and 1b. FIG. 27(b) shows a case where an omnidirectional panorama image and a predetermined area image are created from the image data sent from the photographing devices 1a and 1b, and FIG. ) is a diagram showing a case where the predetermined area image of FIG. 27(b) is changed. FIG. 11 is a sequence diagram showing processing for sharing predetermined area information; 10 is a flowchart showing display processing of a predetermined area image; FIG. 10 is a diagram showing a method of deriving the position of the gaze point of another site in the predetermined area image of the own site; (a) A diagram showing the definition of an angle, (b) a diagram showing the definition of an angle range. The image of the main display area is shown, and is a display example of the predetermined area image including the display direction mark. The image of the main display area is shown, and is a display example of the predetermined area image including the gazing point mark. The image of the main display area is shown, and is a display example of the predetermined area image including the gazing point mark and the display direction mark. FIG. 10 is a sequence diagram showing another process of sharing predetermined area information;

An embodiment of the present invention will be described below with reference to the drawings.

[First embodiment]
First, the first embodiment will be described with reference to FIGS. 1 to 34. FIG.

<<Outline of Embodiment>>
<Method for Generating Spherical Panorama Image>
A method of generating an omnidirectional panorama image will be described with reference to FIGS. 1 to 7. FIG.

First, the appearance of the imaging device 1 will be described with reference to FIG. The photographing device 1 is a digital camera for obtaining a photographed image that is the basis of a three-dimensional omnidirectional (360°) panoramic image. 1(a) is a left side view of the photographing device, FIG. 1(b) is a front view of the photographing device, and FIG. 1(c) is a plan view of the photographing device.

As shown in FIG. 1A, the photographing device 1 is of a size that can be held by a person with one hand. Further, as shown in FIGS. 1A, 1B, and 1C, an image sensor 103a is provided on the front side (front side) and an image sensor 103a on the back side (rear side) on the upper portion of the photographing apparatus 1. ) is provided with an image sensor 103b. These imaging elements (image sensors) 103a and 103b are used in combination with optical members (for example, fisheye lenses 102a and 102b in FIG. 10 described later) capable of capturing hemispherical images (angle of view of 180° or more). Further, as shown in FIG. 1B, an operation unit 115 such as a shutter button is provided on the side opposite to the front side of the photographing apparatus 1 .

Next, with reference to FIG. 2, the use condition of the photographing device 1 will be described. In addition, FIG. 2 is a use image figure of an imaging device. As shown in FIG. 2, the photographing device 1 is used by a user, for example, to photograph a subject around the user. In this case, the imaging elements 103a and 103b shown in FIG. 1 capture images of subjects around the user, respectively, so that two hemispherical images can be obtained.

Next, with reference to FIGS. 3 and 4, the outline of the process until the omnidirectional panorama image is created from the image captured by the imaging device 1 will be described. In addition, FIG. 3(a) is a hemispherical image (front side) photographed by the photographing device, FIG. 3(b) is a hemispherical image (rear side) photographed by the photographing device, and FIG. 3(c) is represented by the Mercator projection. 1 is a diagram showing an image (hereinafter referred to as a "Mercator image"). FIG. 4(a) is a conceptual diagram showing a state in which a sphere is covered with a Mercator image, and FIG. 4(b) is a diagram showing a omnidirectional panorama image.

As shown in FIG. 3A, the image obtained by the imaging device 103a is a hemispherical image (front side) curved by a fisheye lens 102a, which will be described later. Also, as shown in FIG. 3B, the image obtained by the imaging device 103b becomes a curved hemispherical image (rear side) by a fisheye lens 102b, which will be described later. Then, the hemispherical image (front side) and the 180-degree inverted hemispherical image (rear side) are combined by the photographing device 1 to create a Mercator image as shown in FIG. 3(c).

Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is pasted so as to cover the spherical surface as shown in FIG. A spherical panoramic image as shown is created. Thus, the omnidirectional panorama image is represented as an image in which the Mercator image faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the omnidirectional panorama image may be a still image or a moving image.

As described above, since the omnidirectional panoramic image is an image pasted so as to cover the spherical surface, it gives a sense of incongruity when viewed by humans. Therefore, by displaying a predetermined area (hereinafter referred to as a "predetermined area image") of a part of the omnidirectional panorama image as a flat image with little curvature, it is possible to display the image without causing discomfort to the human. This will be described with reference to FIGS. 5 and 6. FIG.

FIG. 5 is a diagram showing positions of a virtual camera and a predetermined area when the omnidirectional image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of the user viewing the omnidirectional image CE displayed as a three-dimensional sphere. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a predetermined area image displayed on the display. Also, in FIG. 6A, the omnidirectional image CE shown in FIG. 4 is represented by a three-dimensional solid sphere CS. Assuming that the omnidirectional image CE generated in this way is the stereosphere CS, the virtual camera IC is positioned inside the omnidirectional image CE as shown in FIG. A predetermined region T in the omnidirectional image CE is a photographing region of the virtual camera IC, and is specified by predetermined region information indicating the photographing direction and angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE. be.

Then, the predetermined area image Q shown in FIG. 6(a) is displayed as an image of the imaging area of the virtual camera IC on a predetermined display, as shown in FIG. 6(b). The image shown in FIG. 6B is a predetermined area image represented by default predetermined area information. Note that the photographing area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, may be used instead of the predetermined area information and the position coordinates of the virtual camera IC. In the following, the shooting direction (rH, rV) and the angle of view (α) of the virtual camera IC will be used.

The relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. FIG. 7 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. As shown in FIG. 7, rH is the Horizontal Radian, rV is the Vertical Radian, and α is the angle of view. That is, the orientation of the virtual camera IC is changed so that the point of gaze of the virtual camera IC indicated by the photographing direction (rH, rV) becomes the central point CP of the predetermined area T which is the photographing area of the virtual camera IC. Become. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP. L is the distance between an arbitrary vertex of the predetermined area T and the center point CP (2L is a diagonal line). In FIG. 7, a trigonometric function generally represented by the following (Equation 1) holds.

Lf=tan(α/2) (Formula 1)
FIG. 8 is a diagram showing points in three-dimensional Euclidean space in spherical coordinates. Let (r, θ, φ) be the position coordinates when the center point CP is expressed in a spherical polar coordinate system. (r, θ, φ) are the radial, polar and azimuthal angles, respectively. The radius r is equal to f because it is the distance from the origin of the three-dimensional virtual space containing the omnidirectional panoramic image to the center point CP. FIG. 8 is a diagram showing these relationships. Hereinafter, the position coordinates (r, θ, φ) of the virtual camera IC will be used.

<Overview of image communication system>
Next, with reference to FIG. 9, an outline of the configuration of the image communication system according to this embodiment will be described. FIG. 9 is a schematic diagram of the configuration of the image communication system of this embodiment.

As shown in FIG. 9, the image communication system of this embodiment includes photographing devices 1a and 1b, video conference terminals 3a and 3b, displays 4a and 4b, a communication management system 5, a PC (Personal Computer) 7, a photographing It is composed of a device 8 and a smartphone 9, and can communicate via a communication network 100 such as the Internet. The connection form of the communication network 100 may be either wireless or wired.

Of these, the photographing devices 1a and 1b are special digital cameras for photographing subjects, landscapes, etc., and obtaining two hemispherical images that are the basis of an omnidirectional panorama image, as described above. On the other hand, the photographing device 8 is a general digital camera for photographing subjects, landscapes, etc. to obtain general planar images.

The video conference terminals 3a and 3b are terminals dedicated to video conferences, and display images of video calls on displays 4a and 4b, respectively, via wired cables such as USB (Universal Serial Bus) cables. The videoconference terminal 3a normally takes pictures with a camera 312 shown in FIG. 11, which will be described later. We can obtain two hemispherical images such that When a wired cable is used, the cradle 2a not only performs communication between the imaging device 1a and the video conference terminal 3a, but also serves to supply power to the imaging device 1a and support the imaging device 1a. Here, the imaging device 1a, the cradle 2a, the video conference terminal 3a, and the display 4a are located at the same base A. At base A, four users A1, A2, A3, and A4 participate in a video call. On the other hand, the video conference terminal 3d and the display 4d are located at the same base D. Also, at base D, three users D1, D2, and D3 are participating in a video call.

The communication management system 5 manages and controls communications of the video conference terminals 3a and 3b, the PC 7, and the smartphone 9, and manages the types of image data transmitted and received (general image and special image type). Therefore, the communication management system is also a communication control system. Here, the special image is a spherical panorama image. The communication management system 5 is installed in a service company or the like that provides video communication services. Further, the communication management system 5 may be constructed by a single computer, or may be constructed by a plurality of computers in which each part (function, means, or storage part) is divided and arbitrarily assigned.

The PC 7 becomes capable of video communication by attaching the photographing device 8 . It should be noted that the PC 7 and the imaging device 8 are located at the base C, which is the same base. In addition, at base C, one user C participates in a video call.

The smartphone 9 displays an image of the video call on a display 917 (described later) provided on the smartphone 9 . The smartphone 9 normally captures images with a CMOS (Complementary Metal Oxide Semiconductor) sensor 905 or the like, which is provided in the device itself. It is possible to acquire two hemispherical image data that are the basis of the omnidirectional panorama image obtained by the photographing device 1b. When wireless communication technology is used, the cradle 2b serves only to supply power to the photographing device 1b and to support the photographing device 1b. Here, the imaging device 1b, the cradle 2b, and the smart phone 9 are located at the same base B, which is the same base. At base B, two users B1 and B2 are participating in a video call.

Also, the video conference terminals 3a and 3d, the PC 7, and the smart phone 9 are examples of communication terminals. OpenGL ES is installed on each communication terminal, and it is possible to create predetermined area information that indicates a part of the spherical panoramic image, and to display panoramic panoramic images sent from other communication terminals. A predetermined area image can be created from the

Note that the arrangement of each terminal (communication terminal, display, photographing device), devices, and users shown in FIG. 9 is an example, and other examples may be used. For example, at the base C, instead of the imaging device 8, the user may use an imaging device capable of capturing an omnidirectional panorama image. Communication terminals also include digital televisions, smart watches, car navigation devices, and the like. Further, hereinafter, when an arbitrary image capturing device out of the image capturing devices 1a and 1b is indicated, it will be referred to as "image capturing device 1". Also, when representing an arbitrary video conference terminal among the video conference terminals 3a and 3b, it is expressed as "video conference terminal 3". Furthermore, when representing an arbitrary display out of the displays 4a and 4b, it is represented as "display 4".

<<Hardware configuration of the embodiment>>
Next, the hardware configurations of the photographing device 1, the video conference terminal 3, the communication management system 5, the PC 7, and the smart phone 9 of this embodiment will be described in detail with reference to FIGS. 10 to 13. FIG. In addition, since the imaging device 8 is a general camera, detailed description thereof will be omitted.

<Hardware configuration of imaging device 1>
First, the hardware configuration of the photographing device 1 will be described with reference to FIG. FIG. 10 is a hardware configuration diagram of the imaging device 1. As shown in FIG. In the following description, the photographing apparatus 1 is assumed to be an omnidirectional (omnidirectional) photographing apparatus using two imaging elements, but the number of imaging elements may be two or more. Further, the device does not necessarily need to be dedicated to omnidirectional photography, and may have substantially the same functions as the imaging device 1 by attaching an omnidirectional photography unit to an ordinary digital camera, smartphone, or the like. .

As shown in FIG. 10, the imaging apparatus 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory). ) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network I/F 116, a communication unit 117, and an antenna 117a.

Among them, the imaging unit 101 includes wide-angle lenses (so-called fish-eye lenses) 102a and 102b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging units provided corresponding to each wide-angle lens. It has elements 103a and 103b. The imaging elements 103a and 103b are image sensors such as CMOS sensors and CCD (Charge Coupled Device) sensors that convert optical images by the fisheye lenses 102a and 102b into image data of electric signals and output them, and horizontal or vertical synchronization signals of the image sensors. , a timing generation circuit that generates a pixel clock, etc., and a group of registers in which various commands and parameters required for the operation of the image sensor are set.

The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I/F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are connected separately from the imaging control unit 105 by a serial I/F bus (I2C bus or the like). The image processing unit 104 and imaging control unit 105 are connected to the CPU 111 via the bus 110 . Furthermore, ROM 112, SRAM 113, DRAM 114, operation unit 115, network I/F 116, communication unit 117, electronic compass 118, and the like are also connected to bus 110. FIG.

The image processing unit 104 fetches the image data output from the imaging elements 103a and 103b through the parallel I/F bus, performs predetermined processing on each image data, and synthesizes these image data. , to create Mercator image data as shown in FIG. 3(c).

The imaging control unit 105 generally uses the I2C bus with the imaging control unit 105 as a master device and the imaging elements 103a and 103b as slave devices to set commands and the like in registers of the imaging elements 103a and 103b. Necessary commands and the like are received from the CPU 111 . The imaging control unit 105 also uses the I2C bus to capture status data and the like of the registers of the imaging elements 103 a and 103 b and sends them to the CPU 111 .

Further, the imaging control unit 105 instructs the imaging devices 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Depending on the photographing device 1, the display (for example, the display of the video conference terminal 3a) may have a preview display function or a function corresponding to moving image display. In this case, the image data is output continuously from the imaging devices 103a and 103b at a predetermined frame rate (frames/minute).

In addition, as will be described later, the imaging control unit 105 also functions as synchronization control means for synchronizing the output timing of the image data of the imaging devices 103a and 103b in cooperation with the CPU 111. FIG. Note that in the present embodiment, the imaging device 1 is not provided with a display unit (display), but a display unit may be provided.

The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in sound data output from the microphone 108 through the I/F bus and performs predetermined processing on the sound data.

The CPU 111 controls the overall operation of the photographing device 1 and executes necessary processing. ROM 112 stores various programs for CPU 111 . The SRAM 113 and DRAM 114 are work memories, and store programs to be executed by the CPU 111, data during processing, and the like. In particular, the DRAM 114 stores image data being processed in the image processing unit 104 and processed Mercator image data.

The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, and the like. A user inputs various shooting modes, shooting conditions, and the like by operating operation buttons.

The network I/F 116 is a general term for interface circuits (such as USB I/F) with external media such as SD cards and personal computers. Also, the network I/F 116 may be wireless or wired. Mercator image data stored in the DRAM 114 is recorded on an external medium via the network I/F 116, or is transmitted to an external device such as the video conference terminal 3a via the network I/F 116 as necessary. or

The communication unit 117 communicates with an external device such as the video conference terminal 3a via an antenna 117a provided in the imaging device 1, using short-range wireless technology such as WiFi or NFC (Near Field Communication). This communication unit 117 can also transmit Mercator image data to an external device of the video conference terminal 3a.

The electronic compass 118 calculates the azimuth and tilt (Roll rotation angle) of the imaging device 1 from the magnetism of the earth, and outputs the azimuth/tilt information. This azimuth/tilt information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of a captured image. The related information also includes data such as the date and time when the image was taken and the data volume of the image data.

<Hardware configuration of the video conference terminal>
Next, the hardware configuration of the video conference terminal 3 will be described with reference to FIG. FIG. 11 is a hardware configuration diagram of a video conference terminal. As shown in FIG. 11, the video conference terminal 3 includes a CPU 301, a ROM 302, a RAM 303, a flash memory 304, an SSD 305, a media I/F 307, an operation button 308, a power switch 309, a bus line 310, a network I/F 311, Equipped with a camera 312, an image sensor I/F 313, a microphone 314, a speaker 315, a sound input/output I/F 316, a display I/F 317, an external device connection I/F 318, a short-range communication circuit 319, and an antenna 319a of the short-range communication circuit 319. ing.

Among these, the CPU 301 controls the operation of the video conference terminal 3 as a whole. A ROM 302 stores a program such as an IPL (Initial Program Loader) that is used to drive the CPU 301 . A RAM 303 is used as a work area for the CPU 301 . The flash memory 304 stores various data such as communication programs, image data, and sound data. An SSD (Solid State Drive) 305 controls reading or writing of various data to/from a flash memory 304 under the control of the CPU 301 . Note that an HDD may be used in place of the SSD. A media I/F 307 controls reading or writing (storage) of data to a recording medium 306 such as a flash memory. The operation button 308 is a button that is operated when selecting a destination of the video conference terminal 3 or the like. A power switch 309 is a switch for switching ON/OFF of the power of the video conference terminal 3 .

A network I/F 311 is an interface for data communication using the communication network 100 such as the Internet. The camera 312 is a type of built-in image capturing means that captures an image of a subject under the control of the CPU 301 and obtains image data. An imaging device I/F 313 is a circuit that controls driving of the camera 312 . The microphone 314 is a type of built-in sound collecting means for inputting sound. A sound input/output I/F 316 is a circuit that processes input/output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301 . A display I/F 317 is a circuit that transmits image data to the external display 4 under the control of the CPU 301 . The external device connection I/F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit such as NFC (registered trademark) and Bluetooth (registered trademark).

A bus line 310 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 301 shown in FIG.

The display 4 is a kind of display means configured by liquid crystal or organic EL (Electro Luminescence) for displaying an image of a subject, icons for operation, and the like. Also, the display 4 is connected to the display I/F 317 by a cable 4c. This cable 4c may be a cable for analog RGB (VGA) signals, a cable for component video, HDMI (High-Definition Multimedia Interface) (registered trademark) or DVI (Digital Video Interface). Interactive) signal cable.

Note that the camera 312 includes a lens and a solid-state imaging device that converts light into electric charge and digitizes an image (video) of a subject. As the solid-state imaging device, a CMOS sensor, a CCD sensor, or the like is used. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I/F 318 via a USB (Universal Serial Bus) cable or the like. When an external camera is connected, the external camera is driven according to the control of the CPU 301 prior to the built-in camera 312 . Similarly, when an external microphone is connected or when an external speaker is connected, according to the control of the CPU 301, the external microphone and speaker are prioritized over the built-in microphone 314 and built-in speaker 315, respectively. Powered by an external speaker.

Also, the recording medium 306 is detachably attached to the video conference terminal 3 . In addition, as long as it is a non-volatile memory from which data is read or written under the control of the CPU 301, not only the flash memory 304 but also an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used.

<Communication management system, PC hardware configuration>
Next, hardware configurations of the communication management system 5 and the PC 7 will be described with reference to FIG. FIG. 12 is a hardware configuration diagram of the communication management system and PC. Since the communication management system 5 and the PC 7 are both computers and have the same configuration, the configuration of the communication management system 5 will be described below, and the description of the configuration of the PC 7 will be omitted.

The communication management system 5 includes a CPU 501 that controls the overall operation of the communication management system 5, a ROM 502 that stores programs used to drive the CPU 501 such as IPL, a RAM 503 that is used as a work area for the CPU 501, and programs for the communication management system 5. HD 504 for storing various data such as; HDD (Hard Disk Drive) 505 for controlling reading or writing of various data to the HD 504 under the control of the CPU 501; A media drive 507 to be controlled, a display 508 for displaying various information such as cursors, menus, windows, characters or images, a network I/F 509 for data communication using the communication network 100, characters, numerical values, various instructions, etc. , a mouse 512 for selecting and executing various instructions, selecting an object to be processed, moving a cursor, etc., and a CD-RW (Compact Disc) as an example of a removable recording medium. -ReWritable) 513, a CD-RW drive 514 for controlling reading of various data, and bus lines such as an address bus and a data bus for electrically connecting the above components as shown in FIG. 510 is provided.

<Hardware configuration of smartphone>
Next, the hardware of the smart phone will be described with reference to FIG. 13 . FIG. 13 is a hardware configuration diagram of a smartphone. As shown in FIG. 13 , the smartphone 9 includes a CPU 901 , ROM 902 , RAM 903 , EEPROM 904 , CMOS sensor 905 , acceleration/direction sensor 906 , media I/F 908 and GPS receiver 909 .

Among these, the CPU 901 controls the operation of the smartphone 9 as a whole. The ROM 902 stores programs such as IPL, which are used to drive the CPU 901 . A RAM 903 is used as a work area for the CPU 901 . The EEPROM 904 reads or writes various data such as smartphone programs under the control of the CPU 901 . A CMOS sensor 905 captures an image of a subject (mainly a self-portrait) and obtains image data under the control of the CPU 901 . The acceleration/direction sensor 906 is various sensors such as an electronic magnetic compass, a gyro compass, and an acceleration sensor for detecting geomagnetism. A media I/F 908 controls reading or writing (storage) of data to a recording medium 907 such as a flash memory. A GPS receiver 909 receives GPS signals from GPS satellites.

In addition, the smartphone 9 includes a long-distance communication circuit 911, a camera 912, an image sensor I/F 913, a microphone 914, a speaker 915, a sound input/output I/F 916, a display 917, an external device connection I/F 918, a short-distance communication circuit 919, An antenna 919 a of a short-range communication circuit 919 and a touch panel 921 are provided.

Among these, the telecommunications circuit 911 is a circuit that communicates with other devices via the communication network 100 . The camera 912 is a type of built-in imaging means for capturing an image of a subject and obtaining image data under the control of the CPU 901 . An imaging device I/F 913 is a circuit that controls driving of the camera 912 . The microphone 914 is a type of built-in sound collecting means for inputting sound. A sound input/output I/F 916 is a circuit that processes input/output of sound signals between the microphone 914 and the speaker 915 under the control of the CPU 901 . A display 917 is a kind of display means such as a liquid crystal or an organic EL that displays an image of a subject, various icons, and the like. The external device connection I/F 918 is an interface for connecting various external devices. A short-range communication circuit 919 is a communication circuit such as NFC or Bluetooth. The touch panel 921 is a type of input means for operating the smartphone 9 by the user pressing the display 917 .

The smartphone 9 also includes a bus line 910 . A bus line 910 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 901 .

Recording media such as CD-ROMs storing the above programs and HDs storing these programs can both be provided domestically or internationally as program products.

<<Function Configuration of Embodiment>>
Next, the functional configuration of this embodiment will be described with reference to FIGS. 14 to 20. FIG. 14 and 15 are functional block diagrams of part of the image communication system.

<Functional configuration of imaging device 1a>
As shown in FIG. 14, the photographing device 1a has a reception section 12a, an imaging section 13a, a sound collection section 14a, a communication section 18a, and a storage/readout section 19a. Each of these units is a function or means implemented by any one of the components shown in FIG. is.

The photographing apparatus 1 also has a storage unit 1000a constructed by the ROM 112, SRAM 113, and DRAM 114 shown in FIG. The storage unit 1000a stores a GUID (Globally Unique Identifier) of its own device.

Note that the imaging device 1b has a reception unit 12b, an imaging unit 13b, a sound collection unit 14b, a communication unit 18b, a storage/readout unit 19b, and a storage unit 1000b. , the imaging unit 13a, the sound collecting unit 14a, the communication unit 18a, the storage/reading unit 19a, and the storage unit 1000a.

(Each functional configuration of the imaging device 1a)
Next, with reference to FIGS. 10 and 14, each functional configuration of the photographing device 1a will be described in further detail.

The reception unit 12a of the photographing device 1 is realized mainly by processing of the operation unit 115 and the CPU 111 shown in FIG. 10, and receives operation input from the user.

The imaging unit 13a is realized mainly by the processing of the imaging unit 101, the image processing unit 104, the imaging control unit 105, and the CPU 111 shown in FIG.

The sound collecting unit 14a is realized by the processing of 108, the sound processing unit 109, and the CPU 111 shown in FIG. 10, and collects sounds around the imaging device 1a.

The communication unit 18a is mainly realized by the processing of the CPU 111, and can communicate with the communication unit 38a of the video conference terminal 3a by short-range wireless communication technology such as the NFC standard, BlueTooth, and WiFi.

The storage/readout unit 19a is mainly realized by the processing of the CPU 111 shown in FIG. read out.

<Functional Configuration of Video Conference Terminal 3a>
As shown in FIG. 14, the videoconference terminal 3a includes a transmission/reception unit 31a, a reception unit 32a, an image/sound processing unit 33a, a display control unit 34a, a determination unit 35a, a creation unit 36a, a calculation unit 37a, a communication unit 38a, and a storage/readout unit 39a. Each of these units is realized by any one of the components shown in FIG. It is a function or a means.

The videoconference terminal 3a also has a storage unit 3000a constructed by the ROM 302, RAM 303, and flash memory 304 shown in FIG. An image type management DB 3001a, an imaging device management DB 3002a, and a predetermined area management DB 3003a are constructed in the storage unit 3000a. Of these, the image type management DB 3001a is composed of the image type management table shown in FIG. The imaging device management DB 3002a is composed of the imaging device management table shown in FIG. The predetermined area management DB 3003a is composed of a predetermined area management table shown in FIG.

The video conference terminal 3d includes a transmission/reception unit 31d, a reception unit 32d, an image/sound processing unit 33d, a display control unit 34d, a determination unit 35d, a creation unit 36d, a calculation unit 37d, a communication unit 38d, a storage/readout unit 39d, and a storage unit 3000d, which respectively include a transmission/reception unit 31a, a reception unit 32a, an image/sound processing unit 33a, a display control unit 34a, a judgment unit 35a, a creation unit 36a, a calculation unit 37a, and a storage unit 3000d. Since the same functions as those of the communication unit 38a, the storage/readout unit 39a, and the storage unit 3000a are realized, descriptions thereof will be omitted. An image type management DB 3001d, an imaging device management DB 3002d, and a predetermined area management DB 3003d are constructed in the storage unit 3000d of the video conference terminal 3d. and the predetermined area management DB 3003a, the description thereof will be omitted.

(Image type management table)
FIG. 16 is a conceptual diagram showing an image type management table. In this image type management table, an image data ID, an IP address as an example of a destination of a transmission source terminal, and a source name are stored and managed in association with each other. Among these, the image data ID is an example of image data identification information for identifying image data when performing video communication. The same image data ID is added to the image data transmitted from the same source terminal. As a result, the destination terminal (communication terminal on the receiving side) can identify the source terminal of the received image data. The IP address of the transmission source terminal indicates the IP address of the communication terminal that transmits the image data indicated by the associated image data ID. A source name is a name for specifying an imaging device that outputs image data indicated by an associated image data ID, and is an example of image type information. This source name is a name created by each communication terminal such as the video conference terminal 3a according to a predetermined naming rule.

For example, four communication terminals whose IP addresses are "1.2.1.3", "1.2.2.3", "1.3.1.3", and "1.3.2.3" are transmitting image data indicated by image data IDs "RS001", "RS002", "RS003", and "RS004", respectively. Further, the types of images indicated by the source names of the respective communication terminals are "Video_Theta", "Video_Theta", "Video", and "Video", and these image types are "special image", "special image", It indicates that it is a "general image" or a "general image". Note that the special image is a spherical panorama image here.

Data other than image data may also be managed in association with the image data ID. Data other than image data are, for example, sound data and document data for screen sharing. Data other than image data may also be managed in association with the image data ID. Data other than image data are, for example, sound data and document data for screen sharing.

(Imaging device management table)
FIG. 17 is a conceptual diagram showing an imaging device management table. In this photographing device management table, vendor IDs and product IDs of the GUIDs of photographing devices that can obtain two hemispherical images that are the basis of the omnidirectional panorama image are stored and managed. As the GUID, for example, the vendor ID (VID) and product ID (PID) used in the USB device can be used. The vendor ID and product ID are stored when the communication terminal such as the video conference terminal is shipped from the factory, but they may be additionally stored after the shipment from the factory.

(predetermined area management table)
FIG. 18 is a conceptual diagram showing a predetermined area management table. In this predetermined area management table, the IP address of the communication terminal that is the transmission source of the captured image data, the IP address of the communication terminal that is the transmission destination of the captured image data, and the predetermined area being displayed on the communication terminal that is the transmission destination of the captured image data. Predetermined area information indicating an image is associated and stored and managed. The communication terminal that is the transmission destination of the captured image data is also the communication terminal that is the transmission source of the predetermined area information. The predetermined area information is, as shown in FIGS. 6 and 7, a transformation parameter for transforming a captured image into an image of a predetermined area T (predetermined area image) in the captured image. Note that the IP address is an example of destination information, and the destination information includes a MAC (Media Access Control) address, a terminal ID (Identification) for specifying a communication terminal, and the like. Also, here, the IP address is represented by simplifying the IPv4 address. The IP address may be IPv6.

For example, the 1st to 3rd rows of the predetermined area management table in FIG. The data is transferred via the communication management system 5 to the video conference terminal 3d with the IP address "1.2.2.3", the PC 7 with the IP address "1.3.1.3", and the IP address "1". .3.2.3” is managed. Furthermore, it is managed that the video conference terminal 3d is the communication terminal that has transmitted the predetermined area information (r=10, θ=20, φ=30). Similarly, it is managed that the PC 7 is the communication terminal that has transmitted the predetermined area information (r=10, θ=30, φ=40). Further, it is managed that the smartphone 9 is the communication terminal that has transmitted the predetermined area information (r=30, θ=40, φ=50).

Further, the transmitting/receiving unit 31a newly updates the predetermined area information including the IP address of the same group as the already managed IP address of the transmission source communication terminal of the captured image data and the IP address of the transmission destination communication terminal of the captured image data. When the information is received at 10:00 a.m., the storage/reading unit 39a rewrites the predetermined area information already managed by the newly received predetermined area information.

(Each functional configuration of the video conference terminal 3a)
Next, each functional configuration of the video conference terminal 3a will be described in more detail with reference to FIGS. 11 and 14. FIG.

The transmission/reception unit 31a of the videoconference terminal 3a is mainly realized by processing of the network I/F 311 and the CPU 301 shown in FIG. send and receive

The reception unit 32a is realized mainly by processing by the operation buttons 308 and the CPU 301, and receives various selections or inputs from the user. In addition to the operation buttons 308, a touch panel or the like may be used as other input means.

The image/sound processing unit 33a is implemented by a command from the CPU 301 shown in FIG. 11, and performs image processing on image data obtained by imaging a subject with the camera 312. FIG. Further, after the user's voice is converted into an audio signal by the microphone 314, the image/sound processing unit 33a performs audio processing on the audio data related to this audio signal.

Further, the image/sound processing unit 33a performs image processing on image data received from another communication terminal based on image type information such as a source name so that the display control unit 34 can display an image on the display 4. do Specifically, when the image type information indicates that the image is a special image, the image/sound processing unit 33a processes the image data (for example, each of the images shown in FIGS. 3(a) and 3(b)). Hemispherical image data) is converted into omnidirectional panorama image data as shown in FIG. Create a given area image as shown. In addition, the image/sound processing unit 33a outputs to the speaker 315 an audio signal related to sound data received from another communication terminal via the communication management system 5, and causes the speaker 315 to output sound.

The display control unit 34a is mainly realized by processing of the display I/F 317 and the CPU 301, and performs control for displaying various images, characters, etc. on the display 4. FIG.

The determination unit 35a is mainly implemented by the processing of the CPU 301, and determines, for example, the image type related to the image data received from the imaging device 1a.

The creating unit 36a is mainly implemented by the processing of the CPU 301, and based on the result of the determination by the determination unit 35a that the image is a general image or a special image (here, a spherical panorama image), determines the image type according to the above naming rule. Create a source name, which is an example of information. For example, when the determination unit 35a determines that the image is a general image, the creation unit 36a creates a source name "Video" indicating that it is a general image. On the other hand, when the determination unit 35a determines that the image is a special image, the creation unit 36a creates a source name "Video_Theta" indicating that it is a special image.

The calculation unit 37a is mainly implemented by the processing of the CPU 301, and calculates the captured image based on the predetermined area information (i2) indicating the predetermined area T2 and the predetermined area information (i1) received from the other communication terminal by the transmission/reception unit 31a. , the position (positional information) of the predetermined area T1 with respect to the predetermined area T2 is calculated. The predetermined area information (i1) indicates the predetermined area T1 in the captured image. In addition, an image when the entire photographed image is displayed is also referred to as a “whole image”.

The communication unit 38a is mainly implemented by the short-range communication circuit 319, the antenna 318a, and the processing of the CPU 301, and can communicate with the communication unit 18a of the imaging device 1a by short-range wireless technology such as NFC, BlueTooth, WiFi, or the like. can. Although the communication unit 38a and the transmission/reception unit 31a have separate communication units, they may be shared.

The storage/readout unit 39a is mainly implemented by the processing of the CPU 301 shown in FIG. read out.

<Functional configuration of communication management system>
Next, each functional configuration of the communication management system 5 will be described in detail with reference to FIGS. 12 and 15. FIG. The communication management system 5 has a transmission/reception section 51 , a judgment section 55 , a generation section 56 and a storage/read section 59 . 12 operates according to commands from the CPU 501 according to the program for the communication management system 5 developed on the RAM 503 from the HD 504. It is a means.

The communication management system 5 also has a storage unit 5000 constructed by the RAM 503 and the HD 504 shown in FIG. A session management DB 5001 , an image type management DB 5002 , and a predetermined area management DB 5003 are constructed in the storage unit 5000 . Of these, the session management DB 5001 is composed of the session management table shown in FIG. The image type management DB 5002 is composed of an image type management table shown in FIG. The predetermined area management DB 5003 is composed of a predetermined area management table shown in FIG.

(session management table)
FIG. 19 is a conceptual diagram showing a session management table. In this session management table, session IDs and IP addresses of participating communication terminals are stored and managed in association with each other. Among these, the session ID is an example of session identification information for identifying a communication session that implements a video call, and is generated for each virtual conference room. The session ID is also managed by each communication terminal such as the video conference terminal 3a, and is used when selecting a communication session in each communication terminal. The IP address of the participating communication terminal indicates the IP address of the communication terminal that has participated in the virtual conference room indicated by the associated session ID.

(Image type management table)
FIG. 20 is a conceptual diagram showing an image type management table. In the image type management table shown in FIG. 20, in addition to each information managed in the image type management table shown in FIG. 16, the same session IDs as those managed in the session management table are associated. managed by Here, the virtual conference rooms indicated by the same session ID "se101" have IP addresses "1.2.1.3", "1.2.2.3", and "1.3.1. 3" are shown participating. In the communication management system 5, the same information as the image data ID, the IP address of the transmission source terminal, and the image type information managed by the communication terminal such as the video conference terminal 3a is managed by the new communication terminal. This is for transmitting image type information and the like to a communication terminal that is already engaged in a video call and a communication terminal that newly participates, such as when entering a virtual conference room. As a result, there is no need to transmit and receive image type information and the like between a communication terminal that is already engaged in a video call and a communication terminal that has newly joined.

(predetermined area management table)
FIG. 21 is a conceptual diagram showing a predetermined area management table. This predetermined area management table basically has the same data structure as the predetermined area management table shown in FIG. However, as will be described later, the transmitting/receiving unit 51 transmits the latest predetermined area information to each communication terminal at regular intervals (for example, 30 seconds). , all the predetermined area information received by the transmitting/receiving unit 51 are stored without being deleted. In FIG. 21, newer predetermined area information is managed at a higher level.

(Each functional configuration of the communication management system)
Next, each functional configuration of the communication management system 5 will be described in detail with reference to FIGS. 12 and 15. FIG.

The transmission/reception unit 51 of the communication management system 5 is mainly realized by processing of the network I/F 509 and CPU 501 shown in FIG. Send and receive (or information).

The determination unit 55 is mainly implemented by the processing of the CPU 501 and makes various determinations.

The generation unit 56 is mainly realized by processing of the CPU 501 and generates an image data ID.

The storage/readout unit 59 is mainly realized by processing of the HDD 505 and the CPU 501 shown in FIG. information).

<PC functional configuration>
Next, the functional configuration of the PC 7 will be described in detail with reference to FIGS. 12 and 14. FIG. The PC 7 basically has the same functions as the video conference terminal 3a. That is, as shown in FIG. 14, the PC 7 includes a transmission/reception unit 71, a reception unit 72, an image/sound processing unit 73, a display control unit 74, a determination unit 75, a creation unit 76, a calculation unit 77, and a communication unit 78. , and a storage/readout unit 79 . Each of these units is a function or means realized by any one of the components shown in FIG. .

The PC 7 also has a storage unit 7000 constructed by the ROM 502, RAM 503, and HD 504 shown in FIG. In this storage unit 7000, an image type management DB 7001, an imaging device management DB 7002, and a predetermined area management DB 7003 are constructed. Note that the image type management DB 7001, the imaging device management DB 7002, and the predetermined area management DB 7003 have the same data structures as the image type management DB 3001a, the imaging device management DB 3002a, and the predetermined area management DB 3003a, respectively.

(Each functional configuration of the PC)
The transmitting/receiving section 71 of the PC 7 is mainly realized by the processing of the network I/F 509 and the CPU 501 shown in FIG. 12, and realizes the same function as the transmitting/receiving section 31a.

The reception unit 72 is realized mainly by the keyboard 511, the mouse 512, and the processing of the CPU 501, and realizes the same functions as the reception unit 32a. The image/sound processing unit 73 is mainly implemented by commands from the CPU 501, and implements the same functions as the image/sound processing unit 33a. The display control unit 74 is mainly implemented by the processing of the CPU 501, and implements the same function as the display control unit 34a. The determination unit 75 is mainly implemented by the processing of the CPU 501 and implements the same function as the determination unit 35a. The creation unit 76 is mainly implemented by the processing of the CPU 501 and implements the same function as the creation unit 36a. The calculation unit 77 is mainly realized by the processing of the CPU 501, and realizes the same function as the calculation unit 37a. The communication unit 78 is mainly implemented by the processing of the CPU 501 and implements the same functions as the communication unit 38a. The storage/readout unit 79a is realized by the processing of the CPU 501, stores various data (or information) in the storage unit 7000, and reads out various data (or information) from the storage unit 7000. FIG.

<Smartphone functional configuration>
Next, the functional configuration of the smartphone 9 will be described in detail with reference to FIGS. 13 and 14. FIG. The smart phone 9 basically has the same functions as the video conference terminal 3a. That is, as shown in FIG. 14, the smartphone 9 includes a transmission/reception unit 91, a reception unit 92, an image/sound processing unit 93, a display control unit 94, a determination unit 95, a creation unit 96, a calculation unit 97, a communication unit 98 and a storage/readout unit 99 . Each of these units is a function or means implemented by any one of the components shown in FIG. be.

The smartphone 9 also has a storage unit 9000 constructed by the ROM 902, RAM 903, and EEPROM 904 shown in FIG. In this storage unit 9000, an image type management DB 9001, an imaging device management DB 9002, and a predetermined area management DB 9003 are constructed. Note that the image type management DB 9001, the imaging device management DB 9002, and the predetermined area management DB 9003 have the same data configurations as the image type management DB 3001a, the imaging device management DB 3002a, and the predetermined area management DB 3003a, respectively.

(Each function configuration of the smartphone)
The transmitting/receiving unit 91 of the smartphone 9 is mainly implemented by the processing of the long-distance communication circuit 911 and the CPU 901 shown in FIG. 13, and implements the same function as the transmitting/receiving unit 31a.

The reception unit 92 is mainly realized by processing of the touch panel 921 and the CPU 901, and realizes the same function as the reception unit 32a.

The image/sound processing unit 93 is implemented mainly by commands from the CPU 901, and implements the same functions as the image/sound processing unit 33a. The display control unit 94 is mainly implemented by the processing of the CPU 901, and implements the same functions as the display control unit 34a. The determination unit 95 is mainly implemented by the processing of the CPU 901, and implements the same function as the determination unit 35a. The creation unit 96 is mainly implemented by the processing of the CPU 901, and implements the same function as the creation unit 36a. The calculation unit 97 is mainly realized by the processing of the CPU 901, and realizes the same function as the calculation unit 37a. The communication unit 98 is mainly implemented by the processing of the CPU 901 and implements the same functions as the communication unit 38a. A storage/readout unit 99 is realized by processing of the CPU 901 , stores various data (or information) in the storage unit 9000 , and reads out various data (or information) from the storage unit 9000 .

<<Processing or operation of the embodiment>>
Next, the processing or operation of this embodiment will be described with reference to FIGS. 22 to 34. FIG.

<Participation processing>
First, processing for joining a specific communication session will be described with reference to FIGS. 22 and 23. FIG. FIG. 22 is a sequence diagram showing the process of joining a specific communication session. FIG. 23 is a diagram showing a communication session (virtual conference room) selection screen.

First, when a user of site A (for example, user A1) performs an operation to display a communication session (virtual conference room) selection screen on video conference terminal 3a, reception unit 32a displays the selection screen. Upon receiving the operation, the display control unit 34a displays a selection screen as shown in FIG. 23 on the display 4a (step S21). On this selection screen, selection buttons b1, b2, b3, etc., indicating virtual conference rooms R1, R2, R3, etc., to be selected are displayed. Each session ID is associated with each selection button b1 and the like.

Here, when the user A1 selects a desired virtual conference room selection button (here, the selection button b1), the reception unit 32a receives selection of a communication session (step S22). The transmitter/receiver 31a then transmits a request to participate in the virtual conference room to the communication management system 5 (step S23). This participation request includes the session ID indicating the communication session whose selection was accepted in step S22, and the IP address of the videoconference terminal 3a, which is the request source terminal. Thereby, the transmission/reception unit 51 of the communication management system 5 receives the participation request.

Next, in the session management DB 5001 (see FIG. 19), the storage/reading unit 99 stores the IP address received in step S23 in the participating terminal IP address field of the record with the same session ID as the session ID received in step S23. By adding the address, processing for joining the communication session is performed (step S24). Then, the transmission/reception unit 51 transmits a participation request response to the video conference terminal 3a (step S25). This participation request response includes the session ID received in step S23 and the result of the participation processing. As a result, the transmission/reception unit 31a of the videoconference terminal 3a receives the participation request response. Hereinafter, a case where the participation processing is successful will be described.

<Management processing of image type information>
Next, the image type information management process will be described with reference to FIG. FIG. 24 is a sequence diagram showing management processing of image type information.

First, when a user at site A (for example, user A1) connects the USB cable of cradle 2a with photographing device 1a attached to video conference terminal 3a, storage/reading unit 19a of photographing device 1a The GUID of its own device (photographing device 1a) stored in the storage unit 1000a is read, and the communication unit 18a transmits the GUID of its own device to the communication unit 38a of the video conference terminal 3a (step S51). As a result, the communication unit 38a of the video conference terminal 3a receives the GUID of the imaging device 1a.

Next, the determination unit 35a of the video conference terminal 3a determines whether the same vendor ID and product ID as those in the GUID received in step S51 are managed in the photographing device management DB 3002a (see FIG. 17). The type of image is determined by determining whether or not it exists (step S52). Specifically, when the same vendor ID and product ID are managed in the imaging device management DB 3002a, the determination unit 35a determines that the imaging device 1a captures a special image (here, a spherical panorama image). It is determined that it is a photographing device. On the other hand, if the same vendor ID and product ID are not managed in the image capturing apparatus management DB 3002a, the determination unit 35a determines that the image capturing apparatus 1a is an image capturing apparatus that captures general images.

Next, the storage/readout unit 39a stores the IP address of its own terminal (videoconference terminal 3a), which is the transmission source terminal, and the determination result determined in step S52 in the image type management DB 3001a (see FIG. 16). It is stored in association with certain image type information (step S53). In this state, no image data ID is associated. The image type information is, for example, a source name determined according to a predetermined naming rule and an image type (general image, special image).

Next, the transmission/reception unit 31a transmits a request for adding image type information to the communication management system 5 (step S54). This image type information addition request includes the IP address of the terminal itself, which is the transmission source terminal, stored in step S53, and the image type information. Accordingly, the transmission/reception unit 51 of the communication management system 5 receives the request for adding the image type information.

Next, the storage/readout unit 59 of the communication management system 5 retrieves the corresponding session ID by searching the session management DB 5001 (see FIG. 19) using the IP address of the source terminal received in step S54 as a search key. is read out (step S55).

Next, the generator 56 generates a unique image data ID (step S56). Then, the storage/readout unit 59 stores the session ID read out in step S55, the image data ID generated in step S56, and the image data ID generated in step S54 as new records in the image type management DB 5002 (see FIG. 20). The received IP address of the transmission source terminal and the image type information are associated and stored (step S57). Then, the transmission/reception unit 51 transmits the image data ID generated in step S56 to the video conference terminal 3a. As a result, the transmission/reception unit 31a of the video conference terminal 3a receives the image data ID (step S58).

Next, the storage/reading unit 39a of the video conference terminal 3a stores the IP of the own terminal (video conference terminal 3a), which is the transmission source terminal, stored in the image type management DB 3001a (see FIG. 16) in step S53. The image data ID received in step S58 is stored in association with the address and image type information (step S59).

On the other hand, the transmission/reception unit 51 of the communication management system 5 transmits an additional notification of image type information to another communication terminal (here, the smart phone 9) (step S60). This additional notification of image type information includes the image data ID generated in step S56, and the IP address and image type information of the own terminal (videoconference terminal 3a) that is the transmission source terminal stored in step S53. is Thereby, the transmitting/receiving unit 91 of the smartphone 9 receives the additional notification of the image type information. The transmission destination of the transmission/reception unit 51 is another IP address associated with the same session ID as the IP address of the video conference terminal 3a in the session management DB 5001 (see FIG. 19). That is, the destination is another communication terminal in the same virtual conference room as the video conference terminal 3a.

Next, the storage/readout unit 99 of the smartphone 9 stores the image data ID, the IP address of the transmission source terminal, and the image type received in step S60 as a new record in the image type management DB 9001 (see FIG. 16). The information is associated and stored (step S61). Similarly, the additional communication of the image type information is also transmitted to the video conference terminals 3d and PC7, which are other communication terminals, and stored in the image type management DBs 3001d and 7001 of the video conference terminals 3d and PC7, respectively. As described above, the image type management DBs 3001a, 3001d, 7001, and 9001 can share the same information in each communication terminal.

<Communication processing of photographed image data>
Next, communication processing of photographed image data in a video call will be described with reference to FIGS. 25 to 34. FIG. FIG. 25 is an image diagram showing the state of a video call. Among them, FIG. 25(a) shows the case where the photographing device 1a is not used, and FIG. 25(b) shows the case where the photographing device 1a is used.

First, as shown in FIG. 25(a), when the camera 312 (see FIG. 11) provided in advance in the video conference terminal 3a is used without using the photographing device 1a, the angle of view is 125 degrees horizontally. , the videoconference terminal 3a must be placed at the end of the desk where each user A1 and the like can be seen. Therefore, each user A1 or the like must speak while facing the video conference terminal 3a. In addition, since each user A1 or the like faces the video conference terminal 3a, the display 4a is also placed on the side of the video conference terminal 3a. As a result, the users A2 and A4, who are away from the videoconference terminal 3a, have to speak in a relatively loud voice because they are away from the microphone 314 (see FIG. 11), and it is difficult for them to see the contents displayed on the display 4a.

On the other hand, as shown in FIG. 25(b), when using the imaging device 1a, it is possible to obtain two hemispherical images that form the basis of the omnidirectional panorama image. The terminal 3a and the display 4a can be relatively placed in the center of the desk. As a result, since each user A1 or the like is close to the microphone 314, they can speak in a relatively low voice, and the contents displayed on the display 4a can be easily seen. On the right side of the site A, a whiteboard 6 is installed on which the user A1 or the like can write letters, pictures, and the like.

Subsequently, using FIG. 26, the photographed image data and sound data obtained at base A shown in FIG. , to the video conference terminal 3d). FIG. 26 is a sequence diagram showing communication processing of captured image data and sound data in a video call.

First, the communication unit 18a of the photographing device 1a transmits the captured image data obtained by capturing an object or scenery and the sound data obtained by collecting sound to the communication unit 38a of the video conference terminal 3a (step S101). ). In this case, since the photographing device 1a is a device capable of obtaining two hemispherical images that are the basis of the omnidirectional panorama image, the photographed image data is composed of data of two hemispherical images. Thereby, the communication unit 38a of the video conference terminal 3a receives the captured image data and the sound data.

Next, the transmission/reception unit 31a of the videoconference terminal 3a transmits the captured image data and sound data sent from the imaging device 1a to the communication management system 5 (step S102). This transmission includes an image data ID for identifying the captured image data to be transmitted. Thereby, the transmission/reception unit 51 of the communication management system 5 receives the captured image data and the image data ID.

Next, the transmission/reception unit 51 of the communication management system 5 sends the captured image data and sound data to the communication terminals (smartphone 9, PC 7, video conference terminal 3d) participating in the same video call as the video conference terminal 3a. It transmits (steps S103, S104, S105). Each of these transmissions includes an image data ID for identifying the captured image data to be transmitted. Thereby, the transmission/reception unit 91 of the smartphone 9, the transmission/reception unit 71 of the PC 7, and the transmission/reception unit 31d of the video conference terminal 3d receive the captured image data, the image data ID, and the sound data, respectively.

Next, a display example of the display 917 at the site B will be described with reference to FIG. 27 . FIG. 27 is an example of a display at base B. FIG. Among them, FIG. 27(a) shows the photographed image data sent from the photographing device 1a at the base A via the video conference terminal 3a and the photographed image data sent from the photographing device 1b at the base B. , the case where the omnidirectional panorama image and the predetermined area image are not created and displayed as they are. On the other hand, FIG. 27(b) shows a case where an omnidirectional panorama image and a predetermined area image are created from the captured image data sent from the imaging devices 1a and 1b. An image of base A is displayed in the left display area (layout number "1") of the display 4d, and an image of base B is displayed in the upper right display area (layout number "2"). Further, the image of base C is displayed in the middle display area (layout number "3") on the right side of the display 4d, and the image of base D (own base) is displayed in the lower right display area (layout number "4"). is displayed. The display area with layout number "1" is the main display area, and the display areas with layout numbers "2", "3" and "4" are sub-display areas. The image in the main display area and the image in the sub display area can be changed by each communication terminal. Normally, at each site, an image of the site where the central person of the video call is present is displayed in the main display area.

Here, if the photographed image data sent from the photographing devices 1a and 1b capable of photographing an omnidirectional panorama image is displayed as it is, base A and base B are displayed as shown in FIG. are displayed as a front hemispherical image and a rear hemispherical image as shown in FIGS. 3(a) and 3(b), respectively.

On the other hand, the image/sound processing unit 93 generates a omnidirectional panoramic image from the photographed image data output by the imaging devices 1a and 1b capable of obtaining two hemispherical images that are the basis of the omnidirectional panoramic image. When the predetermined area image is created and the predetermined area image is created, the predetermined area image, which is a plane image, is displayed as shown in FIG. 27(b). At base C, the photographing device 8 that obtains a general image takes an image, and at base D, a video conference terminal 3d that obtains a general image takes an image. An image (here, a planar image) is displayed.

Further, the user at each site can change the predetermined area related to the predetermined area image in the same omnidirectional panorama image. For example, the user B1 operates the touch panel 921 so that the reception unit 92 receives movement of the predetermined area image, and the display control unit 94 shifts, rotates, reduces, or enlarges the predetermined area image. can do. As a result, as shown in FIG. 28(b), from the predetermined area image in which some of the users A1 and A2 of the site A are displayed in the initial setting (default), the As shown, the predetermined area image can be displayed. Specifically, FIG. 28(c) shows a change from a predetermined area image including users A1 and A2 to a predetermined area image including whiteboard 6 in the photographed image of site A shown in FIG. 25(b). It shows the modified state.

Note that the omnidirectional icons 191 and 192 shown in FIGS. 27(b) and 27(c) are special icons for indicating that they are predetermined area images indicating a predetermined area T that is part of the omnidirectional panorama image. It is an example of an image identification icon. Note that the display positions of the omnidirectional icons 191 and 192 may be anywhere such as the upper left, lower left, and lower right, instead of the upper right. Also, the types of omnidirectional icons 191 and 192 are not limited to those shown in FIGS. 27(b) and (c). Further, instead of the omnidirectional icons 191 and 192, characters such as "omnidirectional image" may be used, or a combination of an icon and characters may be used.

Next, referring to FIG. 28, image communication when the predetermined area image is displayed as shown in FIG. 27(b) and when the predetermined area image is changed from FIG. 27(b) to FIG. 27(c) Processing in the system will be explained. FIG. 28 is a sequence diagram showing processing for sharing predetermined area information. In FIG. 28, the video conference terminal 3a at the site A is the third communication terminal, the video conference terminal 3d at the site D is another communication terminal, and the smart phone 9 at the site B is the communication terminal (own terminal).

First, as shown in FIG. 27(b), when the user D1 or the like at the site D uses the video conference terminal 3d to display the predetermined area image of the site A, the transmitting/receiving unit of the video conference terminal 3d 31d transmits predetermined area information indicating the displayed predetermined area image to the communication management system 5 (step S111). The predetermined area information includes the IP address of the video conference terminal 3a, which is the transmission source of the captured image data, and the IP address of the video conference terminal 3d, which is the transmission destination of the captured image data (the transmission source of the predetermined area information). ing. Thereby, the transmitting/receiving section 51 of the communication management system 5 receives the predetermined area information.

Next, the storage/reading unit 59 of the communication management system 5 associates and stores the predetermined area information received in step S111 and the source and destination IP addresses in the predetermined area management DB 5003 (step S112). The processes of steps S111 and S112 are performed each time the predetermined area image is changed in the video conference terminal 3d, as shown in FIGS. 27(b) to 27(c).

Next, the storage/readout unit 59 of the communication management system 5 selects the latest pair at that time from among the pairs of the predetermined area information and each IP address stored in the predetermined area management DB 5003 (the latest stored pair). A set of predetermined area information and each IP address (which will be a set of data) is read at regular intervals (for example, 30 seconds) (step S113). Then, in step S113, the transmitting/receiving unit 51 transmits the read data to the other communication terminals (video conference terminal 3a, smartphone 9, PC 7) that are performing the same video call as the video conference terminal 3d that is the transmission source of the predetermined area information. Predetermined area information including each IP address issued is distributed (transmitted) (steps S114, S116, S118). As a result, the transmission/reception unit 31a of the video conference terminal 3a receives the predetermined area information. Then, the storage/reading unit 39a stores the predetermined area information received in step S114 and each IP address in association with each other in the predetermined area management DB 3003a (step S115). Similarly, in the smartphone 9, after the transmitting/receiving unit 91 receives the predetermined area information, the storage/reading unit 99 stores the predetermined area information received in step S116 in the predetermined area management DB 9003 while being associated with each IP address. (step S117). Furthermore, in the PC 7, after the transmitting/receiving unit 71 receives the predetermined area information, the storage/reading unit 79 stores the predetermined area management DB 7003 in association with each IP address received in step S118. Store (step S119).

As described above, the predetermined area information indicating the predetermined area image changed at the site A is transmitted to each of the communication terminals of the other sites B, C, and D during the same video call. The predetermined area information indicating the image is shared among the other bases B, C, and D as well. This processing is also performed in the same manner when the predetermined area image is changed at sites B, C, and D, and predetermined area information indicating the predetermined area image being displayed at each other site is shared among all sites during the same video call. be done.

Next, using FIGS. 29 to 34, a method of using the predetermined area information shared by each site will be described. FIG. 29 is a flow chart showing display processing of a predetermined area image. Since the processing of each communication terminal is the same, the processing of the smart phone 9 at the base B will be explained here. Specifically, when the captured image data transmitted from the videoconference terminal 3a at the site A is displayed as a predetermined area image by the videoconference terminal 3d at the site D, the videoconference terminal 3d is in the same video call. Processing executed by the smartphone 9 at the base B by transmitting predetermined area information indicating a predetermined area image to another communication terminal will be described.

First, in the smartphone 9, the storage/readout unit 99 searches the image type management DB 9001 (see FIG. 16) using the image data ID received in step S103 shown in FIG. Image type information (source name) is read out (step S131).

Next, the determination unit 95 determines whether or not the image type information read out in step S131 indicates a "special image" (step S132). If the image type information indicates that it is a "special image" (step S132; YES), the storage/reading unit 99 further causes the predetermined area management DB 9003 to display the image displayed by the communication terminal at the other site. Predetermined area information indicating the predetermined area image is retrieved (step S133). Then, the determination unit 95 determines whether predetermined area information indicating a predetermined area image displayed by a communication terminal at another site is managed in the predetermined area management DB 9003 (step S134). If the predetermined area information indicating the predetermined area image displayed by the communication terminal of the other site is managed (step S134; YES), the calculation unit 97 calculates the predetermined area displayed by the smartphone 9 (own terminal). Predetermined area information (i2) indicating the predetermined area image of T2, and predetermined area information (i1 ), the position of the predetermined region T1 with respect to the predetermined region T2 in the entire image is calculated (step S135). Strictly speaking, the position in this case indicates the gaze point of the predetermined area T1 with respect to the gaze point of the predetermined area T2. The gaze point is the center point as described above, but may be the upper left corner (or the lower left corner, the upper right corner, or the lower right corner) of the rectangle of each predetermined area. Alternatively, the gaze point may be a specific point within each predetermined area.

Here, a method of calculating the fixation point of the predetermined area T1 with respect to the predetermined area T2 in the entire image will be described with reference to FIG. FIG. 30(a) is a diagram showing the definition of the angle of the virtual camera, and FIG. 30(b) is a method of calculating the position of the gaze point of another base in the predetermined area image of the own base by parallel projection viewed from above. It is a figure which shows.

As shown in FIG. 30( a ), the calculation unit 97 calculates the radius r, the polar angle θ , and the azimuth angle φ, which is set as CP1(r0, θ1, φ1). Next, the calculation unit 97 obtains the radius vector r, the polar angle θ, and the azimuth angle φ from the predetermined area information of the other site read out in step S133, and designates them as CP2 (r0, θ2, φ2). do.

When considering a predetermined area T2 centered on the gaze point CP1 of the terminal (smartphone 9), the width w and the height h of the predetermined area T2 are assumed to be w in FIG. , the height is projected to the length of h cos θ1.

In addition, since the radius of the point of gaze CP1 is projected onto the length of r0sinθ1, and the radius of the point of gaze CP2 is projected onto the length of r0sinθ2, the point of gaze CP1 has coordinates (r0sinθ1·r0cosφ1, r0sinθ1·r0sinφ1), The viewpoint CP2 is located at coordinates (r0 sin θ2·r0 cos φ2, r0 sin θ2·r0 cos φ2).

By the above, in FIG.30(b). Since the coordinates of the point of gaze CP1 and the point of gaze CP2 have been derived, the position of the point of gaze CP2 on the plane of the predetermined area T2 having the width w and the height h can be derived using a general coordinate transformation. can.

Here, a method for calculating the direction of the predetermined area T1 with respect to the predetermined area T2 in the entire image will be described with reference to FIG. FIG. 31(a) is a diagram showing the definition of angles, and FIG. 31(b) is a diagram showing the definitions of angle ranges.

As shown in FIG. 31( a ), the calculation unit 97 obtains the azimuth angle φ from the predetermined area information indicating the predetermined area image being displayed by the display control unit 94 of the own terminal (smartphone 9 ). is the rotation angle φ1. Next, the calculation unit 97 acquires the azimuth angle φ from the predetermined area information of the other site read in step S133, and sets it as the rotation angle φ2. Further, the calculator 97 sets the difference between the rotation angle φ2 and the rotation angle φ1 as the rotation angle φ3.

Also, as shown in FIG. 31(b), the angle range centered on the rotation angle φ of the home base is α1, and the angle range centered on the horizontal angle of the home base plus 180 degrees is defined as α1. When α2, the calculation unit 97 calculates the direction of the predetermined region T1 with respect to the predetermined region T2 in the entire image as follows.
(1) If the rotation angle φ3 is included in the angle range α1, the positional relationship is determined to be "forward".
(2) If φ3 is included in the angle range α2, the positional relationship is determined to be "backward".
(3) If φ3 is not included in the angle range α1 and the angle range α2 and is greater than 0 degrees and less than 180 degrees, the positional relationship is determined to be "right".
(4) If φ3 is not included in the angle range α1 and the angle range α2 and is greater than 180 degrees and less than 360 degrees, the positional relationship is determined to be "leftward".

Next, the image/sound processing unit 93 creates a predetermined area image including the gaze point mark indicating the gaze point calculated by the calculation unit 97 and the display direction mark indicating the direction (step S136). The display position of the point-of-regard mark can be obtained directly from the position of the predetermined area T1 with respect to the predetermined area T2 in the entire image. The display position of the display direction mark is obtained by the determination processes (1) to (5) described above using the position of the predetermined region T1 with respect to the predetermined region T2 in the entire image.

At this time, the image/sound processing unit 93 places the omnidirectional icons 191 and 192 indicating the omnidirectional panorama image on the predetermined area image based on the fact that the image type information indicates the "special image". Synthesize. 32(a), (b), (c), FIGS. 33(a), (b), and FIG. An image is displayed (step S137). FIGS. 32A, 32B, and 32C show images of the main display area, and are diagrams showing three display examples of the predetermined area image including the display direction mark. FIGS. 33(a) and 33(b) show images of the main display area, and are diagrams showing two display examples of the predetermined area image including the gazing point mark. Note that FIG. 34 shows an image of the main display area, and is a diagram showing one display example of the predetermined area image including the gazing point mark and the display direction mark. Also, as shown in FIG. 27, the images of all sites during the video call are displayed, but in FIGS. it's shown.

As shown in FIG. 32(a), in the predetermined area image which is a part of the image of the site A, the predetermined area image being displayed at the site B (own site) is used as a reference in the entire image, Display direction marks m11, m13, and m14 are displayed to indicate the directions of the predetermined area images displayed at the other bases. The display direction marks m21, m23 and m24 shown in FIG. 32(b) and the display direction marks m31, m33 and m34 shown in FIG. corresponding to the display direction marks m11, m13, and m14.

Note that the display direction mark is an example of direction information, and may be in another form. Also, the direction information may be indicated by characters such as "right", "left", "rear" and "front" instead of arrows.

Further, as shown in FIG. 33(a), within the predetermined area image which is a part of the image of the site A, the predetermined area image being displayed at the site B (own site) is used as a reference in the entire image. , gazing point marks m41 and m42 indicating the gazing points of the predetermined area images displayed at the other bases are displayed. The point-of-regard mark may be displayed semi-transparently so as not to obscure the predetermined area image. The display direction marks m51 and m52 shown in FIG. 33(b) respectively correspond to the display direction marks m41 and m42 shown in FIG. 33(a).

In FIG. 33(a), in order to identify which base is the gaze point, a gaze point mark m41 indicating the base name "C" and a gaze point mark m42 indicating the base name "D" are shown. is displayed. On the other hand, in FIG. 33(b), different bases are indicated by gaze point marks of different patterns without displaying the base name. In this case, if each site prepares a table that associates patterns with site names, the user at each site can identify the site from the pattern of the point-of-regard mark. The table may be printed on paper, or may be stored as electronic data at each site.

It should be noted that, instead of the pattern for distinguishing the point-of-regard marks, the colors or line types may be changed so that they can be distinguished. Also, the point-of-regard mark is an example of corresponding position information.

In FIG. 34, a display mark m41 indicating the gaze point is displayed when the gaze point of the other site is included in the range of the predetermined area image, and display direction marks m11 and m14 are displayed when it is not included in the range of the predetermined area image. It is

Next, referring back to FIG. 29, if the predetermined area management DB 9003 does not manage the predetermined area information indicating the predetermined area image displayed by the communication terminal of the other site (step S134 ; NO), the image/sound processing unit 93 creates a predetermined area image without including the gazing point mark and the display direction mark (step S138). After that, the process proceeds to step S137.

If the image type information does not indicate a "special image" in the determination in step S132 (step S132; NO), that is, if the image type information indicates a "general image", Otherwise, the image/sound processing unit 93 does not create an omnidirectional panorama image from the captured image data received in step S103, and the display control unit 94 displays a general image (step S139).

As described above, users B1 and B2 at site B can grasp the positional relationship between the predetermined area image displayed at their own site and the predetermined area image displayed at another site. As a result, it is possible to prevent users B1 and B2 of base B from being unable to keep up with topics such as meetings.

<<main effects of the present embodiment>>
As described above, according to the present embodiment, the communication terminal such as the video conference terminal 3a creates an omnidirectional panoramic image based on the image data ID sent together with the image data and the corresponding image type information. Furthermore, a predetermined area image can be created. As a result, as shown in FIG. 27A, it is possible to prevent the hemispherical image on the front side and the hemispherical image on the rear side from being displayed.

Furthermore, since a user at an arbitrary base can grasp which part of the predetermined area image is displayed in the entire image of the omnidirectional panorama image at another base, conferences, etc., are easier than ever before. There is an effect that it becomes easy to follow the topic.

Also, in FIG. 28, if the communication management system 5 transfers the predetermined area information to another communication terminal each time it receives the predetermined area information from the video conference terminal 3d, each gazing point shown in FIG. Users B1 and B2 cannot concentrate on the video call due to the flickering of the mark and the display direction mark. Therefore, as shown in steps S112 to S114, the communication management system 5 distributes the latest set of the predetermined area image and each IP address at regular intervals so that each user can participate in the video call. can concentrate.

[Second embodiment]
Next, a second embodiment will be described with reference to FIG. 35 . FIG. 35 is a sequence diagram showing another example of the processing shown in FIG. 28 and showing another processing for sharing the predetermined area information. In FIG. 35, the videoconference terminal 3a at the base A is the communication terminal (own terminal), and the videoconference terminal 3d at the base D is the other communication terminal.

In the first embodiment, as shown in FIG. 28, the communication management system 5 once manages each predetermined area information transmitted from each communication terminal (see S112), and transmits the information at regular intervals. Each predetermined area information is transmitted to each communication terminal other than the original (see S114 to S119). On the other hand, in this embodiment, as shown in FIG. Information is temporarily managed (see S213), and each predetermined area information is transmitted to each communication terminal other than the own terminal at regular intervals (see S215 to S221). That is, in the present embodiment, the communication terminal that is the transmission source of the captured image data determines the specific area of the predetermined area T1 in which the captured image data transmitted by its own terminal (here, the video conference terminal 3a) is transmitted by another communication terminal. It manages whether it is displayed as an image.

Note that this embodiment has the same configuration as the first embodiment, but differs in the processing shown in FIG. Therefore, in the following description, the same reference numerals are used for the same configurations, and description thereof will be omitted, and portions of processing that differ from those of the first embodiment will be described with reference to FIG. 35 .

First, when the user D1 or the like at the site D uses the video conference terminal 3d to display a predetermined area image of the site A, the transmission/reception unit 31d of the video conference terminal 3d instructs the communication management system 5 to display the image. Predetermined area information indicating the predetermined area image is transmitted (step S211). The predetermined area information includes the IP address of the video conference terminal 3a, which is the transmission source of the captured image data, and the IP address of the video conference terminal 3d, which is the transmission destination of the captured image data (the transmission source of the predetermined area information). ing. Thereby, the transmitting/receiving section 51 of the communication management system 5 receives the predetermined area information.

Next, the transmission/reception unit 51 of the communication management system 5 transmits the predetermined area information including each IP address received in step S211 to the video conference terminal 3a, which is the source of the captured image data (step S212). As a result, the transmission/reception unit 31a of the video conference terminal 3a receives the predetermined area information.

Next, the storage/readout unit 39a of the video conference terminal 3a associates and stores the predetermined area information received in step S212 and the IP addresses of the source and destination in the predetermined area management DB 3003a (step S213). The process of step S213 is a process of managing how the photographed image data transmitted by the own terminal (here, the video conference terminal 3a) is displayed on other communication terminals. Note that the processing of steps S211 to S213 is performed each time the predetermined area image is changed in the video conference terminal 3d.

Next, the storage/readout unit 39a of the videoconference terminal 3a selects the latest pair at that point in time among the sets of the predetermined area information and each IP address stored in the predetermined area management DB 3003a (the latest set). The predetermined area information and each IP address that form a set of data are read at regular intervals (for example, 30 seconds) (step S214). Then, the transmitting/receiving section 31a transmits the predetermined area information including each IP address read in step S214 to the communication management system 5 (step S215). Thereby, the transmitting/receiving section 51 of the communication management system 5 receives the predetermined area information.

Next, the transmission/reception unit 51 of the communication management system 5 transmits (distributes) the predetermined area information including each IP address received in step S215 to each communication terminal (video conference terminal 3d, smartphone 9, PC 7) (step S216, S218, S220). As a result, the transmission/reception unit 31d of the video conference terminal 3d receives the predetermined area information. Then, the storage/reading unit 39d stores the predetermined area information received in step S216 in association with each IP address also received in the predetermined area management DB 3003d (step S217). Similarly, in the smartphone 9, after the transmitting/receiving unit 91 receives the predetermined area information, the storage/reading unit 99 stores the predetermined area information received in step S218 in the predetermined area management DB 9003 to each IP address that was similarly received. They are associated and stored (step S219). Furthermore, in the PC 7 as well, after the transmitting/receiving unit 71 receives the predetermined area information, the storage/reading unit 79 stores the predetermined area management DB 7003 in association with each IP address received in step S220. Store (step S221).

<<main effects of the present embodiment>>
As described above, according to the present embodiment, a communication terminal, which is the transmission source of captured image data, uses a predetermined display method to indicate how the captured image data transmitted from the own terminal is displayed on each communication terminal. Collect area information and deliver to each communication terminal. As a result, in addition to the effects of the first embodiment, there is an effect that it is possible to avoid concentration of the burden on the communication management system 5 when the same conference or the like is held using a large number of communication terminals.

[Third Embodiment]
Next, a third embodiment will be described.

In FIG. 10, the imaging device 1a has one microphone 108, but in this embodiment, the imaging device 1a has a plurality of directional microphones. Since the photographing device 1a uses a plurality of directional microphones, the sound data of each directional microphone is transmitted from the photographing device 1a to the video conference terminal 3a. As a result, the calculation unit 37a of the video conference terminal 3a calculates the direction of the sound source (microphone angle) from the voice data of each microphone, and the calculated direction is used to specify the position of the speaker (the source of the captured image). used for The same is true for the imaging device 1b. The imaging device 1b equipped with a plurality of directional microphones transmits the audio data of each microphone to the smartphone 9, and the calculation unit 97 of the smartphone 9 calculates the direction of the sound source from the audio data of each microphone. (microphone angle) is calculated, and the calculated angle is used to specify the position of the speaker (the transmission source of the captured image).

<<main effects of the present embodiment>>
As described above, it is possible to more accurately display which portion of the predetermined region image is displayed in the entire image of the omnidirectional panorama image at another site.

〔supplement〕
In the above embodiment, the predetermined area T is specified by the predetermined area information indicating the imaging direction and angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE, but is not limited to this. For example, when the angle of view is kept constant, the predetermined area T is specified by predetermined point information indicating the center point CP or arbitrary points at the four corners of the rectangular predetermined area T in FIG. may The "predetermined information" includes predetermined area information and predetermined information.

In the above embodiment, as an example of a panoramic image, a photographed image (whole image) is a three-dimensional omnidirectional panoramic image, but it may be a two-dimensional panoramic image.

In the above embodiment, the communication management system 5 relays the predetermined area information transmitted from each communication terminal. good.

Also, each function of the above embodiments can be realized by one or a plurality of processing circuits. Here, the "processing circuit" in this specification means a processor programmed by software to perform each function, such as a processor implemented by an electronic circuit, or a processor designed to perform each function described above. devices such as ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array), SOC (System on a chip), GPU (Graphics Processing Unit) and conventional circuit modules.

1a Photographing device 1b Photographing device 3a Video conference terminal (an example of a third communication terminal/an example of a communication terminal (own terminal))
3d video conference terminal (an example of other communication terminals)
4a display 4d display (an example of display means)
5 Communication management system 7 PC
8 Shooting device 9 Smartphone (an example of a communication terminal (own terminal))
31d Transmitting/receiving unit (an example of transmitting means, an example of receiving means)
37d calculator (an example of changing means)
3001d Image type management DB (an example of image type management means)
3002d Photographing device management DB (an example of photographing device management means)
3003d Predetermined area management DB (an example of predetermined area management means)
3004d Required resolution management DB (an example of required resolution management means)
m11 Display direction mark (an example of direction information)
m14 Display direction mark (an example of direction information)
m41 gaze point mark (an example of corresponding position information)

JP 2012-178135 A JP 2011-223076 A

Claims (9)

A communication terminal capable of displaying a first predetermined area image, which is an image of a first predetermined area in a whole image transmitted from another communication terminal, and changing the first predetermined area ,
Receiving second predetermined information for specifying the second predetermined area transmitted from another communication terminal displaying a second predetermined area image, which is an image of the second predetermined area in the overall image receiving means for
Based on the first predetermined information and the second predetermined information for specifying a first predetermined area image that is an image of a first predetermined area in the whole image, the first predetermined area in the whole image calculating means for calculating the position of the second predetermined area with respect to the area;
at least one of corresponding position information indicating the position calculated by the calculating means and direction information indicating the direction of the second predetermined area with respect to the first predetermined area, based on the position calculated by the calculating means; Display control means for displaying included in
A communication terminal characterized by having: A communication terminal according to claim 1,
image type management means for managing predetermined image data identification information for identifying predetermined whole image data and predetermined image type information indicating the type of the whole image in association with each other;
When the receiving means receives the specific whole image data and the specific image data identification information for identifying the whole image data transmitted from the other communication terminal, the image type managing means When the specific image type information corresponding to the received specific image data identification information indicates that it is an omnidirectional panorama image, the entire omnidirectional panorama image is obtained based on the data of the specific entire image. creating means for creating the first predetermined area image in the image;
has
the creating means creates the second predetermined area image including at least one of the corresponding position information and direction information based on the position of the second predetermined area calculated by the calculating means;
The communication terminal, wherein the display control means displays the first predetermined area image including at least one of the corresponding position information and the direction information created by the creation means. A communication terminal according to claim 1,
for specifying destination information of a communication terminal that is a transmission source of the overall image data, destination information of a communication terminal that is a transmission destination of the overall image data, and a predetermined area image displayed by the transmission destination communication terminal; predetermined area management means for managing predetermined information in association with the predetermined area;
a transmission means for transmitting data of an entire image of the own base, which is the communication terminal, so as to be received by the other communication terminal;
has
The receiving means transmits destination information of the own terminal, which is a transmission source of data of the whole image which is transmitted from the other communication terminal together with the second predetermined information and is received by the other communication terminal, and the other communication. Receive the destination information of the terminal,
the predetermined area management means associates and manages the destination information of the own terminal, the destination information of the other communication terminal, and the second predetermined information received by the receiving means;
The transmission means receives the destination information of the own terminal and the destination information of the other communication terminal managed by the predetermined area management means, and the destination information of the other communication terminal, so that the communication terminals other than the own terminal during the same video call receive the information. 1. A communication terminal characterized by transmitting predetermined information. 4. The communication terminal according to claim 3 , wherein the data of said whole image is transmitted from said communication terminal. The transmission means transmits the latest destination information of the own terminal, the destination information of the other communication terminal, and the second predetermined information managed by the predetermined area management means at regular intervals. The communication terminal according to claim 4 , wherein 6. The communication terminal according to any one of claims 1 to 5 , wherein the communication terminal is a video conference terminal, personal computer, smart phone, digital television, smart watch, or car navigation device. A communication terminal according to any one of claims 1 to 6;
displaying a second predetermined area image of the second predetermined area in an entire image based on image data captured by an external or built-in image capturing device, and displaying the image data and the image data to the communication terminal; another communication terminal that transmits second predetermined information for specifying the second predetermined area ;
a communication management system for managing communication of captured image data between the communication terminal and another communication terminal;
An image communication system having A display method executed by a communication terminal capable of displaying a first predetermined area image, which is an image of a first predetermined area in a whole image transmitted from another communication terminal, and changing the first predetermined area, ,
receiving second predetermined information for specifying the second predetermined area transmitted by another communication terminal displaying a second predetermined area image that is an image of the second predetermined area in the overall image; a receiving step;
based on the first predetermined information and the second predetermined information for specifying a first predetermined area image, which is an image of the first predetermined area in the entire image, for the first predetermined area in the entire image a calculating step of calculating the position of the second predetermined area;
based on the position calculated by the calculating step, at least one of corresponding position information indicating the position and direction information indicating the direction of the second predetermined area with respect to the first predetermined area is stored in the first predetermined area image; a display control step to be included in and displayed in
A display method characterized by executing A program for displaying a first predetermined area image, which is an image of a first predetermined area in a whole image transmitted from another communication terminal, and executing a communication terminal capable of changing the first predetermined area,
receiving second predetermined information for specifying the second predetermined area transmitted by another communication terminal displaying a second predetermined area image that is an image of the second predetermined area in the overall image; a receiving step;
based on the first predetermined information and the second predetermined information for specifying a first predetermined area image, which is an image of the first predetermined area in the entire image, for the first predetermined area in the entire image a calculating step of calculating the position of the second predetermined area;
based on the position calculated by the calculating step, at least one of corresponding position information indicating the position and direction information indicating the direction of the second predetermined area with respect to the first predetermined area is stored in the first predetermined area image; a display control step to be included in and displayed in
program that has

Images