JP2015035696A - Communication terminal, communication system, and communication method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To confirm a quality level of a content which is shared with respective terminals and whose operation is synchronized.SOLUTION: A communication terminal for transmitting image data shared with another communication terminal to the other communication terminal comprises: a storage unit for storing an attribute acquired in comparing first image data transmitted by the communication terminal with second image data obtained by the reception of the first image data by the other communication terminal, and information showing a criterion used in determining a quality difference between the first image data and second image data on the basis of the attribute; and a control unit which acquires, on the basis of the attribute, a first attribute of the first image data transmitted to the other communication terminal, and determines the quality difference between the first image data and second image data from the information showing the criterion on the basis of the first attribute and a second attribute acquired on the second image data by the other communication terminal.

Description

  The present invention relates to a communication system.

  In recent years, with the increase in capacity of Internet lines, there is an opportunity to select a meeting using a video conferencing terminal or a web conferencing application that can communicate while looking at the face of the other party as one of the communication tools in the business field. It has increased.

  When using video conferencing terminals and web conferencing apps, drawings that are represented mainly by text information, presentation software, drafting software, and photographs, represented by word processing software and spreadsheet software. In many cases, information is mainly shared.

  However, it is difficult to synchronize contents between terminals because there is a great difference in the time from the time when contents are transmitted to the completion of data arrival between terminals mainly due to factors on the network side. For this reason, in video conferences and web conferences, it is not uncommon to operate content while confirming whether the content is synchronized by inquiring by voice.

  A technology is known in which a server having content reflects the operation of the content while synchronizing the content between the terminals so that the operation can be performed while synchronizing the content between the terminals (for example, Patent Documents) 1).

  However, with technology that reflects content operations while synchronizing content between terminals, it is confirmed whether the quality of content that has been received and that synchronization has been completed is the same between the sending side and the receiving side. I can't.

  Even if the transmission side receives a response indicating that the content has been received from the reception side and confirms that the content operations are synchronized, the quality of the content is the same between the transmission side and the reception side. It is not possible to confirm whether or not

  That is, in a video conference or a web conference, it is not easy to check the quality level of content that is shared between terminals and synchronized in operation on the transmission side and the reception side.

  Therefore, an object of the present invention is to confirm the quality level of content that is shared among terminals and synchronized in operation.

A communication terminal according to an embodiment of the disclosure is:
A communication terminal that is connected to a plurality of communication terminals via a network and transmits image data shared with the other communication terminal to another communication terminal,
An attribute acquired when comparing the first image data transmitted by the communication terminal and the second image data obtained by receiving the first image data by the other communication terminal; A storage unit for storing information indicating a reference used when determining a difference in quality between the first image data and the second image data based on
Based on the attribute stored in the storage unit, the first attribute of the first image data transmitted to the other communication terminal is acquired, and the second image is acquired by the first attribute and the other communication terminal. A control unit that determines a quality difference between the first image data and the second image data based on information indicating the reference based on a second attribute acquired for the data.

  According to the disclosed embodiment, it is possible to check the quality level of content that is shared among terminals and synchronized in operation.

It is an example of a block diagram of a telecommunications system. It is an example of the hardware block diagram of a communication terminal. It is an example of the hardware block diagram of a remote communication management system. It is an example of the functional block diagram of each terminal, apparatus, and system which comprises a telecommunications system. It is an example of a functional block diagram of an external input device. It is a figure which shows an example of a material management table. It is a figure which shows an example of a change quality management table. It is a figure which shows an example of a relay apparatus management table. It is a figure which shows an example of a terminal authentication management table. It is a figure which shows an example of a terminal management table. It is a figure which shows an example of a destination list management table. It is a figure which shows an example of a session management table. It is a figure which shows an example of an address priority management table. It is a figure which shows an example of a transmission rate priority management table. It is a figure which shows an example of a quality management table. It is a figure which shows an example of an arrangement | positioning information management table. It is an example of the figure which shows the example of the division | segmentation of a display, and the area | region produced by division | segmentation. It is a figure which shows an example of the relationship between arrangement | positioning information, and the video data or display data allocated to each area | region 1-4. It is an example of the state transition diagram of the arrangement | positioning information selected. It is a figure which shows an example of a response | compatibility of the event transmitted from a relay apparatus, the display data which a arrangement | positioning information selection part sets according to an event, and the display control flag which a display data control part determines. It is an example of the figure explaining the example of a transition of a display apparatus menu. It is a figure which shows an example of a quality synchronous judgment standard. It is a figure which shows an example of the content which mainly has character information, and the content which mainly has drawing information. It is a figure which shows an example of an OCR result. It is a figure which shows an example of color distribution, a frequency component, and edge information. It is a figure which shows an example of the sequence chart which shows the process in which communication terminals establish a session. It is an example of the sequence diagram which showed the process which transmits / receives image data and audio | voice data between communication terminals. It is an example of the flowchart figure of the procedure which transmits display data from an external input device to a communication terminal. Sequence diagram of transmission of video data captured by communication terminal 1, video data captured by communication terminal 2, and display data transmitted from external input devices 1 and 2 connected to communication terminals 1 and 2 It is an example. FIG. 30 is an example of a sequence diagram illustrating a control procedure when the external input device starts transmission of display data in S3 or S4 of FIG. 29. It is an example of the figure which shows typically the state of the screen of a display until a communication terminal receives display data. It is a figure which shows an example of the menu displayed on an external input device. It is an example of the sequence diagram which shows the control procedure when an external input device starts transmission of display data. FIG. 10 is an example of a flowchart illustrating a procedure for determining whether or not to request a distribution stop when a communication terminal receives a “distribution start event from another”. It is an example of the flowchart figure which shows the determination procedure of a relay apparatus. FIG. 30 is an example of a sequence diagram illustrating a procedure in which the external input device stops transmission of display data in a state where the processing of FIG. 29 is completed. It is an example of the flowchart figure which shows the process which communication terminals cut | disconnect a session. It is a figure which shows one Example of the quality judgment process of display data. It is a figure which shows an example of the classification | category of content. It is a figure which shows one Example (the 1) of the quality judgment process of display data. It is a figure which shows one Example (the 2) of the quality judgment process of display data. It is a figure which shows one Example (the 3) of the quality judgment process of display data. It is a figure which shows one Example (the 4) of the quality judgment process of display data. It is a figure which shows one Example (the 5) of the quality judgment process of display data. It is a figure which shows one Example (the 6) of the quality judgment process of display data. It is a sequence chart which shows operation | movement of the telecommunications system concerning a present Example. An example of feedback display is shown. The effect of the telecommunications system concerning a present Example is shown.

Next, the form for implementing this invention is demonstrated, referring drawings based on the following Examples. Examples described below are merely examples, and embodiments to which the present invention is applied are not limited to the following examples.
In all the drawings for explaining the embodiments, the same reference numerals are used for those having the same function, and repeated explanation is omitted.

<Example>
<Telecommunication system>
The general features of the remote communication system 1 of the present embodiment are as follows.
(1) The external input devices A and B at two different bases each display a menu for starting sharing of display data of materials on the display device, and each can distribute the display data. On the other hand, for example, the communication terminal α connected to the external input device A receives an event notification that display data distribution starts from the external input device B via the relay device. The external input device A stops transmission of the display data that has been transmitted so far, and displays a menu for starting display data sharing again. The communication terminal α displays the display data received from the external input device B.

Therefore, the shared display data displayed by the communication terminal α can be limited to one, and the TV conference participant does not need to select from a plurality of display data.
(2) The external input devices A and B display a message indicating the display data sharing state in response to the event notification. Thereby, the participant can grasp | ascertain the sharing aspect of the display data which communication terminal (alpha) and (beta) are displaying.
(3) In addition, the communication terminals α and β place video data and display data in an emphasized (enlarged) manner on a single screen in response to an event notification from the relay device. Display data to which attention should be paid can be provided without switching the display of video data.
(4) Further, when the communication terminals α and β share and synchronize the video data and display data, the communication terminals α and β compare the video data and display data transmitted by the transmission side with the video data and display data received by the reception side. Judge differences based on certain criteria. Thereby, each user on the transmission side and the reception side can easily check the quality level of the shared video data and display data between the terminals.

<Configuration of telecommunications system>
FIG. 1 is an example of a block diagram of a telecommunications system. As an example, the remote communication system is divided into a region A and a region B, and the communication terminal 10 is arranged in each region. However, at least two communication terminals allow remote communication.

  The remote communication system 1 includes a plurality of communication terminals (10aa, 10ab,..., 10db) and a display (100aa, 100ab,..., 100db) for each communication terminal (10aa, 10ab,..., 10db). Further, the remote communication system 1 includes a plurality of relay devices (30a, 30b, 30c, 30d), a remote communication management system 50, a program providing system 90, and an external input device 40.

  Note that “communication terminal 10” is used to indicate an arbitrary communication terminal among the communication terminals (10aa, 10ab,..., 10db). “Display 100” is used to indicate an arbitrary display among the displays (100aa, 100ab,..., 100db). “Relay device 30” is used to indicate an arbitrary relay device among the relay devices (30a, 30b, 30c, 30d). “External input device 40” is used to indicate an arbitrary external input device among the external input devices (40aa, 40ab,..., 40db).

  The communication terminal 10 transmits / receives content such as image data and audio data to / from other communication terminals 10. In the present embodiment, the case where the image of the image data is a moving image will be described. The image of the image data may include both a moving image and a still image. The relay device 30 relays image data and audio data with a plurality of communication terminals 10. The remote communication management system 50 manages the communication terminal 10 and the relay device 30 in an integrated manner.

  The external input device 40 is connected to the communication terminal 10 and transmits display data for displaying material data to the communication terminal 10. The material data here means data used by using document creation software, spreadsheet software, presentation software, and the like.

  Further, the plurality of routers (70a, 70b,..., 70f) shown in FIG. 1 select an optimum route for image data and audio data. In the present embodiment, “router 70” is used to indicate an arbitrary router among the routers (70a, 70b,..., 70f).

  The program providing system 90 includes an HDD (Hard Disk Drive) (not shown) in which a communication terminal program for causing the communication terminal 10 to realize various functions or various means is stored. Program can be sent. The HDD of the program providing system 90 also stores a relay device program for causing the relay device 30 to realize various functions or various means, and can transmit the relay device program to the relay device 30. . Further, the HDD of the program providing system 90 also stores a remote communication management program for causing the remote communication management system 50 to realize various functions or various means, and the remote communication management system 50 stores the remote communication management program. Can be sent.

  The communication terminal 10aa, the communication terminal 10ab, the relay device 30a, and the router 70a are connected to each other via a LAN 2a so as to communicate with each other. The communication terminal 10ba, the communication terminal 10bb, the relay device 30b, and the router 70b are communicably connected to each other via the LAN 2b. The LAN 2a and the LAN 2b are communicably connected via a dedicated line 2ab including the router 70c, and are constructed in a predetermined area A. For example, the region A is Japan, the LAN 2a is constructed in a Tokyo office, and the LAN 2b is constructed in an Osaka office.

  On the other hand, the communication terminal 10ca, the communication terminal 10cb, the relay device 30c, and the router 70d are connected to be communicable with each other via the LAN 2c. The communication terminal 10da, the communication terminal 10db, the relay device 30d, and the router 70e are connected to be communicable with each other via the LAN 2d. The LAN 2c and the LAN 2d are communicably connected by a dedicated line 2cd including the router 70f, and are constructed in a predetermined area B. For example, region B is the United States, LAN 2c is built in a New York office, and LAN 2d is Washington D.C. C. Is built in the office.

  Area A and area B are connected to each other via routers (70c, 70f) via the Internet 2i.

  The remote communication management system 50 and the program providing system 90 are communicably connected to the communication terminal 10 and the relay device 30 via the Internet 2i. The remote communication management system 50 and the program providing system 90 may be installed in the region A or the region B, or may be installed in a region other than these.

  In the present embodiment, the communication network 2 of the present embodiment is constructed by the LAN 2a, the LAN 2b, the dedicated line 2ab, the Internet 2i, the dedicated line 2cd, the LAN 2c, and the LAN 2d.

  Further, in FIG. 1, four sets of numbers shown under each communication terminal 10, each relay device 30, remote communication management system 50, each router 70, and program providing system 90 are in general IPv4. An IP address is simply shown. For example, the IP address of the communication terminal 10aa is “1.2.1.3”. In addition, IPv6 may be used instead of IPv4, but IPv4 will be used for the sake of simplicity.

<Hardware configuration>
Next, the hardware configuration of each device will be described. In the present embodiment, when there is a delay in the reception of image data at the communication terminal 10 as the relay destination, the image resolution of the image data is changed by the relay device 30 and then the image is sent to the communication terminal 10 as the relay destination. A case where data is transmitted will be described.

  FIG. 2 is an example of a hardware configuration diagram of the communication terminal 10. As shown in FIG. 2, the communication terminal 10 of this embodiment includes a CPU (Central Processing Unit) 101 that controls the operation of the communication terminal 10 as a whole, and a ROM (Read Only Memory) 102 that stores a communication terminal program. It has. Further, the communication terminal 10 includes a RAM (Random Access Memory) 103 used as a work area for the CPU 101, a flash memory 104 for storing various data such as image data and audio data, and various data for the flash memory 104 according to the control of the CPU 101. An SSD (Solid State Drive) 105 that controls reading or writing of data is provided. Further, the communication terminal 10 includes a media drive 107 that controls reading or writing (storage) of data with respect to the recording medium 106 such as a flash memory, an operation button 108 that is operated when a destination of the communication terminal 10 is selected, a communication terminal, and the like. 10 is provided with a power switch 109 for switching ON / OFF of the power supply. Further, the communication terminal 10 includes a network I / F 111 for transmitting data using the communication network 2 described later, a CCD (Charge Coupled Device) 112 that captures an image of a subject under the control of the CPU 101, and obtains image data. An image sensor I / F 113 that controls driving is provided. Furthermore, the communication terminal 10 includes a microphone 114 for inputting sound, a speaker 115 for outputting sound, and a sound input / output I / F 116 for processing input / output of sound signals between the microphone 114 and the speaker 115 according to the control of the CPU 101. ing. Further, the communication terminal 10 includes a display I / F 117 that transmits image data to the external display 100 according to the control of the CPU 101, an external device I / F 118 that transmits / receives various data to / from an external device, and the above-described components in FIG. As shown, a bus line 110 such as an address bus or a data bus for electrical connection is provided.

  The recording medium 106 is detachable from the communication terminal 10. Further, as long as it is a non-volatile memory that reads or writes data according to the control of the CPU 101, not only the flash memory 104 but also an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used. Further, the CCD 112 is a solid-state imaging device that converts light into electric charges and digitizes a subject image (video). The CCD 112 is not limited to a CCD as long as it captures a subject, and a CMOS (Complementary Metal Oxide Semiconductor) or the like. It may be used. Further, the display 100 is configured by a liquid crystal or an organic EL that displays an image of a subject, an operation icon, and the like.

  Further, the communication terminal program may be installed in an installable or executable format, recorded on a computer-readable recording medium such as the recording medium 106, and distributed.

  FIG. 3 is an example of a hardware configuration diagram of the remote communication management system 50 according to an embodiment of the present invention. The remote communication management system 50 includes a CPU 201 that controls the overall operation of the remote communication management system 50, a ROM 202 that stores a remote communication management program, a RAM 203 that is used as a work area for the CPU 201, and an HD (Hard Disk) that stores various data. 204 is provided. Further, the remote communication management system 50 reads or writes (stores) data to or from a recording medium 206 such as an HDD (Hard Disk Drive) 205 or a flash memory that controls reading or writing of various data to the HD 204 according to the control of the CPU 201. A media drive 207 to be controlled and a display 208 for displaying various information such as a cursor, menu, window, character, or image are provided. Further, the remote communication management system 50 includes a network I / F 209 for transmitting data using a communication network 2 described later, and a keyboard 211 having a plurality of keys for inputting characters, numerical values, various instructions, and the like. I have. Further, the remote communication management system 50 includes a mouse 212 for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like, and a CD-ROM (Compact Disc Read Only Memory) 213 as an example of a removable recording medium. A CD-ROM drive 214 for controlling reading or writing of data with respect to the recording medium. Further, the remote communication management system 50 includes an external device I / F 215 that transmits / receives information to / from an external device, and an address bus and a data bus for electrically connecting the above components as shown in FIG. The bus line 210 is provided.

  The remote communication management program is a file in an installable or executable format, and is recorded on a computer-readable recording medium such as the recording medium 206 or CD-ROM 213 for distribution. Good.

  The external input device 40 can have the same hardware configuration as that of the remote communication management system 50. However, the ROM 202 stores an external input device program for controlling the external input device 40. Also in this case, the external input device program is a file in an installable or executable format, and is recorded and distributed on a computer-readable recording medium such as the recording medium 206 or the CD-ROM 213. Also good.

  The relay device 30 can be applied with the same hardware configuration as that of the remote communication management system 50. However, the ROM 202 stores a relay device program for controlling the relay device 30. Also in this case, the relay device program is a file in an installable or executable format, and is recorded on a computer-readable recording medium such as the recording medium 206 or CD-ROM 213 for distribution. Good.

  Further, the program providing system 90 can apply a hardware configuration similar to that of the remote communication management system 50. However, a program providing program for controlling the program providing system 90 is recorded in the ROM 202. Also in this case, the program providing program is a file in an installable or executable format, and may be recorded and distributed on a computer-readable recording medium such as the recording medium 206 or the CD-ROM 213. Good.

  As another example of the detachable recording medium, the recording medium is provided by being recorded on a computer-readable recording medium such as a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), or a Blu-ray Disc. May be.

  FIG. 4 is an example of a functional block diagram of each terminal, device, and system constituting the telecommunications system 1 of the present embodiment. In FIG. 4, the communication terminal 10, the relay device 30, and the remote communication management system 50 are connected so that data communication can be performed via the communication network 2. The external input device 40 is connected by, for example, a USB cable so that data can be transmitted to and received from the communication terminal 10. The program providing system 90 shown in FIG. 1 is omitted in FIG. 4 because it is not directly related to the video conference communication. 5 is an example of a functional block diagram of the external input device 40, but for ease of explanation, the functional block diagram of the communication terminal 10 of FIG.

<Functional configuration of communication terminal>
The communication terminal 10 includes a transmission / reception unit 11, an operation input reception unit 12, a login request unit 13, an imaging unit 14a, an image display control unit 14b, a display image analysis unit 14c, a quality synchronization determination unit 14d, a voice input unit 15a, and a voice output unit. 15b, a selection processing unit 16, a delay detection unit 17, an external information transmission / reception unit 18, a storage / reading processing unit 19, an arrangement information selection unit 21, and a display data control unit 22. Each of these units is a function or means realized by any one of the constituent elements shown in FIG. 2 operating according to a command from the CPU 201 according to a program stored in the ROM 202.

  When the external input device 40 is connected to the communication terminal 10, the external input device 40 acquires display data and transmits it to the communication terminal 10. When receiving the display data, the communication terminal 10 transmits the display data to the relay device 30 via the communication network 2.

  Further, the communication terminal 10 has a storage unit 1000 constructed by any one of the ROM 102, the RAM 103, and the flash memory 104 shown in FIG. The storage unit 1000 stores an arrangement information management table 23, an event flag table 24, a material management table 25, and a quality synchronization determination criterion 26.

<Each functional unit of communication terminal>
Next, each part of the communication terminal will be described in detail. The transmission / reception unit 11 of the communication terminal 10 is realized by the network I / F 111 illustrated in FIG. 2, and transmits / receives various data (information) to / from other terminals, devices, or systems via the communication network 2. The operation input receiving unit 12 is realized by the operation button 108 and the power switch 109 shown in FIG. 2 and receives various inputs by the user. For example, when the user turns on the power switch 109 shown in FIG. 2, the operation input receiving unit 12 shown in FIG. 4 receives the power ON and turns on the power. The login request unit 13 is realized by an instruction from the CPU 101 shown in FIG. 2, and requests the login from the transmission / reception unit 11 to the remote communication management system 50 via the communication network 2 when receiving the power ON. Log-in request information indicating that the communication is to be performed and the current IP address of the communication terminal 10ab are automatically transmitted.

  The imaging unit 14a is realized by the CCD 112 and the imaging element I / F 113 shown in FIG. 2, images a subject, and outputs image data obtained by the imaging.

  The image display control unit 14 b is realized by the display I / F 117 illustrated in FIG. 2, and performs control for transmitting image data to the external display 100. The image display control unit 14b displays the display of the external input device 40 connected to the communication terminal 10 as its own device, in addition to the video data captured by the communication terminal 10 as its own device and the imaging unit 14a of the other communication terminal 10. The display data displayed and captured by the device 400 and display data shared with other communication terminals 10 are displayed. The image display control unit 14b has a function of controlling a display area for video data and display data.

  The display image analysis unit 14c analyzes the quality of video data and display data captured by the imaging unit 14a of the other communication terminal 10 displayed by the image display control unit 14b.

  The quality synchronization determination unit 14d, based on the quality synchronization determination standard 26, the video data and display data captured by the imaging unit 14a of the other communication terminal 10 whose quality is analyzed by the display image analysis unit 14c, and the image display The video data and display data displayed by the control unit 14b are compared, and it is determined whether the quality is synchronized based on the quality difference.

  The voice input unit 15a is realized by the microphone 114 and the voice input / output I / F 116 shown in FIG. 2, and inputs a user's voice and converts the voice into a voice signal, thereby relating to the voice signal. Output audio data. The audio output unit 15b is realized by the speaker 115 and the audio input / output I / F 116 shown in FIG. 2, and converts an audio signal related to the audio data into audio and outputs it. The voice input unit 15a measures the input level of the voice signal and compares it with a threshold value or the like to determine the presence or absence of the voice signal.

  The arrangement information selection unit 21 selects a shared flag from the event flag table 24 based on a display data distribution event and sets it in the arrangement information management table 23, whereby the arrangement information of the screen to be displayed on the display 100 by the image display control unit 14 b is displayed. Instruct. Details of control using the arrangement information management table 23, the event flag table 24, and the quality synchronization determination reference 26 will be described later.

  The selection processing unit 16 performs a final narrowing process for finally narrowing down from a plurality of relay devices 30 to one relay device 30. The selection processing unit 16 measures the reception date and time when the pre-transmission information is received by the transmission / reception unit 11 for each pre-transmission information described later received by the transmission / reception unit 11. For each pre-transmission information whose reception date and time is measured, the selection processing unit 16 starts transmission of the pre-transmission information based on the difference between the measured reception time and the transmission date and time included in the pre-transmission information. Calculate the time required to receive. The selection processing unit 16 finally selects one relay device 30 by selecting the relay device 30 to which the pre-transmission information that required the shortest required time is relayed among the calculated required times.

  The delay detection unit 17 is realized by a command from the CPU 101 shown in FIG. 2 and detects a delay time (ms) of image data or audio data sent from another communication terminal 10 via the relay device 30. To do. The external information transmitting / receiving unit 18 transmits / receives data to / from the external input device 40 through the external device I / F 215. Further, the storage / reading processing unit 19 stores the various data in the storage unit 1000 and reads out the various data stored in the storage unit 1000 by using the SSD 105 shown in FIG. The storage unit 1000 includes a terminal ID (Identification) for identifying the communication terminal 10 and a password, as well as a relay device ID for identifying the relay device 30 that transmits image data, audio data, and various data, and a destination terminal. An IP address or the like is stored.

  The display data control unit 22 acquires display data of the external input device 40 and transmits the acquired display data to the communication terminal 10. The display data represents image data obtained by converting an image displayed on the screen of the display device 400 into a format such as JPEG (Joint Photographic Experts Group) or Bitmap, or a rendering command such as GDI (Graphics Device Interface). The display data may include document data or may be a simple desktop screen that does not include document data.

  The display data control unit 22 requests the relay device 30 to start display data distribution or to stop distribution in response to a request from the external input device 40. Further, the state of the display control flag is determined by referring to the event flag table 24 according to the distribution event from the relay device 30, and the display control flag is transmitted to the external input device 40. The display control flag will be described later.

  Note that the terminal ID of this embodiment and the relay device ID described later are identification information such as language, characters, symbols, or various signs used to uniquely identify the communication terminal 10 and the relay device 30, respectively. Show. Further, the terminal ID and the relay device ID may be identification information in which at least two of the language, characters, symbols, and various indicia are combined. In the following description, the communication terminal 10 as a request source requesting the start of a video conference is described as “request source terminal 10A”, and the communication terminal 10 as a request destination is described as “destination terminal 10B”.

<Functional configuration of external input device>
As shown in FIG. 5, the external input device 40 includes a transmission / reception unit 41, a connection detection unit 42, a storage unit 43, a display data acquisition unit 451, a display data transmission unit 452, a menu control unit 44, and an operation input reception unit. 46 and a storage / reading processing unit 49. Each of these units is a function or means realized by any one of the components shown in FIG. 3 operating according to a command from the CPU 201 according to a program stored in the ROM 202. The external input device 40 includes a display device 400 constructed by the display 208 shown in FIG. Further, the external input device 40 is equipped with an OS (Operating System) such as Windows (registered trademark), and thereby has a function of executing a program when connected to another device.

<Functional parts of external input device>
Next, each part of the external input device 40 will be described in detail. The transmission / reception unit 41 of the external input device 40 is realized by the network I / F 209 illustrated in FIG. 3, and transmits / receives various data (information) from the communication terminal 10. The connection detection unit 42 detects that data transmission / reception with the communication terminal 10 can be performed by the external device I / F 215. The operation input receiving unit 46 is realized by a keyboard and a mouse and receives an input by a user operation.

  The display data acquisition unit 451 acquires display data of the external input device 40 by capturing the screen of the display device 400, and the display data transmission unit 452 receives the display data acquired by the display data acquisition unit 451. This is transmitted to the communication terminal 10.

  The storage / reading processing unit 49 stores various data using the HDD 205 shown in FIG. 3 as a substance, and reads various data stored in the storage unit 43. The storage unit 43 has the HDD 205 as an entity, and stores a material management table 432, a transition condition table 431, and the like. The display device 400 is realized by the display 208 and performs display on a screen. The menu control unit 44 causes the display device 400 to display the contents of the menu and the sharing state according to the display control flag.

<Material management table>
A material management table 432 is stored in the HDD 205 of the external input device 40.

  FIG. 6 is a diagram showing an example of the material management table 432. In the material management table 432, the material name that is the name of the material, the size that represents the size of the material, the application name that represents the name of the application software used to create the material, and the date and time when the material was last updated , Material data and the like are managed in association with each other.

<Functional configuration of relay device>
Next, functions or means of the relay device 30 will be described. The relay device 30 includes a transmission / reception unit 31, a state detection unit 32, a data quality confirmation unit 33, a change quality management unit 34, a data quality change unit 35, and a storage / read processing unit 39. Each of these units is a function or means realized by any one of the components shown in FIG. 3 operating according to a command from the CPU 201 according to a program stored in the ROM 202. In addition, the relay device 30 includes a storage unit 3000 constructed by the HD 204 illustrated in FIG.

<Change quality control table>
In the storage unit 3000, a change quality management DB 3001 configured by a change quality management table as shown in FIG. In the change quality management table, the IP address of the communication terminal 10 as the relay destination of the image data and the image quality of the image data relayed by the relay device 30 are managed in association with the relay destination.

  The resolution is, for example, 160 pixels in the horizontal direction and 120 pixels in the vertical direction, a low-resolution image that is the base image, a medium-resolution image that is 320 pixels in the horizontal direction and 240 pixels in the vertical direction, 640 pixels in the horizontal direction, Is a high-resolution image composed of 480 pixels. Among these, when passing through a narrow band route, low-quality image data consisting only of low-resolution image data serving as a base image is relayed. When the band is relatively wide, low-resolution image data serving as a base image and medium-quality image data composed of medium-resolution image data are relayed. When the bandwidth is very wide, high-resolution image data composed of low-resolution image data serving as the base image quality, intermediate-resolution image data, and high-resolution image data is relayed. For example, in the modified quality management table shown in FIG. 7, when the relay device 30 relays image data to the destination terminal 10db having the IP address “1.3.2.4”, the relay is performed. The image quality (image quality) of the image data is “high quality”.

<Each functional unit of the relay device>
Next, each functional configuration of the relay device 30 will be described in detail. In the following, in describing each part of the relay device 30, a relationship with main components for realizing each part of the relay device 30 among the respective components illustrated in FIG. 2 will also be described.

  The transmission / reception unit 31 of the relay device 30 shown in FIG. 4 is realized by the network I / F 209 shown in FIG. 3, and communicates with other terminals, devices, or systems via the communication network 2 and various data (information ). The state detection unit 32 is realized by a command from the CPU 201 illustrated in FIG. 3, and detects the operating state of the relay device 30 having the state detection unit 32. The operating state includes “ON line”, “OFF line”, and “failing” state.

  The data quality confirmation unit 33 is realized by a command from the CPU 201 shown in FIG. The data quality confirmation unit 33 searches the change quality management table in FIG. 7 using the IP address of the destination terminal 10B as a search key, and extracts the image quality of the corresponding relayed image data, thereby extracting the relayed image data. Check the image quality. The change quality management unit 34 is realized by a command from the CPU 201 shown in FIG. 3 and changes the contents of the change quality management DB 3001 based on quality information to be described later sent from the remote communication management system 50. . For example, during a video conference by transmitting / receiving high-quality image data between the request source terminal 10aa having the terminal ID “01aa” and the destination terminal 10db having the terminal ID “01db”. When the request source terminal bb and the destination terminal ca that conduct another video conference start a video conference via the communication network 2 or the like, and the reception of image data is delayed at the destination terminal 10db, the relay device No. 30 needs to lower the image quality of image data relayed up to now from high image quality to medium image quality. In such a case, the content of the change quality management DB 3001 is changed so as to lower the image quality of the image data relayed by the relay device 30 from the high image quality to the medium image quality based on the quality information indicating the medium image quality.

  The data quality changing unit 35 is realized by a command from the CPU 201 shown in FIG. 3, and the image quality of the image data sent from the transmission source terminal 10 is based on the contents of the changed change quality management DB 3001. To change. The storage / reading processing unit 39 is realized by the HDD 205 illustrated in FIG. 3, and performs processing of storing various data in the storage unit 3000 and reading out various data stored in the storage unit 3000.

<Functional configuration of remote communication management system>
Next, functions or means of the remote communication management system 50 will be described. The remote communication management system 50 includes a transmission / reception unit 51, a terminal authentication unit 52, a state management unit 53, a terminal extraction unit 54, a terminal state acquisition unit 55, a narrowing unit 56, a session management unit 57, a quality determination unit 58, a storage / readout A processing unit 59 and a delay time management unit 60 are included. Each of these units is a function or means realized by any one of the components shown in FIG. 3 operating according to a command from the CPU 201 according to a program stored in the ROM 202. Further, the remote communication management system 50 has a storage unit 5000 constructed by the HD 204 shown in FIG.

<Relay device management table>
In the storage unit 5000, a relay device management DB 5001 configured by a relay device management table as shown in FIG. 8 is constructed. In this relay device management table, for each relay device ID of each relay device 30, the operating status of each relay device 30, the reception date and time when the status information indicating the operating status is received by the remote communication management system 50, the relay device 30 The IP address and the maximum data transmission rate (Mbps) in the relay device 30 are associated and managed. For example, in the relay device management table shown in FIG. 8, the relay device 30a with the relay device ID “111a” has the operating state “ON line” and the date and time when the remote communication management system 50 received the status information. Is “November 10, 2009, 13:00”, the IP address of the relay device 30a is “1.2.1.2”, and the maximum data transmission rate in the relay device 30a is 100 Mbps. It is shown.

<Terminal authentication management table>
Furthermore, in the storage unit 5000, a terminal authentication management DB 5002 configured by a terminal authentication management table as shown in FIG. In this terminal authentication management table, each password is associated with each terminal ID of all communication terminals 10 managed by the remote communication management system 50 and managed. For example, the terminal authentication management table shown in FIG. 9 indicates that the terminal ID of the communication terminal 10aa is “01aa” and the password is “aaaa”.

<Terminal management table>
Further, in the storage unit 5000, a terminal management DB 5003 configured by a terminal management table as shown in FIG. 10 is constructed. In this terminal management table, for each terminal ID of each communication terminal 10, the operating state of each communication terminal 10, the date and time when the remote communication management system 50 received login request information described later, and the IP address of the communication terminal 10 are Associated and managed. For example, in the terminal management table shown in FIG. 10, the communication terminal 10aa with the terminal ID “01aa” has an operation state of “ON line” and the date and time when the login request information is received by the remote communication management system 50. “13:40 on November 10, 2009” indicates that the IP address of the communication terminal 10aa is “1.2.1.3”.

<Destination list management table>
Further, in the storage unit 5000, a destination list management DB 5004 configured by a destination list management table as shown in FIG. In this destination list management table, all terminal IDs of destination terminals 10B registered as candidates for destination terminal 10B are managed in association with terminal IDs of request source terminal 10A requesting the start of a video conference. For example, in the destination list management table shown in FIG. 11, a candidate for the destination terminal 10B that can request the start of the video conference from the request source terminal 10aa having the terminal ID “01aa” has the terminal ID “01ab”. ", A communication terminal 10ab having a terminal ID" 01ba ", and a communication terminal 10db having a terminal ID" 01db ". The destination terminal 10B candidates are updated by being added or deleted in response to a request for addition or deletion from the request source terminal 10A to the remote communication management system 50.

<Session management table>
Further, in this storage unit 5000, a session management DB 5005 configured by a session management table as shown in FIG. 12 is constructed. In this session management table, for each selection session ID used to execute a session for selecting the relay device 30, the relay device ID and request of the relay device 30 used for relaying image data, audio data, and display data The terminal ID of the original terminal 10A, the terminal ID of the destination terminal 10B, the reception delay time (ms) when image data is received at the destination terminal 10B, and the delay information indicated by this delay time are received from the destination terminal 10B. The reception date and time received and received by the remote communication management system 50 is associated and managed.

  For example, in the session management table shown in FIG. 12, the relay device 30a (relay device ID “111a”) selected in the session executed using the selection session ID “se1” has the terminal ID “01aa”. "And the destination terminal 10db with the terminal ID" 01db "are relayed image data and audio data, and the destination terminal 10db reads" November 10, 2009 14:00 It is shown that the delay time of the image data at the time is 200 (ms). When a video conference is performed between the two communication terminals 10, the reception date and time of the delay information may be managed based on the delay information transmitted from the request source terminal 10A instead of the destination terminal 10B. . However, when a video conference is performed between three or more communication terminals 10, reception of delay information is performed based on the delay information transmitted from the communication terminal 10 on the receiving side of image data, audio data, and display data. Manage date and time.

<Address priority management table>
Further, in the storage unit 5000, a priority management DB 5006 configured by an address priority management table as shown in FIG. 13 is constructed. In this address priority management table, in accordance with the difference in the four dot address (Dot Address) portions of the general IPv4 IP addresses, the address priority points are associated and managed so as to increase. The

  For example, in the address priority management table shown in FIG. 13, when the three values are the same IP address from the top to the bottom of the dot address, the address priority point is “5”. When the two values are the same IP address from the top to the bottom of the dot address, the address priority point is “3”. In this case, whether the value of the lowest dot address is the same or not is not related to the priority. In the case of IP addresses having the same highest value of dot addresses and different second values from the top, the address priority point is “1”. In this case, whether or not the values of the third and lowest dot addresses from the top are the same has no relation to the priority. In the case of IP addresses having different uppermost values of dot addresses, the address priority point is “0”. In this case, whether or not the values of the second, third, and lowest dot addresses from the top are the same has no relation to the priority.

<Transmission speed priority management table>
The priority management DB 5006 built in the storage unit 5000 also includes a transmission speed priority management table as shown in FIG. In this transmission rate priority management table, the transmission rate priority points are associated and managed in accordance with the value of the maximum data transmission rate (Mbps) in the relay device 30 so as to increase.

  For example, in the transmission rate priority management table shown in FIG. 14, when the maximum data transmission rate in the relay device 30 is 1000 Mbps or more, the transmission rate priority point is “5”. When the maximum data transmission rate in the relay device 30 is 100 Mbps or more and less than 1000 Mbps, the transmission rate priority point is “3”. When the maximum data transmission rate in the relay device 30 is 10 Mbps or more and less than 100 Mbps, the transmission rate priority point is “1”. When the maximum data transmission rate in the relay device 30 is less than 10 Mbps, the transmission rate priority point is “0”.

<Quality control table>
Furthermore, in the storage unit 5000, a quality management DB 5007 configured by a quality management table as shown in FIG. In this quality management table, the image quality (image quality) of the image data relayed by the relay device 30 is associated and managed according to the delay time (ms) of the image data at the request source terminal 10A or the destination terminal 10B.

<Each functional unit of remote communication management system>
Next, each functional unit of the remote communication management system 50 will be described in detail. In the following, in describing each part of the remote communication management system 50, among the components shown in FIG. 3, the relationship with the main components for realizing each part of the remote communication management system 50 is also shown. explain.

  The transmission / reception unit 51 is executed by the network I / F 209 illustrated in FIG. 3, and transmits / receives various data (information) to / from other terminals, devices, or systems via the communication network 2. The terminal authentication unit 52 searches the terminal authentication management DB 5002 in the storage unit 5000 using the terminal ID and password included in the login request information received via the transmission / reception unit 51 as search keys, and is identical to the terminal authentication management DB 5002. Terminal authentication is performed by determining whether the terminal ID and password are managed. The state management unit 53 stores the terminal ID of the request source terminal 10A, the operation status of the request source terminal 10A, the remote status in the terminal management table (see FIG. 10) in order to manage the operation status of the request source terminal 10A that has requested login. The communication management system 50 stores and manages the reception date and time when the login request information is received and the IP address of the request source terminal 10A in association with each other.

  The terminal extraction unit 54 searches the destination list management table (see FIG. 11) using the terminal ID of the request source terminal 10A requested to log in as a key, and can select candidate destination terminals 10B that can remotely communicate with the request source terminal 10A. The terminal ID is extracted by reading the terminal ID. Further, the terminal extraction unit 54 searches the destination list management table (see FIG. 11) using the terminal ID of the request source terminal 10A that requested the login as a key, and sets the terminal ID of the request source terminal 10A as a candidate for the destination terminal 10B. As well as the terminal IDs of the other request source terminals 10A registered.

  The terminal status acquisition unit 55 searches the terminal management DB table (see FIG. 10) using the candidate terminal ID of the destination terminal 10B extracted by the terminal extraction unit 54 as a search key, and is extracted by the terminal extraction unit 54. The operating state is read for each terminal ID. Thereby, the terminal state acquisition unit 55 can acquire the operation state of the candidate of the destination terminal 10B that can remotely communicate with the request source terminal 10A that has requested the login. Further, the terminal state acquisition unit 55 searches the terminal management table (see FIG. 10) using the terminal ID extracted by the terminal extraction unit 54 as a search key, and also acquires the operating state of the request source terminal 10A that requested the login. To do.

  The narrowing unit 56 performs a primary narrowing process before the final narrowing process in order to support a final narrowing process that finally narrows down from a plurality of relay apparatuses 30 to one relay apparatus 30. A terminal IP address extraction unit 56b, a primary selection unit 56c, and a priority determination unit 56d are included. Among these, the selection session ID generation unit 56 a generates a selection session ID used for executing a session for selecting the relay device 30. Based on the terminal ID of the request source terminal 10A and the terminal ID of the destination terminal 10B included in the start request information sent from the request source terminal 10A, the terminal IP address extracting unit 56b The IP address of each corresponding communication terminal 10 is extracted. The primary selection unit 56c selects the relay device ID of the relay device 30 whose operation state is “ON line” from among the relay devices 30 managed in the relay device management table (see FIG. 7). The relay device 30 is selected.

  Further, the primary selection unit 56c searches the relay device management table (see FIG. 8) based on the IP address of the request source terminal 10A and the IP address of the destination terminal 10B extracted by the terminal IP address extraction unit 56b. Thus, it is investigated whether the dot addresses of the IP addresses of the selected relay device 30 are the same as or different from the dot addresses in the IP addresses of the request source terminal 10A and the destination terminal 10B. Further, the primary selection unit 56c, for each relay device, the upper two relays having the higher points among the integrated points obtained by integrating the higher points with respect to the communication terminal 10 at the address priority points and the transmission speed priority points. By selecting the device 30, the relay device 30 is further selected.

  In the present embodiment, the top two relay devices 30 with high points are selected. However, the present invention is not limited to this. If one relay device 30 can be narrowed down as many as possible, the top points with high points are selected. Three or more relay devices 30 may be selected.

  The priority determination unit 56d refers to the priority management table (see FIG. 13) and determines an address priority point for each relay device 30 investigated by the primary selection unit 56c. Further, the priority determination unit 56d searches the transmission rate priority management table (see FIG. 14) based on the maximum data transmission rate of each relay device 30 managed in the relay device management table (see FIG. 8). Thus, the transmission speed priority point is determined for each relay device 30 narrowed down by the primary selection unit 56c.

  The session management unit 57 stores the selection session ID generated by the selection session ID generation unit 56a, the terminal ID of the request source terminal, and the terminal ID of the destination terminal in the session management table (see FIG. 12) of the storage unit 5000. Store and manage in association. In addition, the session management unit 57 finally selects one relay device of the relay device 30 selected by the selection processing unit 16 of the communication terminal 10 for each session ID for selection with respect to the session management table (see FIG. 11). Store and manage IDs.

  The quality determining unit 58 searches the quality management table (see FIG. 15) using the delay time as a search key, and extracts the image quality of the corresponding image data, thereby determining the image quality of the image data to be relayed to the relay device 30. To do. The storage / reading processing unit 59 is executed by the HDD 205 shown in FIG. 3, and performs processing of storing various data in the storage unit 5000 and reading out various data stored in the storage unit 5000. The delay time management unit 60 extracts a corresponding terminal ID by searching the terminal management table (see FIG. 10) using the IP address of the destination terminal 10B as a search key, and further extracts a session management table (see FIG. 12). In the session management table (1), the delay time indicated by the delay information is stored and managed in the field of delay time in the record including the extracted terminal ID.

[Display control of display data]
Hereinafter, display control of display data according to this embodiment will be described in detail.

<Allocation information management table>
FIG. 16 is a diagram illustrating an example of the arrangement information management table 23. In the arrangement information management table 23, information for managing the arrangement of display data is registered. “Arrangement information No” is identification information for identifying a record, and “Arrangement information” stores information on how to arrange video data and display data on the display 100. The arrangement information includes “VIEW_MULTI” and “SHARED_MULTI”. As will be described later, “VIEW_MULTI” means that only video data is displayed, and “SHARED_MULTI” means that video data and display data are mixedly displayed.

  Further, the “shared flag” is determined by the arrangement information selection unit 21 with reference to the event flag table 24 of FIG. 20 based on the distribution event. The “location information No” is uniquely determined by the “shared flag”. There are the following two states in which the communication terminal 10 receives display data.

A state where the communication terminal 10 to which the external input device 40 is connected is receiving display data from the external input device 40. A communication terminal 10 is displayed from the external input device 40 connected to another communication terminal 10. State in which data is received In the present embodiment, both of these two states are referred to as “SHARED”. A state in which the communication terminal 10 has not received display data at all is referred to as “UNSHARED”. If the shared flag of the communication terminal 10 is “SHARED”, the arrangement information management table 23 is associated with the arrangement information “SHARED_MULTI” of the arrangement information No 2, and “VIEW_MULTI” of the arrangement information No 1 if the sharing flag is “UNSHARED”. It is linked with the arrangement information of.

  Note that the association between the shared flag and the arrangement information is an example, and in an extreme case, the association can be reversed. Further, the two states in which the communication terminal 10 receives the display data are not set to the same “SHARED”, but the two reception states can be set to “SHARED1” and “SHARED2”, and can be associated with different arrangement information. Setting of the arrangement information management table 23 can be performed by each participant of the communication terminal.

  FIG. 17 is an example of a diagram illustrating an example of division of the display 100 and an area generated by the division. Since different video data or display data can be displayed in each of the areas 1 to 4, up to four video data and display data can be displayed in this example. In FIG. 17, the area 1 is allocated wider than the other areas 2 to 4. The “placement information” determines which area of the areas 1 to 4 the video data is displayed and in which area the display data is displayed.

  FIG. 18 is a diagram illustrating an example of the relationship between the arrangement information and video data or display data assigned to each of the areas 1 to 4. As shown in FIG. 18, when arrangement information = “VIEW_MULTI”, video data 1 is displayed in area 1, video data 2 is displayed in area 2, and video data is displayed in all areas 1 to 4. .

  When arrangement information = “SHARED_MULTI”, display data is displayed in area 1, video data 1 is displayed in area 2, and video data 2 is displayed in area 3.

  The arrangement information will be described in detail.

In the case of “VIEW_MULTI” This is arrangement information in which video data is allocated by the number of video data received by each terminal in order from area 1. For example, when two communication terminals 10 are connected to the remote communication system 1, video data is displayed up to area 2, and nothing is displayed in areas 3 and 4. In the area 1, the communication terminal 10 displays the video data of the other party to which the participant is currently interested, and the video data is taken by the imaging unit 14a of the communication terminal 10 on the utterance side. In order to identify this video data, when the audio input unit 15a of each communication terminal detects an utterance based on the audio level, the display data control unit 22 transfers the terminal ID of the corresponding communication terminal 10 to the relay device 30. Since the relay device 30 broadcasts this terminal ID to each communication terminal 10, each communication terminal 10 can specify video data to be displayed in the area 1. The display data control unit 22 may transmit the terminal ID of the corresponding communication terminal 10 instead of the relay device 30 to the remote communication management system 50. In the area 2, video data on the side not speaking is displayed.

In the case of “SHARED_MULTI” The display data is displayed in the area 1 and the arrangement information in which the video data is allocated to the other areas 2 to 4. When the two communication terminals 1 and 2 are connected to the remote communication system 1, the video data of each communication terminal 10 is displayed in the area 2 and the area 3, and nothing is displayed in the area 4.

  The arrangement information selection unit 21 determines whether or not display data is received from the external input device 40 connected to the communication terminal 10 of its own device, and whether or not display data is received from another communication terminal. Depending on the distribution event that will occur, the sharing flag of the own device is set to “SHARED” or “UNSHARED”. Setting to “SHARED” assigns “SHARED”, “1” or “ON” to the value of the shared flag, and setting to “UNSHARED” means “UNSHARED”, “0” or “ Substituting “OFF”.

  As described above, when the arrangement information selection unit 21 updates the shared flag, based on the result of referring to the sharing flag in the arrangement information management table 23, the image display control unit 14b is moved to the video data and display data areas 1-4. Change the assignment of.

  FIG. 19 is an example of a state transition diagram of arrangement information selected by the arrangement information selection unit 21. Whether the initial state is No1 or No2 in the arrangement information depends on whether display data is being transmitted to the communication terminal 10 immediately after startup, which is the own device. As illustrated in FIG. 19, when the arrangement information selection unit 21 sets “SHARED” in the shared flag in a state where the arrangement information No1 is selected, the arrangement information selection unit 21 selects the arrangement information No2. When the arrangement information No2 is selected and the arrangement information selection unit 21 sets “UNSHARED” in the shared flag, the arrangement information selection unit 21 selects the arrangement information No1. When the arrangement information No1 is selected and the arrangement information selection unit 21 sets “UNSHARED” in the shared flag, the arrangement information selection unit 21 selects the arrangement information No1 as it is and the arrangement information No2 is selected. When the arrangement information selection unit 21 sets “SHARED” as the shared flag, the arrangement information selection unit 21 selects the arrangement information No2 as it is.

<Display control flag>
The display control flag is used for the communication terminal 10 to control display of a menu or the like displayed on the display device 400 of the external input device 40 by the external input device 40.

  FIG. 20 illustrates an example of correspondence between a distribution event transmitted from the relay device 30, a display flag set by the arrangement information selection unit 21 according to the distribution event, and a display control flag determined by the display data control unit 22 (event). Flag table 24).

  Distribution events include “distribution start event”, “distribution start event”, “distribution stop event”, and “distribution stop event from other”. The sharing flag of the former two distribution events is “SHARED” corresponding to the distribution event in which the communication terminal 10 receives the display data, and the communication flag of the latter two distribution events is that the communication terminal 10 does not receive the display data. “UNSHAED” corresponding to the delivery event.

  Also, the display control flag corresponding to “delivery start event” is “SHARED”, the display control flag corresponding to “delivery start event from other” is “SHARED_ANOTHER”, and “delivery start event” and “from other” The display control flag corresponding to “Delivery stop event” is “UNSHARED”. The display control flag will be described below. When receiving the display control flag from the communication terminal 10, the menu control unit 44 of the external input device 40 changes the menu displayed on the display device 400.

  FIG. 21A is an example of a diagram that schematically illustrates an example of the transition condition table 431. The display device 400 immediately after startup displays “Start sharing” on the assumption that the communication terminal 10 does not share display data. “Start sharing” is a menu for starting to share display data of the external input device 40 with a communication terminal 10 other than the communication terminal connected to the external input device 40, and specifically accepts an operation of a participant. This is a “share start” button (hereinafter, “share start” and “share start” buttons are used strictly without being properly used).

When the external input device 40 receives the display control flag “SHARED” while “sharing start” is displayed, the menu control unit 44 displays “sharing stop” for canceling sharing of display data. “Stop sharing” is a menu for stopping sharing display data displayed by the external input device 40 with the communication terminal 10 that is not directly connected to the external input device 40. “Stop sharing” button that accepts the user's operation (hereinafter, “stop sharing” and “stop sharing” buttons are used strictly without being used properly)
If the external input device 40 receives a display control flag other than “SHARED” (“UNSHARED”, “SHARED_ANOTHER”) while “sharing is stopped” is displayed, the menu control unit 44 “sharing” so that the display data can be shared. “Start” is displayed.

  When the external input device 40 receives a display control flag other than “SHARED” while “sharing start” is being displayed, the menu remains unchanged, and the display control flag is displayed while the external input device 40 is displaying “sharing stop”. If “SHARED” is received, the menu remains unchanged.

  As described above, since the sharing flag and the display control flag are associated with the distribution event, the display of the external input device 40 is displayed according to the display mode of the display data and video data displayed on the display 100 by the communication terminal 10. The display mode (menu) of the device 400 can also be determined.

  Then, while the menu is displayed on the display device 400 of the external input device 40, the menu control unit 44 displays the shared state according to the display control flag.

  FIG. 21B is a diagram illustrating an example of a correspondence relationship between the display control flag and the shared state. Such a correspondence relationship is stored in the storage unit 43 together with the transition condition table 431.

  As will be described later, this “shared status” is displayed in the message field on the screen of the external input device 40, so that the participant can grasp the “current shared status” of the communication terminal 10. "Shared state" has the following states.

(A) Communication terminal 10 does not share display data (UNSHARED)
(B) The communication terminal 10 shares the display data displayed by the external input device 40 connected to its own device (SHARED)
(C) The communication terminal 10 shares display data displayed by the external input device 40 connected to the other communication terminal 10 (SHARED_ANOTHER)
[About quality judgment of display data]
Hereinafter, quality judgment of display data according to the present embodiment will be described in detail.

(Quality synchronization criteria)
FIG. 22 is a diagram illustrating an example of the quality synchronization determination standard 26. In the arrangement information management table 23, item information for performing comparison with respect to target quality information is registered. “Target quality information” is identification information for identifying attributes such as target quality. The target quality information includes “capture of content itself”, “OCR (Optical Character Recognition) result”, “color distribution”, “frequency component”, and “edge information”. Furthermore, “capturing the content itself” includes “text subject” and “screen subject”.

  FIG. 23 shows content mainly composed of character information and content mainly composed of drawing information. Content shared between communication terminals is content mainly composed of character information and content mainly composed of drawing information. Content mainly composed of character information is content mainly composed of characters, and is mainly created by word processing software or spreadsheet software. Content mainly composed of drawing information is content mainly composed of drawings, and is mainly created by presentation software or drafting software. The content shared between communication terminals is not limited to content mainly composed of character information and content mainly composed of drawing information as long as the content can be shared between communication terminals. Furthermore, the content shared between the communication terminals may be a still image or a moving image.

  Capturing the text-based content itself means storing mainly the text portion of the display data as data. Capturing the drawing-based content itself means storing mainly the drawing (image portion) of the display data as data.

  FIG. 24 shows an example of the OCR result. In FIG. 24, the left figure shows information obtained by capturing transmitted / received content itself, and the right figure shows information obtained by performing OCR on the transmitted / received content. The OCR result means the result of character recognition.

  FIG. 25 shows an example of “color distribution”, “frequency component”, and “edge information”. In FIG. 25, the left figure is information obtained by capturing the transmitted / received content itself, the upper right figure shows information obtained by extracting the color distribution from the transmitted / received content, and the middle right figure shows information obtained by converting the transmitted / received content into frequency components. The lower right diagram shows information obtained by extracting the edge portion from the transmitted / received content.

  The color distribution means a color distribution of display data. The frequency component means a frequency component obtained when an image is converted into the spatial frequency domain. The edge information means an edge detected from display data.

  The “determination item” is information indicating what kind of comparison is performed on the target quality information.

  Judgment items for capturing the text-based content itself include “color difference in the entire image” and “difference in pixel value in the entire image”. The color difference in the entire image means the color difference recognized in the character portion of the display data. The difference in pixel value in the entire image means the difference in the value of each pixel in the character portion of the display data.

  The determination items for capturing the screen-based content itself include “color difference in the entire image” and “difference in pixel value in the entire image”. The color difference in the entire image means a color difference recognized in the image portion of the display data. The difference in pixel value in the entire image means the difference in the value of each pixel in the image portion of the display data.

  Judgment items of the OCR result include “character recognition rate” and “character recognition rate”. The character recognition rate indicates the recognition rate of characters recognized by OCR. The misrecognition rate of characters indicates the misrecognition rate of characters recognized by OCR.

  The judgment items of the color distribution include “distribution shift (color tone change)” and “distribution range difference (saturation change)”. A deviation in distribution means a deviation in color distribution of display data. The difference in the distribution range means a difference in the color distribution range of the display data.

  The determination items of the frequency component include “change in low frequency component (edge enhancement / blurring increase / decrease)” and “change in high frequency component (edge enhancement / blurring increase / decrease)”. The change of the low frequency component means the degree of edge enhancement in the low frequency part and the increase / decrease in the blur of the edge part in the frequency domain obtained by converting the image into the spatial frequency domain. The change of the high frequency component means the degree of edge enhancement in the high frequency part and the increase / decrease in the blur of the edge part in the frequency domain obtained by converting the image into the spatial frequency domain.

  The judgment item of the edge information includes “edge portion increase / decrease (edge enhancement / blurring increase / decrease)”. The increase / decrease in the edge portion means the degree of edge enhancement of the edge portion detected from the display data and the increase / decrease in the blur of the edge portion.

  The “judgment criterion” stores information indicating how much a change is made or a difference is made with respect to the judgment item and the quality is judged to be synchronized.

  Criteria for capturing text-based content itself include “a color difference within the entire image is within 500” and “a pixel value difference within the entire image is within 500”. That the color difference in the entire image is within 500 means that it can be determined that the quality difference is small when the color difference recognized in the character portion of the display data is within 500. The difference in the pixel values in the entire image is within 500 means that it can be determined that the difference in quality is small when the difference in the value of each pixel in the character portion of the display data is within 500.

  The criteria for capturing the screen-based content itself may be “the color difference of the entire image is within 500” or “the pixel value difference of the entire image is within 500”. The color difference of 500 or less on the entire screen means that it can be determined that the quality difference is small when the color difference recognized in the image portion of the display data is 500 or less. The difference in the pixel values in the entire image is within 500 means that it can be determined that the difference in quality is small when the difference in the value of each pixel in the image portion of the display data is within 500.

  The criteria for determining the OCR result may be “a difference in character recognition rate is within 5%” and “a difference in character recognition rate is within 5%”. A difference in the recognition rate of characters within 5% means that it can be determined that the difference in quality is small when the difference in recognition rates of characters recognized by OCR is within 5%. The difference in the misrecognition rate of characters within 5% means that it can be determined that the difference in quality is small when the difference in the misrecognition rate of characters recognized by OCR is within 5%.

  The criteria for determining the color distribution are “there is no difference in the main color direction with respect to the white point” and “the area difference of the distribution with respect to the white point is small”. The fact that there is no difference in the main color direction with respect to the white point means that when there is no difference in the color distribution in a predetermined direction with respect to the white point when judging the color distribution deviation of the display data. It means that it can be judged that the difference in quality is small. The small area difference of the distribution with respect to the white point means that the quality difference is small when there is no difference in the area of the color distribution with respect to the white point when judging the color distribution deviation of the display data. It can be judged.

  The criterion for determining the frequency component may be “the difference in amplitude of the low frequency component is within 10%” or “the difference in amplitude of the high frequency component is within 10%”. That the difference in amplitude of the low frequency component is within 10% can be judged that the difference in quality is small when the amplitude difference in the low frequency part is within 10% in the frequency domain obtained by converting the image into the spatial frequency domain. Means. The difference in amplitude of the high frequency component is within 10% is that it can be determined that the difference in quality is small when the amplitude difference in the high frequency part is within 10% in the frequency domain obtained by converting the image into the spatial frequency domain. Means.

  The criterion for determining edge information may be “the difference in pixel values of edge information is within 100”. The difference between the pixel values of the edge information being within 100 means that the difference in quality can be determined to be small when the difference between the pixel values of the edge portion detected from the display data is within 100.

  The above-described determination criterion is an example, and a determination criterion different from the determination criterion illustrated in FIG. 22 may be set.

  A case where the communication terminal 10 to which the external input device 40 is connected receives display data from the external input device 40 and transmits it to another communication terminal will be described.

  The communication terminal 10 transmits display data to other communication terminals.

  The display image analysis unit 14c of the communication terminal 10 analyzes the quality of the display data transmitted to other communication terminals for the item information corresponding to the target quality information, and obtains an analysis result (first analysis result). The target quality information is set in advance.

  Other communication terminals receive display data from the communication terminal 10. The display image analysis unit 14c of another communication terminal analyzes the quality of the display data for the item information corresponding to the target quality information, and obtains an analysis result (second analysis result). The target quality information is set in advance. Other communication terminals transmit the second analysis result to the communication terminal 10.

  The communication terminal 10 receives the second analysis result from another communication terminal. The quality synchronization determination unit 14d of the communication terminal 10 compares the second analysis result from another communication terminal with the first analysis result obtained by the display image analysis unit 14c, and based on the quality synchronization determination reference 26. It is determined whether the quality is synchronized by determining whether the difference in quality is small.

  The communication terminal 10 continues to transmit data when it is determined that the difference in quality is small, and transmits a predetermined alarm when it is determined that the difference in quality is large.

[Operation procedure]
FIG. 26 is an example of a flowchart illustrating a process for establishing a session between communication terminals. When the participant presses the operation button 108 shown in FIG. 2 to select the communication terminal 1, the operation input receiving unit 12 shown in FIG. 4 issues a request to start remote communication with the communication terminal 2. Accept.

  S1: The transmission / reception unit 11 of the communication terminal 1 transmits start request information to the remote communication management system 50. The start request information includes the terminal ID “01aa” of the requesting communication terminal 1 and the terminal ID “01db” of the destination communication terminal 2. Thus, the transmission / reception unit 51 of the remote communication management system 50 receives the start request information and grasps the IP address “1.2.1.3” of the communication terminal 1 that is the transmission source.

  S2: The terminal IP address extraction unit 56b of the remote communication management system 50 includes the terminal ID of the request source communication terminal 2 included in the start request information sent from the request source communication terminal 1 and the destination communication. By searching the terminal management table (see FIG. 10) based on the terminal ID of the terminal 2, the IP addresses of the corresponding communication terminals 1 and 2 are extracted.

  S3: Next, the transmission / reception unit 51 of the remote communication management system 50 transmits relay start request information indicating a request to start relaying to the relay device 30 via the communication network 2. The relay start request information includes the IP addresses (“1.2.1.3” and “1.3.2.4”) of the request source communication terminal 1 and the destination communication terminal 2 to be relayed. It is. As a result, the relay device 30 establishes a session for remotely communicating the three image data of low resolution, medium resolution, and high resolution and the audio data between the communication terminals 1 and 2. Thereby, the communication terminals 1 and 2 can start a video conference.

  S4: The relay device 30 establishes a session for remotely communicating image data and audio data between the communication terminals 1 and 2. Specifically, the communication terminal 1 acquires the IP addresses of the relay device 30 and the communication terminal 2, and the communication terminal 2 acquires the IP addresses of the relay device 30 and the communication terminal 1.

  S5: The transmission / reception unit 11 of the communication terminal 1 transmits a session establishment event to the external input device 40. The session establishment event is information for notifying that a session has been established between communication terminals.

  FIG. 27 is an example of a sequence diagram illustrating processing for transmitting and receiving image data and audio data between the communication terminals 10. A process of taking out two (communication terminal 1 and communication terminal 2) from the communication terminal 10 and transmitting and receiving image data and audio data to perform a video conference between the two terminals will be described.

  First, the communication terminal 1 transmits the image data of the subject imaged by the imaging unit 14a and the audio data of the audio input by the audio input unit 15a from the transmission / reception unit 11 to the relay device 30 via the communication network 2. (Step S81). In the present embodiment, high-quality image data including three resolutions of low resolution, medium resolution, and high resolution, and audio data are transmitted. Thereby, in the relay device 30, the transmission / reception unit 31 receives the image data and the audio data of the three resolutions. Then, the data quality confirmation unit 33 searches the change quality management table (see FIG. 7) using the IP address “1.3.2.4” of the communication terminal 2 as a search key, and the image quality of the corresponding relayed image data Is extracted to check the quality of the image data to be relayed (step S82). In the present embodiment, the image quality of the confirmed image data is “high image quality” and is the same as the image quality of the image data received by the transmission / reception unit 31. The voice data is transferred to the communication terminal 2 (step S83). As a result, the communication terminal 2 receives the image data and the audio data at the transmission / reception unit 11, the image display control unit 14b displays an image based on the image data on the display 100, and the audio output unit 15b is based on the audio data. Audio can be output.

  Although not shown, the relay device 30 may return the video data to the communication terminal 1 regardless of whether the quality is changed or whether the quality is changed. By doing so, the video displayed on the display 100 by the communication terminal 1 can be made the same quality as the other communication terminals 2.

  Next, the delay detection unit 17 of the communication terminal 2 detects the reception delay time of the image data received by the transmission / reception unit 11 at regular time intervals (for example, every second) (step S84). In the present embodiment, the following description is continued for a case where the delay time is 200 (ms).

  The transmission / reception unit 11 of the communication terminal 2 transmits delay information indicating the delay time “200 (ms)” to the remote communication management system 50 via the communication network 2 (step S85).

  Accordingly, the remote communication management system 50 can grasp the delay time and the IP address “1.3.2.4” of the communication terminal 2 that is the transmission source of the delay information.

  Next, the delay time management unit 60 of the remote communication management system 50 searches the terminal management table (see FIG. 10) using the IP address “1.3.2.4” of the communication terminal 2 as a search key. The corresponding terminal ID “01db” is extracted, and further, in the session management table (see FIG. 12) of the session management DB 5005, the delay time field portion of the record of the terminal ID “01db” is indicated by the delay information. The stored delay time “200 (ms)” is stored and managed (step S86).

  Next, the quality determination unit 58 searches the quality management table (see FIG. 14) using the delay time “200 (ms)” as a search key, and extracts the image quality “medium image quality” of the corresponding image data. The image quality is determined as “medium image quality” (step S87).

  Next, the transmission / reception unit 51 uses the relay device ID “111a” associated with the terminal ID “01db” in the session management table (see FIG. 12) as a search key, and the relay device management DB table (see FIG. 8). And the IP address “1.2.1.2” of the corresponding relay device 30 is extracted (step S88). Then, the transmission / reception unit 51 transmits quality information indicating the image quality “medium image quality” of the image data determined in step S87 to the relay apparatus 30 via the communication network 2 (step S89). This quality information includes the IP address “1.3.2.4” of the communication terminal 2 used as the search key in step S86. Thereby, in the relay device 30, the change quality management unit 34 adds the IP address “1.3.2.” Of the destination communication terminal 10 (here, the communication terminal 2) to the change quality management table (see FIG. 7). 4 ”and the image quality“ medium image quality ”of the image data to be relayed are stored and managed in association with each other (step S90).

  Next, the communication terminal 1 continues to transmit the high-quality image data including the low image quality, the medium image quality, and the high image quality, and the audio data to the relay device 30 as in step S81 (step S81). S91). As a result, in the relay device 30, as in step S82, the data quality confirmation unit 33 uses the IP address “1.3.2.4” of the communication terminal 2 as a search key to change the quality management table (see FIG. 7). ) And the image quality “medium image quality” of the corresponding image data to be relayed is extracted to check the image quality of the image data to be relayed (step S92). In this embodiment, since the image quality of the confirmed image data is “medium image quality” and lower than the image quality “high image quality” of the image data received by the transmission / reception unit 31, the data quality changing unit 35 The image quality of the image data is changed by suppressing the image quality from “high image quality” to “medium image quality” (step S93). Then, the transmitting / receiving unit 31 transmits the image data in which the image quality of the image data is changed to “medium image quality” and the audio data in which the sound quality of the audio is not changed to the communication terminal 2 via the communication network 2 ( Step S94). As described above, when a delay in reception occurs in the communication terminal 2 that receives the image data, the relay device 30 changes the quality of the image so that the person participating in the video conference does not feel uncomfortable. can do.

<Send display data>
FIG. 28A is an example of a flowchart of a procedure for transmitting display data from the external input device 40 to the communication terminal.

  When the external input device 40 and the communication terminal 10 are connected to each other, the connection detection unit 42 of the external input device 40 detects that the communication terminal 10 is connected (S10). The connection detection unit 42 detects that data transmission / reception with the communication terminal 10 is possible because, for example, the external device I / F 215 detects a voltage or a signal.

  The connection detection unit 42 causes the display device 400 to display a menu for accepting the start of sharing display data. When the participant presses this menu, the operation input accepting unit 46 accepts an operation for starting sharing (S20).

  When the operation input accepting unit 46 accepts an operation to start sharing, the display data obtaining unit 451 obtains display data in order to transmit display data (S30). The display data changes because it is image data used for a video conference. For this reason, the display data acquisition unit 451 acquires display data periodically (for example, once per second) or by detecting a change in image data on the screen.

  When the display data acquisition unit 451 acquires the display data, the display data transmission unit 452 transmits the display data acquired by the display data acquisition unit 451 to the communication terminal 2 via the communication terminal 1 (S40).

  All the external input devices 40 connected to each communication terminal 10 participating in the video conference (for example, registered in the destination list management table) may transmit display data. Send display data.

  FIG. 28B is another example of a flowchart of a procedure for transmitting display data from the external input device 40 to the communication terminal. In FIG. 28B, it is confirmed that a session with the destination communication terminal 2 is established, and the display data is transmitted to the destination communication terminal.

  When the external input device 40 and the communication terminal 10 are connected to each other, the connection detection unit 42 of the external input device 40 detects that the communication terminal 10 is connected (S10). The connection detection unit 42 detects that data transmission / reception with the communication terminal 10 is possible because, for example, the external device I / F 215 detects a voltage or a signal.

  When the connection detection unit 42 detects that the communication terminal 10 is connected, the menu control unit 44 of the external input device 1 displays the screen shown in FIG. 32A on the display device 400 (S12). Each screen in FIG. 32 shows an example of a screen displayed on the external input device 1. As shown in FIG. 32A, no characters are displayed on the menu 401, and a message “Conference is not taking place” is displayed on the menu 402. This message indicates that no session has been established. Thereby, the participant can grasp that the conference is not being held.

  The menu control unit 44 determines whether or not a session establishment event is notified from the communication terminal 1 (S14). This session establishment event corresponds to S5 in FIG. 26, and is notified from the communication terminal 10 to the external input device 40 when it is detected that a session has been established between the communication terminals 1 and 2.

  When the session establishment event is not notified (No in S14), the external input device 40 maintains the screen of FIG.

  When the session establishment event is notified (Yes in S14), the menu control unit 44 of the external input device 40 displays the screen of FIG. 32B (S16).

  In FIG. 32B, a “sharing start” button is displayed as the menu 401, and a message “not sharing” is displayed as the sharing state 402. The screen of FIG. 32B indicates that sharing is not performed in the menu 402 and that the operation input for starting sharing can be performed in the menu 401. The “share start” button can be pressed directly by the participant pressing the touch panel or by operating the keyboard 211 or the mouse 212.

  When the operation input accepting unit 46 accepts an operation to start sharing, the display data obtaining unit 451 obtains display data from the communication terminal 1 in order to transmit display data (S30). The display data changes because it is image data used for a video conference. For this reason, the display data acquisition unit 451 acquires display data periodically (for example, once per second) or by detecting a change in image data on the screen.

  When the display data acquisition unit 451 acquires the display data, the display data transmission unit 452 transmits the display data acquired by the display data acquisition unit 451 to the communication terminal 2 via the communication terminal 1 (S40).

  Note that, when the external input device 40 is connected to the communication terminal 1 in a state where a session has already been established, a session establishment event is notified to the external input device 40 immediately after the connection, so that FIG. 32 (a) The screen of FIG. 32 (b) is displayed by skipping the screen.

  29 shows video data captured by the communication terminal 1, video data captured by the communication terminal 2, and display data transmitted from the external input devices 1 and 2 connected to the communication terminals 1 and 2. It is an example of the sequence diagram of transmission. Although audio data is omitted, it is transmitted and received together with video data.

  S1: The video data captured by the communication terminal 1 is once transmitted to the relay device 30 via the transmission / reception unit 11 by the image display control unit 14b.

  S1-1, S1-2: When the transmission / reception unit 31 of the relay device 30 receives the video data, the transmission / reception unit 31 transmits the video data to the communication terminals 1 and 2 after performing the quality change processing of the video data.

  S2: Similarly, the video data captured by the communication terminal 2 is once transmitted to the relay device 30.

  S2-1, S2-2: The relay device 30 transmits the video data to the communication terminals 1 and 2 through the transmission / reception unit 31 after performing the quality change process of the video data.

S3: When display data is transmitted from the external input device 1, the display data is transmitted to the communication terminal 1 via the transmission / reception unit 41 by the display data transmission unit 452 of the external input device 1. (This process corresponds to the process of step 40 in FIG. 28.)
S3-1: The display data received by the external information transmitting / receiving unit 18 of the communication terminal 1 is once transmitted to the relay device 30 in the same manner as the video data.

  S3-1-1, S3-1-2: The relay device 30 transmits display data to each of the communication terminals 1 and 2.

  S4: Similarly, when display data is transmitted from the external input device 2, the display data is transmitted to the communication terminal 2 via the transmission / reception unit 41 by the display data transmission unit 452 of the external input device 2.

  S4-1: The display data received by the external information transmitting / receiving unit 18 of the communication terminal 2 is once transmitted to the relay device 30 in the same manner as the video data.

  S4-1-1, S4-1-2: The relay device 30 transmits display data to each of the communication terminals 1 and 2.

  The relay device 30 may be fixed or one selected from the plurality of relay devices 30. In FIG. 29, two communication terminals are used for simplicity of explanation, but transmission can be performed in the same procedure with three or more communication terminals. The external input device 1 can also be connected to the communication terminal 2, and the external input device 1 may transmit display data from the communication terminal 2 side.

  Thus, since the remote communication system 1 of this embodiment transmits the display data of the external input device 40 via the communication terminal 10, display data can be obtained only by preparing a network infrastructure technology common to video data. It can be transmitted to another communication terminal 10.

  In addition, regardless of which external input terminal 40 transmits display data, the delay time until reception by each terminal is substantially the same by passing through the relay device 30. For this reason, even if the presenter changes the page of the material, the display content of the communication terminal 10 at each site is changed almost at the same time, so communication between the sites is hardly hindered.

<Display data sharing start processing>
FIG. 30 is an example of a sequence diagram illustrating a control procedure when the external input devices 1 and 2 start transmitting display data in S3 or S4 of FIG.

  By the way, as described above, the display flag until the communication terminal 10 receives the display data is “UNSHARED”. Therefore, only video data is displayed on the screen of the display 100 of the communication terminal 10.

  FIG. 31A is an example of a diagram schematically illustrating a screen state of the display 100 until the communication terminal 10 receives display data. That the display flag of the arrangement information management table 23 is “UNSHARED” is a state in which the arrangement information selection unit 21 has selected the arrangement information No1. For this reason, as shown in FIG. 31A, the video data received from the relay device 30 is displayed in the areas 1 and 2. Whether the placement information selection unit 21 displays the video data of its own device or the video data of another device in the area 1 is as described in the explanation of the placement information management table 23. You may implement | achieve so that it may switch.

  Further, until the communication terminal 10 is notified of the distribution event, the display data control unit 22 does not transmit any display control flag to the external input device 40 or sends a display control flag of “UNSHARED” to the external input device 40. Sending. The menu control unit 44 of the external input device 40 displays a menu corresponding to the display control flag “UNSHARED” on the display device 400 as a state immediately after activation.

  FIG. 32 is a diagram showing several screen examples displayed on the external input device 40. FIG. 32A has already been described. FIG. 32B shows an example of the menu 401 and the sharing state 402 displayed on the external input device 40. In FIG. 32B, a “sharing start” button is displayed as the menu 401, and a message “not sharing” is displayed as the sharing state 402. The “share start” button can be pressed directly by the participant pressing the touch panel or by operating the keyboard 211 or the mouse 212.

  In FIG. 32B, the display data of the material possessed by the external input device 40 is displayed, and in addition, the status display window and the display data of another material are displayed. When the participant shares the display data being displayed by the external communication terminal 40 with another communication terminal 2, the participant presses a “start sharing” button. Thereby, the external input device 40 starts transmission of display data.

Returning to FIG.
S1: The operation input receiving unit 46 receives a press of an item “start sharing” button displayed on the external input device 1 by a participant. This process corresponds to S20 in FIG.

  S2: The display data transmission unit 452 notifies the communication terminal 1 of the start of sharing and starts transmission of display data. This process corresponds to S40 in FIG. 28 and S3 and S4 in FIG.

  S2-1: The display data control unit 22 of the communication terminal 1 requests the relay device 30 to start distribution, and transmits display data to the relay device 30. This process corresponds to S3-1 in FIG.

  S3: The data quality changing unit of the relay device 30 changes the quality of the display data, notifies the communication terminal 1 of the “distribution start event”, and starts distributing the display data. This process corresponds to S3-1-1 and S4-1-1 in FIG.

  S3-1: The arrangement information selection unit 21 of the communication terminal 1 acquires the “distribution start event”, and sets the sharing flag to “SHARED” according to the event flag table 24.

  S3-2: The arrangement information selection unit 21 of the communication terminal 1 selects the arrangement information No2 according to the arrangement information management table 23, and instructs the image display control unit 14b to allocate video data and display data for each of the areas 1 to 4. To do.

  As a result, the state of the display 100 changes from the state shown in FIG. 31A to the state shown in FIG. In FIG. 31B, display data is displayed in area 1 and video data is displayed in areas 2 and 3.

  S3-3: Since the “distribution start event” is notified to the communication terminal 1 in step S3, the display data control unit 22 sets the display control flag to “SHARED” according to the event flag table 24 and sets the external input device 1 is notified.

  S3-3-1: The menu control unit 44 of the external input device 1 determines whether to change the display of the menu 401 according to the display control flag according to the transition condition table 431 of FIG.

  S3-3-2: Since the external input device 2 has received the display control flag “SHARED” while the “sharing start” button is displayed as shown in FIG. The menu 401 of the device 400 is changed to a “stop sharing” button. FIG. 32C shows a screen example of the display device 400 on which the “share stop” button is displayed.

  S3-3-3: Since the menu control unit 44 receives the display control flag “SHARED”, the menu control unit 44 displays “This display is shared” in the sharing state 402 as shown in FIG.

  S4: The data quality changing unit of the relay device 30 changes the quality of the display data, notifies the communication terminal 2 of the “distribution start event from another”, and starts the display data distribution. This process corresponds to S3-1-2 and S4-1-2 in FIG.

  S4-1: The display data control unit 22 of the communication terminal 2 determines whether or not the relay device 30 is requested to stop distribution when the “distribution start event from another” is notified. This determination is necessary to determine which display data is to be displayed when the communication terminal 2 already shares the display data. Although this determination method will be described later, here, since the shared flag is “UNSHARED”, no distribution stop request is made.

  S4-2: When receiving the “distribution start event from another”, the arrangement information selection unit 21 of the communication terminal 2 sets the shared flag of the arrangement information management table 23 to “SHARED” according to the event flag table 24.

  S4-3: The arrangement information selection unit 21 of the communication terminal 2 selects the arrangement information No2 from the arrangement information management table 23 because the sharing flag is set to “SHARED”, and the image display control unit 14b receives the information for each of the areas 1 to 4 Is instructed to assign video data and display data. As a result, the state of the display 100 changes from the state shown in FIG. 31A to the state shown in FIG.

  S4-4: Since the communication terminal 2 has been notified of the “distribution start event from another” in step 4, the display data control unit 22 sets the display control flag to “SHARED_ANOTHER” according to the event flag table 24 and sets the external input device 2 is notified.

  S4-4-1: The menu control unit 44 of the external input device 1 determines whether or not to change the display of the menu 401 according to the display control flag according to the transition condition table 431 of FIG. Since the external input device 2 receives the display control flag “SHARED_ANOTHER” while the “share start” button is displayed as shown in FIG. 32B, the menu control unit 44 displays the menu 401 of the display device 400. Not going to change.

  S4-4-2: The menu control unit 44 refers to the display control flag and displays a sharing state 402 “other displays are shared” on the display device 400.

  FIG. 32D shows an example of a screen displayed on the display device 400 of the external input device 2 when the communication terminal 2 shares the display data of the communication terminal 1. In FIG. 32 (d), the menu 401 is “Start sharing”, but the sharing state 402 displays “Share other displays”.

  As described above, in the external input terminal 1, the menu 401 is changed from the “start sharing” button to the “stop sharing” button, and the sharing state 402 changes from “not shared” to “share this display”. Is changed. In the external input terminal 2, the menu 401 is not changed, and the sharing state 402 is changed from “not shared” to “other display is shared”. Therefore, the participant can share the display data by operating the menu 401, and by viewing the sharing state 402, the communication terminal 10 to be operated is displayed with the external input device 40 connected to the own device. You can see at a glance whether you are sharing data.

<Switching display data to be shared>
A processing procedure for transmitting display data from the external input device 2 after the processing of FIG. 30 will be described. FIG. 31B shows the display 100 of the communication terminals 1 and 2, FIG. 32C shows the display device 400 of the external input device 1, and FIG. 32D shows the display device 400 of the external input device 2. Each is displayed.

  FIG. 33 is an example of a sequence diagram illustrating a control procedure when the external input device 2 starts to transmit display data.

  Description of S1 to S4 is omitted because only the external input device 1 is changed to the external input device 2 in FIG.

  S4-1: The display data control unit 22 of the communication terminal 1 determines whether or not the relay device 30 is requested to stop distribution when the “distribution start event from another” is notified.

  S4-2: As will be described later, since the sharing flag of the communication terminal 1 is “SHARED”, the display data control unit 22 requests the relay device 30 to stop distribution. The data quality changing unit of the relay device 30 performs processing to be described later and stops transmission of display data (stops transmission of display data transmitted in S3-1-1 and S3-1-2 in FIG. 29). .) Here, since the relay device 30 is requested to start distribution from the communication terminal 2, there is another display data distributed by the relay device 30, and therefore no distribution stop event occurs.

  S4-3: The display data control unit 22 of the communication terminal 1 requests the external input device 1 to stop display data transmission in order to respond to the “distribution start event from another”. As described below in S4-6, the display data to be shared can be limited to one. The display control flag “SHARED_ANOTHER” may be used to request stop of display data transmission.

  S4-3-1: The display data transmission unit 452 of the external input device 1 stops the transmission of display data. This stop stops the display data transmission in steps 3 and 3-1 of FIG.

  S4-4: The arrangement information selection unit 21 of the communication terminal 1 receives the “delivery start event from another”, and therefore sets the shared flag to “SHARED” according to the event flag table 24.

  S4-5: The arrangement information selection unit 21 of the communication terminal 1 selects the arrangement information No2 in the arrangement information management table 23, and instructs the image display control unit 14b to associate the areas 1 to 4 with display data and video data.

  S4-6: Further, since “communication start event from another” is notified to the communication terminal 1 in step S4, the display data control unit 22 sets the display control flag “SHARED_ANOTHER” according to the event flag table 24 to the external input device. 1 is notified.

  S4-6-1: The menu control unit 44 of the external input device 1 determines whether or not to change the display of the menu 401 based on the transition condition table 431 in FIG.

  S4-6-2: Since the menu control unit 44 has received the notification of the display control flag “SHARED_ANOTHER” in a state where “sharing stop” is displayed, the menu control unit 44 sets the “sharing start” button on the display device 400. Display.

  S4-6-3: Further, the menu control unit 44 displays a message “other display is shared” in the sharing state 402 on the display device 400 with reference to the display control flag “SHARED_ANOTHER”. With S4-6-2 and S4-6-3, the display device 400 of the external input device 1 becomes as shown in FIG.

  S3-1: On the other hand, when the arrangement information selection unit 21 of the communication terminal 2 acquires the distribution start event, it sets the sharing flag to “SHARED” according to the event flag table 24.

  S3-2: The arrangement information selection unit 21 of the communication terminal 2 selects the arrangement information No2 according to the arrangement information management table 23, and assigns video data and display data to the image display control unit 14b for each of the areas 1 to 4. Since the arrangement information selection unit 21 has already selected the arrangement information No2 in S4-3 of FIG. 30, the arrangement information No2 is selected again.

  S3-3: Since the communication terminal 2 is notified of the distribution start event in step S3, the display data control unit 22 sets the display control flag to “SHARED” according to the event flag table 24 and notifies the external input device 2 To do.

  S3-3-1: The menu control unit 44 of the external input device 2 determines whether to change the display of the menu 401 according to the display control flag according to the transition condition table 431 of FIG.

  S3-3-2: Since the external input device 2 has received the display control flag “SHARED” while the “sharing start” button is displayed as shown in FIG. The menu 401 of the device 400 is changed to a “stop sharing” button.

  S3-3-3: Since the menu control unit 44 receives the display control flag “SHARED”, the menu control unit 44 displays “This display is shared” in the sharing state 402 as shown in FIG. Therefore, the screen of the display device 400 is as shown in FIG.

  As described above, when the processing in FIG. 33 is completed, the display device 400 of the external input terminal 1 becomes as shown in FIG. 32D, and the display device 400 of the external input terminal 2 becomes as shown in FIG. . That is, in the external input terminal 1, the menu 401 is changed from “sharing stop” to “sharing start”, and the sharing state 402 is changed from “sharing this display” to “sharing other displays”. Is done. In the external input terminal 2, the menu 401 is changed from “start sharing” to “stop sharing”, and the sharing state 402 is changed from “sharing other displays” to “sharing this display”. Is done.

  As described in S4-3, the external input device 1 stops the transmission of display data in response to the “distribution start event from other” from the relay device 30, so that the display data displayed by each communication terminal 10 is 1 Can be limited to one.

<Determination of distribution stop request in S4-1 in FIGS. 30 and 33>
FIG. 34 is an example of a flowchart illustrating a procedure for determining whether or not to request a distribution stop when the communication terminal 10 receives a “distribution start event from another”.

  This process is activated when the communication terminal 10 receives a distribution start event from another (S410).

  When receiving the “delivery start event from another”, the display data control unit 22 determines whether or not the sharing flag is set to “SHARED” (S420). If the sharing flag is “SHARED”, the display data distributed by the external input device 1 or other external input device 2 connected to the own device is shared. Can't handle “distribution start event from others”.

  For this reason, when the sharing flag is “SHARED” (Yes in S420), the display data control unit 22 requests the relay device 30 to stop distribution (S430). The relay device 30 stops the distribution of display data to the communication terminal 10 that requested the distribution stop.

<Decision of S4-2 in FIG. 33>
The relay device 30 updates the display flag and the display control flag of the communication terminal 10 in response to a distribution stop request from the communication terminal 10, so that the “distribution stop event” or “delivery stop from other” is sent to all other communication terminals. It is determined whether to notify “event”.

  FIG. 35 is an example of a flowchart illustrating a determination procedure of the relay device 30.

  First, the relay device 30 stops the distribution of the display data requested to stop distribution (S510).

  Next, the relay device 30 determines whether there is still display data being distributed (S520). That is, it is determined whether or not display data is transmitted from the base participating in the video conference to the relay device 30 and is being distributed.

  If there is still display data being distributed (Yes in S520), the relay device 30 does not notify the communication terminal of any “distribution stop event” in order to distribute the display data.

  When there is no more display data being distributed (No in S520), in order to update the display flag and the display control flag of all the communication terminals, the relay device 30 sends a “distribution stop event” to the communication terminal that requested the distribution stop. And the other communication terminal is notified of the “distribution stop event from other” (S530).

  In this way, when there is still display data being distributed, the communication terminals 1 and 2 can accept display data distribution with the sharing flag “SHARED”, and there is no display data being distributed. In this case, the shared control flag “UNSHARED” can be notified to the external input devices 1 and 2.

[Stop sending display data]
Next, a procedure in which the external input device 2 stops transmission of display data in the state where the processing of FIG. 33 has been completed will be described using the sequence diagram of FIG. When the participant wants to stop transmission of display data from the external input device 2 of his / her own device, the participant presses a “stop sharing” button of the menu 401 displayed on the external input device 2.

  S1: The operation input accepting unit 46 of the external input device 2 accepts pressing of the “stop sharing” button.

  S2: The display data transmission unit 452 notifies the communication terminal 2 that transmission is stopped.

  S2-1: The display data control unit 22 of the communication terminal 2 requests the relay device 30 to stop distribution.

  The data quality changing unit of the relay device 30 determines whether or not to notify the “delivery stop event” according to FIG.

  S3, S4: Since the display data already distributed by the relay device 30 is only the display data transmitted from the communication terminal 2, the relay device 30 sends a “distribution stop event” to the communication terminal 2 (S3), “ The “distribution stop event” is distributed to the communication terminal 1 (S4, S530 in FIG. 35). Thereby, the distribution of the display data in S4-1-1 and S4-1-2 in FIG. 29 is stopped. When the relay device 30 transmits the “distribution stop event”, the relay device 30 does not have to transmit the stop of display data distribution in S510 of FIG. 35. A delivery stop may be sent.

  S3-1: When receiving the “distribution stop event”, the display data control unit 22 of the communication terminal 2 requests the external input device 2 to stop transmission of display data.

  S3-1-1: The display data transmission unit 452 of the external input device 2 stops the transmission of display data. Due to this stop, transmission of display data in S-4 and S4-1 in FIG. 29 is stopped. Note that the transmission of display data may be stopped in S2.

  S3-2: The arrangement information selection unit 21 of the communication terminal 2 sets the shared flag to “UNSHARED” according to the event flag table 24 of FIG.

  S3-3: The arrangement information selection unit 21 of the communication terminal 2 selects the arrangement information No1 in the arrangement information management table 23 according to the setting of the shared flag. As a result, the image display control unit 14b assigns video data to the areas 1 and 2 and displays the screen as shown in FIG.

  S3-4: Further, since the “distribution stop event” is notified to the communication terminal 2 in S3, the display data control unit 22 sets the display control flag “UNSHARED” to the external input device 2 based on the event flag table 24. Notice.

  S3-4-1: The menu control unit 44 of the external input device 2 determines whether to change the menu 401 according to the transition condition table 431 of FIG.

  S3-4-2: Since the display control flag “UNSHARED” is received while the external input device 2 displays “stop sharing”, the menu control unit 44 displays the menu 401 on the display device 400 as the “share start” button. Is displayed.

  S3-4-3: Further, the menu control unit 44 displays a message of the sharing state 402 on the display device 400 with reference to the display control flag. When the display control flag is “UNSHARED”, the message is “not shared”.

  S4: Steps 3.2 to 3.4 can be applied to S4.1 to 4.3.1.

  S4-1: When receiving the “delivery stop event from another”, the display data control unit 22 of the communication terminal 2 sets the sharing flag to “UNSHARED” according to the event flag table 24 of FIG. Since display data transmission has already been stopped, there is no need to request the external input device 1 to stop display data transmission.

  S4-2: The arrangement information selection unit 21 of the communication terminal 1 selects the arrangement information No1 in the arrangement information management table 23 according to the setting of the shared flag. As a result, the image display control unit 14b assigns video data to the areas 1 and 2 and displays the screen as shown in FIG.

  S4-3: Since the communication terminal 1 is notified of the “distribution stop event from other” in S4, the display data control unit 22 externally inputs the display control flag “UNSHARED” based on the event flag table 24. Notify the device 1.

  S4-3-1: The menu control unit 44 of the external input device 1 determines whether or not to change the menu 401 according to the transition condition table 431 of FIG.

  S4-3-2: Since the display control flag “UNSHARED” is received while the external input device 1 displays “start sharing”, the menu control unit 44 continues to display the “start sharing” button.

  S4-3-2: Further, the menu control unit 44 displays a message of the shared state 402 on the display device 400 with reference to the display control flag. When the display control flag is “UNSHARED”, the message is “not shared”.

  When the above processing is completed, the display devices 400 of the external input terminal 1 and the external input terminal 2 are both as shown in FIG. In the external input terminal 1, the menu 401 remains “start sharing” and is not changed, but the message in the sharing state 402 is changed from “sharing other displays” to “not shared”. In the external input terminal 2, the menu 401 is changed from “sharing stop” to “sharing start”, and the message of the sharing state 402 is changed from “sharing this display” to “not sharing”. The

<When session is disconnected>
FIG. 37 is an example of a flowchart illustrating a process in which communication terminals disconnect a session. When the participant operates the operation button 108 shown in FIG. 2 to instruct the end of the conference (session disconnection), the operation input reception unit 12 shown in FIG. Accept termination request. The operation input receiving unit 12 notifies the external input device 40 of a session disconnection event. In addition, when the participant disconnects the USB cable connecting the communication terminal 10 and the external input device 40 or when the communication terminal 1 is powered off, the external input device 40 detects or acquires a session disconnection event.

  The menu control unit 44 determines whether the notification of the session disconnection event has been acquired while the screens of FIGS. 32B to 32D are being displayed (S110). When the notification of the session disconnection event has not been acquired (No in S110), the menu control unit 44 maintains the display of the screens of FIGS.

  When the notification of the session disconnection event is acquired (Yes in S110), the menu control unit 44 displays the screen of FIG. 32A (S120). This screen is a screen for notifying participants that a conference is not being held. The participant can grasp that it is not necessary to operate the external input device 40 by the message “the conference is not being performed”.

  If the display data transmission unit 452 of the external input device 40 is transmitting display data, the display data transmission unit 452 stops transmitting display data to the communication terminal 2 via the communication terminal 1 (S130).

  Further, the display data acquisition unit 451 stops acquiring display data from the communication terminal 1 or 2 (S140).

  Since the remote communication system 1 of the present embodiment can display a menu 401 for starting display data sharing on the external input device 40, a participant in a video conference who wants to share the display data with other bases is “shared”. Display data can be displayed at other locations by a simple operation of pressing the “Start” button. Therefore, participants at other sites can specify display data to be noted without performing any operation such as selecting display data to be shared.

  Furthermore, since the external input device 40 displays the message of the sharing state 402, the conference participants can reliably check whether the display data currently displayed on the display device 400 by the external input device 40 is shared. I can know. For example, when a plurality of external input devices 40 displaying the same material are connected to a communication terminal at a plurality of bases, the participant can immediately determine which base material is shared, and the operability can be improved. It can be improved.

  Further, when the communication terminal 10 receives a notification of a distribution event from another as in the processing of FIG. 34, the external input device 40 connected to its own device requests the distribution stop, so that the display data that can be shared is 1 Can be limited to one. This eliminates the need for the participant to select display data to be noticed from among a plurality of display data, thereby improving operability.

[About quality judgment of display data]
FIG. 38 shows an embodiment of display data quality judgment processing.

  A case where the communication terminal 10 to which the external input device 40 is connected receives display data from the external input device 40 and transmits it to another communication terminal will be described. The present invention can also be applied to a case where display data created by the communication terminal 10 is transmitted to another communication terminal.

  The communication terminal 10 transmits display data to other communication terminals.

  Other communication terminals receive display data from the communication terminal 10. The display image analysis unit 14c of another communication terminal analyzes the quality of the display data for the item information corresponding to the target quality information, and obtains an analysis result (second analysis result).

  S3802: The display image analysis unit 14c of another communication terminal starts content classification when obtaining the second analysis result.

  S3804: The display image analysis unit 14c of another communication terminal performs OCR of the display data.

  S3806: The display image analysis unit 14c of the other communication terminal performs frequency component extraction of the display data.

  S3808: The display image analysis unit 14c of another communication terminal performs color distribution extraction of display data.

  S3810: The display image analysis unit 14c of another communication terminal classifies the content based on the OCR execution result in S3904 and the color distribution extraction result in S3906.

<Content classification>
FIG. 39 shows an example of content classification. The display image analysis unit 14c includes a content classification table used when quality of display data is determined.

  “Classification of content to be judged” is content type information for which quality judgment is performed. The contents to be judged are classified into “character subject (monochrome)”, “character subject (color)”, “drawing subject (monochrome)”, “drawing subject (color)”, and “character + drawing (monochrome)”. And "character + drawing (color)". These are examples, and may be classified differently.

  Character-based (monochrome) is character-based content and display data is composed of black and white. Character-based (color) is character-based content, and display data is composed of color.

  Drawing-based (monochrome) is content mainly composed of image information, and display data is composed of black and white. The drawing main body (color) is content mainly including image information, and the display data is composed of color.

  Character + drawing (monochrome) is a content made up of characters and images, and display data is made up of black and white. Character + drawing (color) is a content composed of characters and images, and display data is composed of colors.

  The “OCR result” is the presence or absence of characters that can be extracted by OCR.

  The character-based (monochrome) OCR result is “Yes”. This is because it is assumed that characters can be extracted by OCR because the content is mainly text. The character-based (color) OCR result is “Yes”. This is because it is assumed that characters can be extracted by OCR because the content is mainly text.

  The OCR result for drawings (monochrome) is “none”. This is because it is assumed that it is impossible to extract characters by OCR because it is image-based content. The OCR result of the drawing subject (color) is “none”. This is because it is assumed that it is impossible to extract characters by OCR because it is image-based content.

  The OCR result for text + drawing (monochrome) is “Yes”. This is because the content is composed of characters and images, and it is assumed that characters can be extracted by OCR. The OCR result of text + drawing (color) is “Yes”. This is because the content is composed of characters and images, and it is assumed that characters can be extracted by OCR.

  The “frequency component extraction result” is frequency component distribution information.

  The result of extracting the frequency component of the main character (monochrome) is “only high frequency component”. This is because it is assumed that only high-frequency components can be obtained because the color change is large because the content is mainly text. The extraction result of the frequency component of the main character (color) is “only high frequency component”. This is because it is assumed that only high-frequency components can be obtained because the color change is large because the content is mainly text.

  The extraction result of the frequency component of the drawing main body (monochrome) is “high frequency component only” or “high frequency component + low frequency component”. This is because the content is mainly an image, and therefore, the content may consist of only a portion with a large color change, or a portion with a large color change and a portion with a small color change. The extraction result of the frequency component of the drawing main body (color) is “high frequency component only” or “high frequency component + low frequency component”. This is because the content is mainly an image, and therefore, the content may consist of only a portion with a large color change, or a portion with a large color change and a portion with a small color change.

  The extraction result of the frequency component of the character + drawing (monochrome) is “high frequency component + low frequency component”. This is because the content is composed of characters and images, and there may be a portion with a large color change and a portion with a small color change. The extraction result of the frequency component of the character + drawing (color) is “high frequency component + low frequency component”. This is because the content is composed of characters and images, and there may be a portion with a large color change and a portion with a small color change.

  The “color distribution extraction result” is information indicating the color distribution of the display data.

  The extraction result of the character-based (monochrome) color distribution is “no distribution”. This is because the change in color is large and there is no color distribution because it is monochrome. However, luminance information may be obtained. The extraction result of the color distribution of the main character (color) is “with distribution”. This is because the color is assumed to change.

  The result of extracting the color distribution of the drawing main body (monochrome) is “no distribution”. This is because the change in color is large and there is no color distribution because it is monochrome. However, luminance information may be obtained. The extraction result of the color distribution of the drawing main body (color) is “with distribution”. This is because the color is assumed to change.

  The extraction result of the character + drawing (monochrome) color distribution is “no distribution”. This is because the change in color is large and there is no color distribution because it is monochrome. However, luminance information may be obtained. The extraction result of the color distribution of the character + drawing (color) is “with distribution”. This is because the color is assumed to change.

  The display image analysis unit 14c of another communication terminal has display data whose character data (monochrome), character data (color), drawing data (monochrome), drawing data (color), character + drawing (monochrome) shown in FIG. ) And character + drawing (color).

  The display image analysis unit 14c analyzes the quality of the display data according to the content classification.

<When it is mainly text (monochrome)>
FIG. 40 shows an example (part 1) of the display data quality judgment process.

  S4002: The display image analysis unit 14c starts display data quality analysis.

  S4004: The display image analysis unit 14c determines whether the quality information used as a determination material is a capture of the content itself (mainly text) or an OCR result.

  S4006: When the display image analysis unit 14c determines that the quality information used as the determination material is a capture of the content itself (mainly text), the color difference of the entire image is determined according to the quality synchronization determination criterion 26 stored in the storage unit 1000. Determine whether it is large or small.

  S4008: When the display image analysis unit 14c determines that the color difference of the entire image is small, the display image analysis unit 14c determines whether the pixel value difference of the entire image is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000.

  S4010: When the display image analysis unit 14c determines that the quality information used as the determination material is an OCR result, whether the difference in the character recognition rate is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000. Judging.

  S4012: When the display image analysis unit 14c determines that the difference in character recognition rate is small, the display image analysis unit 14c determines whether the difference in character recognition rate is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000. To do.

  S4014: If the display image analysis unit 14c determines that the color difference of the entire image is large in step S4006, or if the determination of the pixel value difference of the entire image is completed in step S4008, it determines that the character recognition rate is high in step S4010. In this case, when the determination of the erroneous recognition rate of characters is completed in step S4012, an analysis result of display data is created from the determined result.

  S4016: The display image analysis unit 14c determines whether or not to use a plurality of determination materials. If it is determined that a plurality of determination materials are used, the process returns to step S4004. If the display image analysis unit 14c does not determine in step S4016 that a plurality of determination materials are to be used, the display image analysis unit 14c ends.

<When it is mainly text (color)>
FIG. 41 shows an embodiment (No. 2) of the display data quality judgment processing.

  S4102: The display image analysis unit 14c starts display data quality analysis.

  S4104: The display image analysis unit 14c determines whether the quality information used as a determination material is a capture of the content itself (mainly text), an OCR result, or a color distribution result.

  S4106: When the display image analysis unit 14c determines that the quality information used as the determination material is a capture of the content itself (mainly text), the color difference of the entire image is determined according to the quality synchronization determination criterion 26 stored in the storage unit 1000. Determine whether it is large or small.

  S4108: When the display image analysis unit 14c determines that the color difference of the entire image is small, the display image analysis unit 14c determines whether the pixel value difference of the entire image is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000.

  S4110: When the display image analysis unit 14c determines that the quality information used as the determination material is an OCR result, whether the difference in the character recognition rate is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000. Judging.

  S4112: When the display image analysis unit 14c determines that the difference in the character recognition rate is small, the display image analysis unit 14c determines whether the difference in the character recognition rate is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000. To do.

  S4114: When the display image analysis unit 14c determines that the quality information used as the determination material is a color distribution result, whether the difference in color tone is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000. Judging.

  S4116: When the display image analysis unit 14c determines that the difference in color tone is small, the display image analysis unit 14c determines whether the saturation difference is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000.

  S4118: If the display image analysis unit 14c determines that the color difference of the entire image is large in step S4106, or if the determination of the pixel value difference of the entire image is completed in step S4108, it determines that the character recognition rate is high in step S4110. If the determination of the erroneous recognition rate of characters is completed in step S4112, if the difference in color change is determined to be large in step S4114, if the determination of the difference in saturation change is completed in step S4116, the determination is made. The display data analysis result is created from the result.

  S4120: The display image analysis unit 14c determines whether or not to use a plurality of determination materials. If it is determined that a plurality of determination materials are used, the process returns to step S4104. If the display image analysis unit 14c does not determine in step S4120 that a plurality of determination materials are to be used, the display image analysis unit 14c ends.

<Drawing subject (monochrome)>
FIG. 42 shows an embodiment (No. 3) of the display data quality judgment processing.

  S4202: The display image analysis unit 14c starts display data quality analysis.

  S4204: The display image analysis unit 14c determines whether the quality information used as the determination material is a capture of the content itself (mainly text), a frequency component extraction result, or an edge detection result.

  S4206: When the display image analysis unit 14c determines that the quality information used as the determination material is a capture of the content itself (mainly text), the color difference of the entire image is determined according to the quality synchronization determination criterion 26 stored in the storage unit 1000. Determine whether it is large or small.

  S4208: When the display image analysis unit 14c determines that the color difference of the entire image is small, the display image analysis unit 14c determines whether the pixel value difference of the entire image is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000.

  S4210: When the display image analysis unit 14c determines that the quality information used as the determination material is the extraction result of the frequency component, is the difference between the high frequency components large according to the quality synchronization determination criterion 26 stored in the storage unit 1000? Judge whether it is small.

  S4212: When the display image analysis unit 14c determines that the difference between the high frequency components is small, the display image analysis unit 14c determines whether the difference between the low frequency components is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000.

  S4214: When the display image analysis unit 14c determines that the quality information used as the determination material is an edge extraction result, is the difference in the information amount of the edge portion large according to the quality synchronization determination criterion 26 stored in the storage unit 1000? Judge whether it is small.

  S4216: When the display image analysis unit 14c determines that the color difference of the entire image is large in step S4206, or when the determination of the pixel value difference of the entire image is completed in step S4208, the display image analysis unit 14c determines that the difference of the high frequency component is large in step S4210. In this case, if the determination of the low frequency component is completed in step S4212, or if the determination of the difference in the information amount of the edge portion is completed in step S4214, the analysis result of the display data is created from the determined result.

  S4218: The display image analysis unit 14c determines whether or not to use a plurality of determination materials. If it is determined that a plurality of determination materials are used, the process returns to step S4204. If the display image analysis unit 14c does not determine in step S4218 that a plurality of determination materials are to be used, the display image analysis unit 14c ends.

<If the subject is a drawing (color)>
FIG. 43 shows an embodiment (No. 4) of the display data quality judgment processing.

  S4302: The display image analysis unit 14c starts display data quality analysis.

  S4304: The display image analysis unit 14c determines whether the quality information used as the determination material is a capture of the content itself (drawing subject), a color distribution result, a frequency component extraction result, or an edge detection result. Judging.

  S4306: When the display image analysis unit 14c determines that the quality information used as the determination material is a capture of the content itself (mainly text), the color difference of the entire image is determined according to the quality synchronization determination criterion 26 stored in the storage unit 1000. Determine whether it is large or small.

  S4308: When the display image analysis unit 14c determines that the color difference of the entire image is small, the display image analysis unit 14c determines whether the pixel value difference of the entire image is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000.

  S4310: When the display image analysis unit 14c determines that the quality information used as the determination material is a color distribution result, whether the difference in color tone is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000. Judging.

  S4312: When the display image analysis unit 14c determines that the difference in color tone is small, the display image analysis unit 14c determines whether the difference in saturation change is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000.

  S4314: If the display image analysis unit 14c determines that the quality information used as the determination material is the extraction result of the frequency component, is the difference between the high frequency components large according to the quality synchronization determination criterion 26 stored in the storage unit 1000? Judge whether it is small.

  S4316: When the display image analysis unit 14c determines that the difference between the high frequency components is small, the display image analysis unit 14c determines whether the difference between the low frequency components is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000.

  S4318: If the display image analysis unit 14c determines that the quality information used as the determination material is an edge extraction result, is the difference in the information amount of the edge portion large according to the quality synchronization determination criterion 26 stored in the storage unit 1000? Judge whether it is small.

  S4320: When the display image analysis unit 14c determines that the color difference of the entire image is large in step S4306, or when the determination of the pixel value difference of the entire image is completed in step S4308, the difference in change in color tone is large in step S4310. If it is determined, if the determination of the saturation change is completed in step S4312, if the difference in the high frequency component is determined to be large in step S4314, if the determination of the low frequency component is completed in step S4316, the edge in step S4318 When the determination of the difference in the information amount of the part is completed, an analysis result of the display data is created from the determined result.

  S4322: The display image analysis unit 14c determines whether or not to use a plurality of determination materials. If it is determined that a plurality of determination materials are used, the process returns to step S4304. If the display image analysis unit 14c does not determine in step S4322 that a plurality of determination materials are to be used, the display image analysis unit 14c ends.

<Character + Drawing (Monochrome)>
FIG. 44 shows an embodiment (No. 5) of the display data quality judgment processing.

  S4402: The display image analysis unit 14c starts display data quality analysis.

  S4404: The display image analysis unit 14c determines whether the quality information used as a determination material is a capture of the content itself (mainly text), an OCR result, a frequency component extraction result, or an edge detection result. to decide.

  S4406: When the display image analysis unit 14c determines that the quality information used as the determination material is a capture of the content itself (mainly text), the color difference of the entire image is determined according to the quality synchronization determination criterion 26 stored in the storage unit 1000. Determine whether it is large or small.

  S4408: When the display image analysis unit 14c determines that the color difference of the entire image is small, the display image analysis unit 14c determines whether the pixel value difference of the entire image is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000.

  S4410: When the display image analysis unit 14c determines that the quality information used as the determination material is an OCR result, whether the difference in the character recognition rate is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000. Judging.

  S4412: When the display image analysis unit 14c determines that the difference in the character recognition rate is small, the display image analysis unit 14c determines whether the difference in the character recognition rate is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000. To do.

  S4414: If the display image analysis unit 14c determines that the quality information used as the determination material is the extraction result of the frequency component, is the difference between the high frequency components large according to the quality synchronization determination criterion 26 stored in the storage unit 1000? Judge whether it is small.

  S4416: When the display image analysis unit 14c determines that the difference between the high frequency components is small, the display image analysis unit 14c determines whether the difference between the low frequency components is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000.

  S4418: If the display image analysis unit 14c determines that the quality information used as the determination material is an edge extraction result, is the difference in the information amount of the edge portion large according to the quality synchronization determination standard 26 stored in the storage unit 1000? Judge whether it is small.

  S4420: When the display image analysis unit 14c determines that the color difference of the entire image is large in step S4006, or when the determination of the pixel value difference of the entire image is completed in step S4008, it determines that the character recognition rate is high in step S4010. If the determination of the erroneous recognition rate of characters is completed in step S4012, if the difference in high frequency components is determined to be large in step S4414, if the determination of low frequency components is completed in step S4416, the result is displayed. Create data analysis results.

  S4422: The display image analysis unit 14c determines whether or not to use a plurality of determination materials. If it is determined that a plurality of determination materials are used, the process returns to step S4404. If the display image analysis unit 14c does not determine in step S4422 to use a plurality of determination materials, the display image analysis unit 14c ends.

<Character + drawing (color)>
FIG. 45 shows an example (No. 6) of the display data quality judgment process.

  S4502: The display image analysis unit 14c starts display data quality analysis.

  S4504: The display image analysis unit 14c determines whether the quality information used as the determination material is capture of the content itself (mainly text), an OCR result, a color distribution result, or a frequency component extraction result. It is determined whether it is an edge detection result.

  S4506: When the display image analysis unit 14c determines that the quality information used as the determination material is a capture (character-based) of the content itself, the color difference of the entire image is determined according to the quality synchronization determination criterion 26 stored in the storage unit 1000. Determine whether it is large or small.

  S4508: When the display image analysis unit 14c determines that the color difference of the entire image is small, the display image analysis unit 14c determines whether the pixel value difference of the entire image is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000.

  S4510: When the display image analysis unit 14c determines that the quality information used as the determination material is an OCR result, whether the difference in the character recognition rate is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000. Judging.

  S4512: When the display image analysis unit 14c determines that the difference in the character recognition rate is small, the display image analysis unit 14c determines whether the difference in the character recognition rate is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000. To do.

  S4514: When the display image analysis unit 14c determines that the quality information used as the determination material is a color distribution result, whether the difference in color tone is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000. Judging.

  S4516: When the display image analysis unit 14c determines that the difference in color change is small, the display image analysis unit 14c determines whether the difference in saturation change is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000.

  S4518: If the display image analysis unit 14c determines that the quality information used as the determination material is the extraction result of the frequency component, is the difference between the high frequency components large according to the quality synchronization determination criterion 26 stored in the storage unit 1000? Judge whether it is small.

  S4520: When the display image analysis unit 14c determines that the difference between the high frequency components is small, the display image analysis unit 14c determines whether the difference between the low frequency components is large or small according to the quality synchronization determination criterion 26 stored in the storage unit 1000.

  S4522: If the display image analysis unit 14c determines that the quality information used as the determination material is an edge extraction result, is the difference in the information amount of the edge portion large according to the quality synchronization determination criterion 26 stored in the storage unit 1000? Judge whether it is small.

  S4524: If the display image analysis unit 14c determines that the color difference of the entire image is large in step S4506, or if the determination of the pixel value difference of the entire image is completed in step S4508, it determines that the character recognition rate is high in step S4510. In step S4512, when the character recognition rate determination is completed, in step S4514 it is determined that the difference in color change is large, in step S4516 when the change in saturation is completed, in step S4518 When it is determined that the difference between the high frequency components is large, when the determination of the low frequency component is completed at step S4520, when the difference of the information amount of the edge portion is determined at step S4522, the analysis result of the display data from the determined result Create

  S4526: The display image analysis unit 14c determines whether to use a plurality of determination materials. If it is determined that a plurality of determination materials are used, the process returns to step S4504. If the display image analysis unit 14c does not determine in step S4526 that a plurality of determination materials are to be used, the display image analysis unit 14c ends.

  The display image analysis unit 14c of the communication terminal 10 analyzes the quality of the display data transmitted to other communication terminals for the item information corresponding to the target quality information, and obtains an analysis result (first analysis result). 38-45 can be applied to the processing executed when the display image analysis unit 14c of the communication terminal 10 obtains the first analysis result.

  Other communication terminals transmit the second analysis result to the communication terminal 10.

  The communication terminal 10 receives the second analysis result from another communication terminal. The quality synchronization determination unit 14d of the communication terminal 10 compares the second analysis result from another communication terminal with the first analysis result obtained by the display image analysis unit 14c, and based on the quality synchronization determination reference 26. The quality synchronization is determined by determining whether the quality difference is small.

  The communication terminal 10 continues to transmit data when it is determined that the difference in quality is small, and transmits a predetermined alarm when it is determined that the difference in quality is large.

  FIG. 46 is a sequence chart showing the operation of the telecommunications system according to the present embodiment.

  In step S4602, the shared content is input from the external input device 40 to the external information transmitting / receiving unit 18 of the communication terminal 10.

  In step S 4604, the external information transmission / reception unit 18 inputs shared content to the transmission / reception unit 11.

  In step S4606, the transmission / reception unit 11 transmits the shared content to another communication terminal via the communication network 2.

  In step S4608, the shared content received by the transmission / reception unit 11 of another communication terminal is input to the image display control unit 14b.

  In step S4610, the image display control unit 14b of another communication terminal displays the shared content.

  In step S4612, the image display control unit 14b of another communication terminal inputs the shared content to the display image analysis unit 14c.

  In step S4614, the display image analysis unit 14c of another communication terminal acquires a quality synchronization determination criterion.

  In step S4616, the display image analysis unit 14c of another communication terminal analyzes the quality of the display data.

  In step S4618, the display image analysis unit 14c of another communication terminal inputs the second analysis result to the transmission / reception unit.

  In step S4620, the transmission / reception unit of another communication terminal transmits the second analysis result to communication terminal 10 via communication network 2.

  In step S4622, the transmission / reception unit 11 of the communication terminal 10 inputs the second analysis result received from another communication terminal to the quality synchronization determination unit 14d.

  In step S4624, the external information transmitting / receiving unit 18 inputs the shared content to the display image analyzing unit 14c.

  In step S4626, the display image analysis unit 14c of the communication terminal 10 acquires a quality synchronization determination criterion.

  In step S4628, the display image analysis unit 14c of the communication terminal 10 analyzes the quality of the display data.

  In step S4630, the display image analysis unit 14c of the communication terminal 10 inputs the first analysis result to the quality synchronization determination unit 14d.

  In step S4632, the quality synchronization determination unit 14d of the communication terminal 10 determines whether the quality of the display data is synchronized.

  In step S4634, the quality synchronization determination unit 14d of the communication terminal 10 inputs feedback to the transmission / reception unit 11 when it is determined that the display data quality is not synchronized based on the determination result of whether or not the display data quality is synchronized.

  In step S4636, the transmission / reception unit 11 transmits feedback to another communication terminal via the communication network 2.

  In step S4638, the feedback received by the transmission / reception unit 11 of another communication terminal is input to the image display control unit 14b.

  In step S4640, the image display control unit 14b of another communication terminal displays feedback.

  FIG. 47 shows an example of feedback display. Based on the feedback from the communication terminal 10, the image display control unit 14b of another communication terminal displays on the shared content an explanation related to the shared content and a word prompting to wait for an operation on the shared content. For example, the image display control unit 14b displays a message that prompts the user that the quality is not synchronized with the transmission destination, the explanation of the content, and the refrain from operating the content. The message prompting the user is preferably changed according to the degree of synchronization of the quality of the content shared between the communication terminal and other communication terminals. For example, one or a plurality of thresholds are set as a criterion for sending a different message from when the image is somewhat disturbed until when the image cannot be recognized at all. In addition, one or more thresholds are set as a criterion for sending a different message from when the character is somewhat disturbed until no character can be recognized.

  The order in the sequence chart shown in FIG. 46 is not limited to this order. For example, the processing in steps 4604 to S4622 and the processing in steps S4624 to S4630 may be performed in the reverse order, or may be performed simultaneously.

  Further, the communication terminal 10 transmits the first analysis result to another communication terminal, and the other communication terminal performs quality synchronization based on the first analysis result and the second analysis result from the communication terminal 10. You may make it judge. When other communication terminals determine that they are not synchronized based on the determination result of quality synchronization, the communication terminal displays a description on the shared content and a word prompting to wait for an operation on the shared content. . The other communication terminal transmits the quality synchronization determination result to the communication terminal 10. By doing so, the processing load on the transmission side can be reduced.

  FIG. 48 shows the effect of the telecommunications system according to the present embodiment.

  According to the remote communication system according to the present embodiment, it is possible to determine the synchronization of the quality of the shared content between the transmission side and the reception side of the shared content. For example, when content is shared and synchronized in a video / web conference, the content transmitted by the transmission side can be compared with the content received by the reception side, and the difference can be determined based on a certain standard.

  It is possible to feed back to the user the determination result of whether or not the quality of the shared content is synchronized between the transmission side and the reception side. For this reason, the user can easily determine whether or not the quality of the content is synchronized. Since it is possible to easily determine whether or not the quality of content is synchronized, the progress of the conference can be made smoother than before.

  The remote communication system according to the present embodiment can be applied to a communication system for exchanging information between remote points such as a conference, education, and medical care.

  Although the present invention has been described with reference to specific embodiments and modifications, each embodiment and modification is merely an example, and those skilled in the art will recognize various modifications, modifications, alternatives, substitutions, and the like. Will understand. For convenience of explanation, an apparatus according to an embodiment of the present invention has been described using a functional block diagram, but such an apparatus may be implemented in hardware, software, or a combination thereof. The present invention is not limited to the above-described embodiments, and various variations, modifications, alternatives, substitutions, and the like are included without departing from the spirit of the present invention.

DESCRIPTION OF SYMBOLS 1 Remote communication system 10 Communication terminal 11 Transmission / reception part (an example of a display data transmission means)
12 operation input reception unit 13 login request unit 14a imaging unit 14b image display control unit 14c display image analysis unit 14d quality synchronization determination unit 15a audio input unit 15b audio output unit 16 selection processing unit 17 delay detection unit 18 external information transmission / reception unit (display) Example of data receiving means)
19 Storage / Read Processing Unit 21 Arrangement Information Selection Unit 22 Display Data Control Unit 23 Arrangement Information Management Table 24 Event Flag Table 25 Material Management Table 26 Quality Synchronization Judgment Standard 30 Relay Device 40 External Input Device 41 Transmission / Reception Unit 42 Connection Detection Unit 43 Storage Unit 431 transition condition table 44 menu control unit 451 display data acquisition unit 452 display data transmission unit 46 operation input reception unit 49 storage / read processing unit 50 remote communication management system 400 display device 1000 storage unit (an example of destination information storage means)

JP 2005-208710 A

Claims (10)

A communication terminal that is connected to a plurality of communication terminals via a network and transmits image data shared with the other communication terminal to another communication terminal,
An attribute acquired when comparing the first image data transmitted by the communication terminal and the second image data obtained by receiving the first image data by the other communication terminal; A storage unit for storing information indicating a reference used when determining a difference in quality between the first image data and the second image data based on
Based on the attribute stored in the storage unit, obtain the first attribute of the first image data transmitted to the other communication terminal,
Based on the first attribute and the second attribute acquired for the second image data by the other communication terminal, the quality between the first image data and the second image data is determined. And a control unit that determines a difference based on information indicating the reference.   When the difference in quality between the first image data and the second image data is equal to or greater than a predetermined threshold, the control unit feeds back feedback that the quality is not synchronized with the other communication terminal. The communication terminal according to claim 1. The storage unit stores a plurality of attributes and information indicating a criterion used when judging a difference in quality for each of the plurality of attributes.
The control unit acquires different attributes depending on whether the first image data transmitted to the other communication terminal is character information, drawing information, or one containing both character information and drawing information. The communication terminal according to claim 1, wherein the difference in quality is determined based on information indicating a criterion corresponding to the different attribute.   When the first image data transmitted to the other communication terminal is character information, the control unit stores the character portion of the first image data as data, and the first image data 4. The communication terminal according to claim 3, wherein one or both of the character recognition results of the character part are acquired as attributes, and the quality difference is determined based on information indicating a criterion corresponding to the attributes.   When the first image data transmitted to the other communication terminal is drawing information, the control unit stores a drawing portion of the first image data as data, a result of extracting a frequency component, and The communication terminal according to claim 3, wherein one or more of the results of detecting the edge portion are acquired as attributes, and the quality difference is determined based on information indicating a criterion corresponding to the attributes.   When the first image data transmitted to the other communication terminal includes both character information and drawing information, the control unit stores the first image data, the first image data As a result of character recognition of the character part, one or more of the result of extracting the frequency component and the result of detecting the edge part are acquired as attributes, and a difference in quality is determined based on information indicating a criterion corresponding to the attribute; The communication terminal according to claim 3.   When the first image data transmitted to the other communication terminal is color, the control unit acquires the color distribution of the first image data as an attribute, and uses information indicating a reference corresponding to the attribute The communication terminal according to claim 4, wherein a difference in quality is determined. A communication system in which a plurality of communication terminals are connected via a network, and image data is shared between the first communication terminal and the second communication terminal,
The first communication terminal is
Attributes acquired when comparing the first image data transmitted by the first communication terminal with the second image data obtained by receiving the first image data by the second communication terminal A first storage unit that stores information indicating a reference used when determining a quality difference between the first image data and the second image data based on the attribute;
Based on the attribute stored in the first storage unit, obtain the first attribute of the first image data transmitted to the second communication terminal,
Quality between the first image data and the second image data based on the first attribute and the second attribute acquired for the second image data by the second communication terminal A first control unit that determines a difference between the two based on information indicating the reference,
The second communication terminal is
Attributes acquired when comparing the first image data transmitted by the first communication terminal with the second image data obtained by receiving the first image data by the second communication terminal A second storage unit for storing
Based on the attribute stored in the second storage unit, the second attribute is acquired for the second image data from the first communication terminal, and the second attribute is transmitted to the first communication terminal. And a second control unit for notifying. A communication method executed by a communication terminal connected to a plurality of communication terminals via a network and transmitting image data shared with the other communication terminal to another communication terminal,
An attribute acquired when comparing the first image data transmitted by the communication terminal and the second image data obtained by receiving the first image data by the other communication terminal; Based on an attribute stored in a storage unit that stores information indicating a reference used when determining a difference in quality between the first image data and the second image data based on the attribute, Obtaining the first attribute of the first image data transmitted to another communication terminal;
Based on the first attribute and the second attribute acquired for the second image data by the other communication terminal, the quality between the first image data and the second image data is determined. A communication method for determining a difference based on information indicating the reference. A communication terminal connected to a plurality of communication terminals via a network and transmitting image data shared with the other communication terminal to the other communication terminal.
An attribute acquired when comparing the first image data transmitted by the communication terminal and the second image data obtained by receiving the first image data by the other communication terminal; Based on an attribute stored in a storage unit that stores information indicating a reference used when determining a difference in quality between the first image data and the second image data based on the attribute, Obtaining the first attribute of the first image data transmitted to another communication terminal;
Based on the first attribute and the second attribute acquired for the second image data by the other communication terminal, the quality between the first image data and the second image data is determined. The difference is judged based on the information indicating the criterion.
A program that lets you do that.