JP5754117B2 - External input device, display data creation method, program - Google Patents

Description

  The present invention relates to a remote communication terminal that shares display data with other remote communication terminals by transmitting screen display data to the other remote communication terminals.

  With the spread and speeding up of communication networks such as the Internet, remote communication systems that perform video conferences between remote locations have come to be used. If a telecommunications system is used, business trip expenses and business trip time can be reduced.

  In a remote communication system capable of realizing a video conference, a plurality of remote communication terminals transmit and receive image data and audio data directly or via a server. With the recent enhancement of broadband environment, it has become possible to send and receive high-quality image data and high-quality audio data, making it easier to grasp the status of the other party in a video conference and to facilitate advanced communication through conversation. ing.

  By the way, the remote communication system often transmits not only audio data for informing the situation here and image data showing the user on this side, but also display data of materials used in the conference to the other party of the conference. The display data of the material allows the other party of the video conference to refer to the same material as this side.

  Whether the remote communication terminal sends audio data or image data or sends display data, it sends the data to the delivery device via the network, and the received delivery device sends the data to the other party's destination. A technique for distributing to a remote communication terminal has been devised (for example, see Patent Document 1). In this way, participants in the video conference can listen to the other party's voice, look at their faces, and refer to the materials while the video conference is being held.

  By the way, when explaining presentation materials on a projector or the like at a conference, for example, the presenter often uses an indicator bar or a laser pointer to make the explanation easy to understand. Similarly, when materials used in a video conference are shared with the other party of the video conference, it is easier to convey information if there is some instruction means than referring only the material to the other party.

  In a video conference, when the user on this side transmits display data such as materials to the other party, the remote communication terminal on this side captures a screen on which the materials are displayed. For capture, a “Print Screen” button mounted on a general keyboard is used, but there is a limitation that the mouse cursor is not captured by the “Print Screen” button.

  For this reason, as in Cited Document 1, there is a problem that instruction information such as a mouse cursor cannot be shared between remote communication terminals simply by the remote communication terminal on this side simply transmitting display data. For this reason, it is difficult to tell which parts of the material are viewed by users who are far away from each other.

  For such problems, software that can simultaneously capture a mouse cursor is available on the market for a fee or free of charge, and developers of telecommunications systems can use it for video conferencing.

  However, although it is possible to capture the mouse cursor only with software that captures the instruction means such as the mouse cursor, the display data cannot be shared with a remote place as it is. In addition, since the mouse cursor and screen data are combined and captured, the frequency of capture increases and the load on the remote communication terminal and communication network increases in order for the destination terminal to reproduce the mouse movement smoothly. There is also the problem of doing.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an external input device capable of sharing instruction means for instructing on a screen in a remote communication system in which display data such as materials can be shared by a plurality of remote communication terminals.

  In view of the above problems, the present invention is an external input device connected to a remote communication terminal that delivers conference data including audio or video to other devices connected via a network, and displays an image. A display unit; a cursor operation unit that receives an operation of a cursor displayed on the display unit; an image data acquisition unit that acquires image data of the image; and a cursor that acquires cursor information including at least coordinate data of the cursor. Information acquisition means; information acquisition frequency determination means for determining a first frequency at which the cursor information acquisition means acquires the cursor information; and display data generation means for generating display data obtained by synthesizing the cursor with the image data. And transmitting means for transmitting the display data to another device.

  In a remote communication system in which display data such as documents can be shared by a plurality of remote communication terminals, it is possible to provide an external input device that can also share instruction means for instructing on the screen.

It is an example of a block diagram of a telecommunications system. It is an example of the hardware block diagram of a remote communication terminal. It is an example of the hardware block diagram of a remote communication management system. 1 is an example of a functional block diagram of each terminal, device, and system that constitutes a remote communication system 1. FIG. It is an example of a functional block diagram of an external input device. It is a figure which shows an example of a material management table. It is a figure which shows an example of a change quality management table. It is a figure which shows an example of a relay apparatus management table. It is a figure which shows an example of a terminal authentication management table. It is a figure which shows an example of a terminal management table. It is a figure which shows an example of a destination list management table. It is a figure which shows an example of a session management table. It is a figure which shows an example of an address priority management table. It is a figure which shows an example of a transmission rate priority management table. It is a figure which shows an example of a quality management table. It is an example of the sequence diagram which showed the process which transmits / receives image data and audio | voice data between remote communication terminals. It is an example of a sequence diagram showing a procedure for displaying material data stored in an HDD by an external input device and sharing the entire screen. It is a figure which shows the example of a screen of the screen which the image display control part of a remote communication terminal displays. FIG. 18 is an example of a sequence diagram illustrating a procedure in which the external input device captures image data and instruction information of a material during “display data acquisition in step S75” in FIG. It is an example of the figure which illustrates composition of two images typically. FIG. 18 is an example of a flowchart showing a procedure for an external input device to capture image data and instruction information of a material at the time of “display data acquisition in step S75” in FIG. FIG. 18 is an example of a diagram illustrating a procedure in which an external input device captures image data and instruction information of a material at the time of “display data acquisition in step S75” in FIG. FIG. 18 is an example of a diagram illustrating a procedure in which an external input device captures image data and instruction information of a material at the time of “display data acquisition in step S75” in FIG. FIG. 18 is an example of a diagram illustrating a procedure in which an external input device captures image data and instruction information of a material at the time of “display data acquisition in step S75” in FIG.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
The general features of the remote communication system 1 of the present embodiment are as follows.
(1) Capturing the image data of the material and the instruction information such as the mouse cursor separately and combining them into one display data (the mouse cursor can be combined with the image data of the material)
(2) The frequency of capturing the image data of the material and the frequency of capturing the instruction information such as the mouse cursor can be respectively determined (the mouse cursor can be smoothly moved even at the distribution destination while reducing the processing load).
(3) Deciding whether to create display data according to the presence or absence of changes in the image data of the material and the amount of movement of the mouse cursor. (The load on the communication network can be reduced)

  FIG. 1 is an example of a block diagram of a telecommunications system. In the remote communication system, as an example, the remote communication terminals 10 are divided and arranged in areas A and B. However, at least two remote communication terminals can be used for remote communication.

  The remote communication system 1 includes a plurality of remote communication terminals (10aa, 10ab,..., 10db), a display (100aa, 100ab,..., 100db) for each remote communication terminal (10aa, 10ab,. A device (30a, 30b, 30c, 30d), a remote communication management system 50, a program providing system, and an external input device;

  In the present embodiment, the “remote communication terminal 10” is used to indicate any remote communication terminal among the remote communication terminals (10aa, 10ab,..., 10db), and the display (100aa, 100ab,..., 100db). The “display 100” is used when an arbitrary display is shown, and the “relay device 30” is used when an arbitrary relay device among the relay devices (30a, 30b, 30c, 30d) is shown. In the case of indicating an arbitrary external input device among (40aa, 40ab,..., 140db), “external input device 40” is used.

  The remote communication terminal 10 transmits / receives image data, audio data, and the like to / from other remote communication terminals 10. In the present embodiment, the case where the image of the image data is a moving image will be described. The image of the image data may include both a moving image and a still image. The relay device 30 relays image data and audio data between the plurality of remote communication terminals 10. The remote communication management system 50 manages the remote communication terminal 10 and the relay device 30 in an integrated manner.

  The external input device 40 is connected to the remote communication terminal 10 and transmits display data for displaying material data to the remote communication terminal 10. The material data here means data used by using document creation software, spreadsheet software, presentation software, and the like.

  Further, the plurality of routers (70a, 70b,..., 70f) shown in FIG. 1 select an optimum route for image data and audio data. In the present embodiment, “router 70” is used to indicate an arbitrary router among the routers (70a, 70b,..., 70f).

  The program providing system 90 includes an HDD (Hard Disk Drive) (not shown) that stores a remote communication terminal program for realizing various functions or various means in the remote communication terminal 10. The remote communication terminal program can be transmitted. The HDD of the program providing system 90 also stores a relay device program for causing the relay device 30 to realize various functions or various means, and can transmit the relay device program to the relay device 30. . Further, the HDD of the program providing system 90 also stores a remote communication management program for causing the remote communication management system 50 to realize various functions or various means, and the remote communication management system 50 stores the remote communication management program. Can be sent.

  The remote communication terminal 10aa, the remote communication terminal 10ab, the relay device 30a, and the router 70a are connected to each other via a LAN 2a so as to communicate with each other. The remote communication terminal 10ba, the remote communication terminal 10bb, the relay device 30b, and the router 70b are communicably connected to each other via the LAN 2b. The LAN 2a and the LAN 2b are communicably connected via a dedicated line 2ab including the router 70c, and are constructed in a predetermined area A. For example, the region A is Japan, the LAN 2a is constructed in a Tokyo office, and the LAN 2b is constructed in an Osaka office.

  On the other hand, the remote communication terminal 10ca, the remote communication terminal 10cb, the relay device 30c, and the router 70d are connected to be communicable with each other via the LAN 2c. The remote communication terminal 10da, the remote communication terminal 10db, the relay device 30d, and the router 70e are communicably connected via a LAN 2d. The LAN 2c and the LAN 2d are communicably connected by a dedicated line 2cd including the router 70f, and are constructed in a predetermined area B. For example, region B is the United States, LAN 2c is built in a New York office, and LAN 2d is Washington D.C. C. Is built in the office.

  Area A and area B are connected to each other via routers (70c, 70f) via the Internet 2i.

  The remote communication management system 50 and the program providing system 90 are communicably connected to the remote communication terminal 10 and the relay device 30 via the Internet 2i. The remote communication management system 50 and the program providing system 90 may be installed in the region A or the region B, or may be installed in a region other than these.

  In the present embodiment, the communication network 2 of the present embodiment is constructed by the LAN 2a, the LAN 2b, the dedicated line 2ab, the Internet 2i, the dedicated line 2cd, the LAN 2c, and the LAN 2d.

  Further, in FIG. 1, four sets of numbers shown under each remote communication terminal 10, each relay device 30, remote communication management system 50, each router 70, and program providing system 90 are general IPv4. The IP address in is simply shown. For example, the IP address of the remote communication terminal 10aa is “1.2.1.3”. Further, IPv6 may be used instead of IPv4. In order to simplify the description, description will be made using IPv4.

<< Hardware configuration >>
Next, the hardware configuration of each device will be described. In this embodiment, when there is a delay in the reception of image data at the remote communication terminal 10 as the relay destination, the resolution of the image data of the image data is changed by the relay device 30 and then the remote communication terminal 10 as the relay destination. A case where image data is transmitted to will be described.

  FIG. 2 is an example of a hardware configuration diagram of the remote communication terminal 10. As shown in FIG. 2, the remote communication terminal 10 of this embodiment includes a CPU (Central Processing Unit) 101 that controls the operation of the entire remote communication terminal 10 and a ROM (Read Only) that stores a program for the remote communication terminal. Memory) 102, RAM (Random Access Memory) 103 used as a work area for the CPU 101, flash memory 104 for storing various data such as image data and audio data, reading various data to the flash memory 104 according to the control of the CPU 101, or It is operated when an SSD (Solid State Drive) 105 that controls writing, a media drive 107 that controls reading or writing (storage) of data to a recording medium 106 such as a flash memory, and a destination of the remote communication terminal 10 are selected. Operation button 108, remote communication terminal 10 power supply A power switch 109 for switching ON / OFF, a network I / F 111 for transmitting data using a communication network 2 described later, and a CCD (Charge Coupled Device) 112 that captures an image of a subject and obtains image data under the control of the CPU 101 The image sensor I / F 113 that controls the driving of the CCD 112, the microphone 114 that inputs sound, the speaker 115 that outputs sound, and the sound that processes the input and output of sound signals between the microphone 114 and the speaker 115 according to the control of the CPU 101 2 shows an input / output I / F 116, a display I / F 117 that transmits image data to an external display 100 according to the control of the CPU 101, an external device I / F 118 that transmits / receives various data to / from an external device, and the above-described components. For electrical connection as And a Resubasu, a data bus, and the like bus line 110.

  Note that the recording medium 106 is detachable from the remote communication terminal 10. Further, as long as it is a non-volatile memory that reads or writes data according to the control of the CPU 101, not only the flash memory 104 but also an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used. Further, the CCD 112 is a solid-state imaging device that converts light into electric charges and digitizes a subject image (video). The CCD 112 is not limited to a CCD as long as it captures a subject, and a CMOS (Complementary Metal Oxide Semiconductor) or the like. It may be used. Further, the display 100 is configured by a liquid crystal or an organic EL that displays an image of a subject, an operation icon, and the like.

  Furthermore, the remote communication terminal program may be distributed in a computer-readable recording medium such as the recording medium 106 as an installable or executable file.

  FIG. 3 is an example of a hardware configuration diagram of the remote communication management system 50 according to an embodiment of the present invention. The remote communication management system 50 includes a CPU 201 that controls the overall operation of the remote communication management system 50, a ROM 202 that stores a remote communication management program, a RAM 203 that is used as a work area for the CPU 201, and an HD (Hard Disk) that stores various data. 204, an HDD (Hard Disk Drive) 205 that controls reading or writing of various data to the HD 204 according to the control of the CPU 201, a media drive 207 that controls reading or writing (storage) of data to a recording medium 206 such as a flash memory, and a cursor , A display 208 for displaying various information such as menus, windows, characters, images, etc., a network I / F 209 for transmitting data using the communication network 2 described later, input of characters, numerical values, various instructions, etc. A keyboard 211 having a plurality of keys, a mouse 212 for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like, and a CD-ROM (Compact Disc Read Only Memory) as an example of a removable recording medium ) CD-ROM drive 214 for controlling reading / writing of data to / from 213, external device I / F 215 for transmitting / receiving information to / from an external device, and the above components are electrically connected as shown in FIG. For this purpose, a bus line 210 such as an address bus or a data bus is provided.

  The remote communication management program is a file in an installable or executable format, and is recorded on a computer-readable recording medium such as the recording medium 206 or CD-ROM 213 for distribution. Good.

  Further, since the external input device 40 has the same hardware configuration as the remote communication management system 50, the description thereof is omitted. However, the ROM 202 stores an external input device program for controlling the external input device 40. Also in this case, the external input device program is a file in an installable or executable format, and is recorded and distributed on a computer-readable recording medium such as the recording medium 206 or the CD-ROM 213. Also good.

  Further, since the relay device 30 has the same hardware configuration as the remote communication management system 50, the description thereof is omitted. However, the ROM 202 stores a relay device program for controlling the relay device 30. Also in this case, the relay device program is a file in an installable or executable format, and is recorded on a computer-readable recording medium such as the recording medium 206 or CD-ROM 213 for distribution. Good.

  Furthermore, since the program providing system 90 has the same hardware configuration as that of the remote communication management system 50, the description thereof is omitted. However, a program providing program for controlling the program providing system 90 is recorded in the ROM 202. Also in this case, the program providing program is a file in an installable or executable format, and may be recorded and distributed on a computer-readable recording medium such as the recording medium 206 or the CD-ROM 213. Good.

  As another example of the detachable recording medium, the recording medium is provided by being recorded on a computer-readable recording medium such as a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), or a Blu-ray Disc. May be.

  FIG. 4 is an example of a functional block diagram of each terminal, device, and system constituting the telecommunications system 1 of the present embodiment. In FIG. 4, the remote communication terminal 10, the relay device 30, and the remote communication management system 50 are connected so that data communication can be performed via the communication network 2. The external input device 40 is connected so that data can be transmitted to and received from the remote communication terminal 10. The program providing system 90 shown in FIG. 1 is omitted in FIG. 4 because it is not directly related to the video conference communication. FIG. 5 is an example of a functional block diagram of the external input device 40.

<Functional configuration of remote communication terminal>
The remote communication terminal 10 includes a transmission / reception unit 11, an operation input reception unit 12, a login request unit 13, an imaging unit 14a, an image display control unit 14b, a voice input unit 15a, a voice output unit 15b, a selection processing unit 16, and a delay detection unit 17. The external information transmitting / receiving unit 18 and the storage / reading processing unit 19 are provided. Each of these units is a function or means realized by any one of the constituent elements shown in FIG. 2 operating according to a command from the CPU 201 according to a program stored in the ROM 202. Further, the remote communication terminal 10 has a storage unit 1000 constructed by the SSD 105 shown in FIG.

(Functions of remote communication terminal)
Next, each part of the remote communication terminal will be described in detail. The transmission / reception unit 11 of the remote communication terminal 10 is realized by the network I / F 111 illustrated in FIG. 2, and transmits / receives various data (information) to / from other terminals, devices, or systems via the communication network 2. The operation input receiving unit 12 is realized by the operation button 108 and the power switch 109 shown in FIG. 2 and receives various inputs by the user. For example, when the user turns on the power switch 109 shown in FIG. 2, the operation input accepting unit 12 shown in FIG. 4 accepts the power on and turns on the power. The login request unit 13 is realized by an instruction from the CPU 101 shown in FIG. 2, and requests the login from the transmission / reception unit 11 to the remote communication management system 50 via the communication network 2 when receiving the power ON. Log-in request information indicating that it is to be transmitted, and the current IP address of the remote communication terminal 10ab are automatically transmitted.

  The imaging unit 14a is realized by the CCD 112 and the imaging element I / F 113 shown in FIG. 2, images a subject, and outputs image data obtained by the imaging. The image display control unit 14 b is realized by the display I / F 117 illustrated in FIG. 2, and performs control for transmitting image data to the external display 100. The voice input unit 15a is realized by the microphone 114 and the voice input / output I / F 116 shown in FIG. 2, and inputs the voice of the user and converts the voice into a voice signal. Output data. The audio output unit 15b is realized by the speaker 115 and the audio input / output I / F 116 shown in FIG. 2, and converts an audio signal related to audio data into audio and outputs the audio.

  Since the selection processing unit 16 performs a final narrowing process that finally narrows down from a plurality of relay devices 30 to one relay device 30, a measurement unit 16a, a calculation unit 16b, And the selection part 16c is implement | achieved. Among these, the measurement unit 16 a measures the reception date and time when the pre-transmission information is received by the transmission / reception unit 11 for each pre-transmission information described later received by the transmission / reception unit 11. For each pre-transmission information whose reception date and time is measured by the measurement unit 16a, the calculation unit 16b determines the pre-transmission information based on the difference between the measured reception time and the transmission date and time included in the pre-transmission information. The time required from transmission to reception is calculated. The selection unit 16c finally selects one relay device by selecting the relay device 30 to which the pre-transmission information requiring the shortest required time is relayed among the required times calculated by the calculation unit 16b.

  The delay detection unit 17 is realized by a command from the CPU 101 shown in FIG. 2 and calculates the delay time (ms) of image data or audio data sent from another remote communication terminal 10 via the relay device 30. To detect. The external information transmitting / receiving unit 18 transmits / receives data to / from the external input device 40 through the external device I / F 215. Further, the storage / reading processing unit 19 stores various data in the storage unit 1000 and reads various data stored in the storage unit 1000 with the SSD 105 shown in FIG. The storage unit 1000 includes a terminal ID (Identification) for identifying the remote communication terminal 10, a password, a relay device ID for identifying the relay device 30 that transmits image data, audio data, and various data, and a destination terminal Is stored.

  Further, the storage unit 1000 stores a display data acquisition unit 451 and a display data transmission unit 452 that are transmitted to the external input device 40 and operate on the external input device 40. The display data acquisition unit 451 acquires display data of the external input device 40, and the display data transmission unit 452 transmits display data acquired by the display data acquisition unit 451 to the remote communication terminal 10. The display data represents image data obtained by converting an image displayed on the screen of the display device into a format such as JPEG (Joint Photographic Experts Group) or Bitmap, or a drawing command such as GDI (Graphics Device Interface). The display data may include document data or may be a simple desktop screen that does not include document data.

  The terminal ID of this embodiment and the relay device ID described later are identification information such as language, characters, symbols, or various signs used to uniquely identify the remote communication terminal 10 and the relay device 30, respectively. Indicates. Further, the terminal ID and the relay device ID may be identification information in which at least two of the language, characters, symbols, and various signs are combined. In the following description, the remote communication terminal 10 serving as a request source requesting the start of a video conference is referred to as “request source terminal 10A”, and the remote communication terminal 10 serving as a request destination is referred to as “destination terminal 10B”.

<Functional configuration of external input device>
As shown in FIG. 5A, the external input device 40 includes a transmission / reception unit 41, a connection detection unit 42, an installation determination unit 43, a program acquisition unit 44, an operation input reception unit 46, a display control unit 47, and a mount. Section 48 and a storage / reading processing section 49. Further, a display data acquisition unit 451 and a display data transmission unit 452 are illustrated. Each of these units is a function or means realized by any one of the components shown in FIG. 3 operating according to a command from the CPU 201 according to a program stored in the ROM 202. The external input device 40 includes a display device 400 constructed by the display 208 shown in FIG. Further, the external input device 40 is equipped with an OS (Operating System) such as Windows (registered trademark), and thereby has a function of executing a program when connected to another device.

<Functional parts of external input device>
Next, each part of the external input device 40 will be described in detail. The transmission / reception unit 41 of the external input device 40 is realized by the network I / F 209 illustrated in FIG. 3, and transmits / receives various data (information) from the remote communication terminal 10. The connection detection unit 42 detects that data transmission / reception with the remote communication terminal 10 can be performed by the external device I / F 215. The installation determination unit 43 determines whether the display data acquisition unit 451 and the display data transmission unit 452 are installed in the external input device 40. The program acquisition unit 44 acquires the display data acquisition unit 451 and the display data transmission unit 452 from the storage unit 1000 of the remote communication terminal 10 connected via the transmission / reception unit 41 and installs them. The operation input receiving unit 46 receives an input by a user operation. The display control unit 47 causes the display device 400 to display an image read by a storage / reading processing unit 49 described later. The mount unit 48 mounts a storage unit of various devices connected to the external input device 40. The storage / reading processing unit 49 stores various data using the HDD 205 shown in FIG. 3 as a substance, and reads various data stored in the HDD 205. The HDD 205 stores a material management table and the like.

  5B, the display data acquisition unit 451 of the external input device includes an information acquisition frequency determination unit 4511, an instruction information acquisition unit 4512, a screen information acquisition unit 4513, and a display data creation unit 4514. Have.

  The information acquisition frequency determination unit 4511 determines the acquisition frequency of image data and instruction information (for example, once per second). Thus, by acquiring the image data and the instruction information at relatively short intervals, the remote communication system 1 can distribute the image data as a continuous image, that is, a moving image. The acquisition frequency may be fixed or variable. For example, the information acquisition frequency determination unit 4511 inquires the remote communication management system 50 about the delay time, and determines the acquisition frequency according to the delay time read from the session management table (see FIG. 12) by the remote communication management system 50 (see FIG. 12). For example, if the delay time is 50 ms, the acquisition frequency is 0.3 seconds, if the delay time is 100 ms, the acquisition frequency is 0.6 seconds, and if the delay time is 200 ms, the acquisition frequency is 1 second). Alternatively, in order to eliminate the need for the information acquisition frequency determination unit 4511 to make an inquiry to the remote communication management system 50, the remote communication terminal whose remote communication management system 50 has been confirmed to be operating in the terminal management table (see FIG. 10). The delay time may be distributed to 10.

  The screen information acquisition unit 4513 acquires the image data of the material displayed on the display device 400 by the external input device 40. A technique called hard copy or capture is used to acquire image data. For example, the screen information acquisition unit 4513 requests the OS to store a hard copy of the screen on the clipboard, and acquires image data on the clipboard from the OS.

  The instruction information acquisition unit 4512 acquires instruction information such as the position of a mouse cursor and image data. As will be described later, the instruction information is coordinate data of the mouse cursor on the screen, or coordinate data of the mouse cursor on the screen and image data thereof. The coordinate data of the mouse cursor can be acquired by inquiring the OS with a predetermined command. Similarly, the mouse cursor image data may be acquired from the OS, or a previously known mouse cursor icon may be used as the image data.

  The display data creation unit 4514 creates display data by combining the image data of the material and the instruction information. Since the external input device 40 transmits the display data to the remote communication terminal 10, each user can commonly grasp the position specified by the material on the screen of the mouse cursor.

(Material management table)
A material management table is stored in the HDD 205 of the external input device 40.
FIG. 6 is a diagram showing an example of the material management table. In the material management table, the material name that is the name of the material, the size that represents the size of the material, the application name that represents the name of the application software used to create the material, the update date and time when the material was last updated, Document data and the like are managed in association with each other.

<Functional configuration of relay device>
Next, functions or means of the relay device 30 will be described. The relay device 30 includes a transmission / reception unit 31, a state detection unit 32, a data quality confirmation unit 33, a change quality management unit 34, a data quality change unit 35, and a storage / read processing unit 39. Each of these units is a function or means realized by any one of the components shown in FIG. 3 operating according to a command from the CPU 201 according to a program stored in the ROM 202. In addition, the relay device 30 includes a storage unit 3000 constructed by the HD 204 illustrated in FIG.

(Change quality control table)
In the storage unit 3000, a change quality management DB 3001 configured by a change quality management table as shown in FIG. In the change quality management table, the IP address of the remote communication terminal 30 as a relay destination of image data and the image quality of the image data relayed by the relay device 30 are associated with the relay destination and managed.

  The resolution is, for example, 160 pixels in the horizontal direction and 120 pixels in the vertical direction, a low-resolution image that is the base image, a medium-resolution image that is 320 pixels in the horizontal direction and 240 pixels in the vertical direction, 640 pixels in the horizontal direction, Is a high-resolution image composed of 480 pixels. Among these, when passing through a narrow band route, low-quality image data consisting only of low-resolution image data serving as a base image is relayed. When the band is relatively wide, low-resolution image data serving as a base image and medium-quality image data composed of medium-resolution image data are relayed. When the bandwidth is very wide, high-resolution image data composed of low-resolution image data serving as base image quality, intermediate-resolution image data, and high-resolution image data is relayed. For example, in the modified quality management table shown in FIG. 7, when the relay device 30 relays image data to the destination terminal 10db having the IP address “1.3.2.4”, the relay is performed. The image quality (image quality) of the image data is “high quality”.

<Each functional unit of the relay device>
Next, each functional configuration of the relay device 30 will be described in detail. In the following, in describing each part of the relay device 30, a relationship with main components for realizing each part of the relay device 30 among the respective components illustrated in FIG. 2 will also be described.

  The transmission / reception unit 31 of the relay device 30 shown in FIG. 4 is realized by the network I / F 209 shown in FIG. 3, and communicates with other terminals, devices, or systems via the communication network 2 and various data (information ). The state detection unit 32 is realized by a command from the CPU 201 illustrated in FIG. 3, and detects the operating state of the relay device 30 having the state detection unit 32. The operating state includes “ON line”, “OFF line”, and “failing” state.

  The data quality confirmation unit 33 is realized by a command from the CPU 201 shown in FIG. 3, searches the change quality management table of FIG. 7 using the IP address of the destination terminal 10B as a search key, and the corresponding relayed image. The image quality of the relayed image data is confirmed by extracting the image quality of the data. The change quality management unit 34 is realized by a command from the CPU 201 shown in FIG. 3 and changes the contents of the change quality management DB 3001 based on quality information to be described later sent from the remote communication management system 50. . For example, during a video conference by transmitting / receiving high-quality image data between the request source terminal 10aa having the terminal ID “01aa” and the destination terminal 10db having the terminal ID “01db”. When a delay in receiving image data occurs at the destination terminal 10db due to the request source terminal bb and the destination terminal ca performing another video conference starting the video conference via the communication network 2, etc., the relay device No. 30 needs to lower the image quality of image data relayed up to now from high image quality to medium image quality. In such a case, the content of the change quality management DB 3001 is changed so as to lower the image quality of the image data relayed by the relay device 30 from the high image quality to the medium image quality based on the quality information indicating the medium image quality.

  The data quality changing unit 35 is realized by a command from the CPU 201 shown in FIG. 3, and the image quality of the image data sent from the transmission source terminal 10 is based on the contents of the changed change quality management DB 3001. To change. The storage / reading processing unit 39 is realized by the HDD 205 shown in FIG. 3, and stores various data in the storage unit 3000 or reads various data stored in the storage unit 3000.

<Functional configuration of remote communication management system>
Next, functions or means of the remote communication management system 50 will be described. The remote communication management system 50 includes a transmission / reception unit 51, a terminal authentication unit 52, a state management unit 53, a terminal extraction unit 54, a terminal state acquisition unit 55, a narrowing unit 56, a session management unit 57, a quality determination unit 58, a storage / readout A processing unit 59 and a delay time management unit 60 are included. Each of these units is a function or means realized by any one of the components shown in FIG. 3 operating according to a command from the CPU 201 according to a program stored in the ROM 202. Further, the remote communication management system 50 has a storage unit 5000 constructed by the HD 204 shown in FIG.

(Relay device management table)
In the storage unit 5000, a relay device management DB 5001 configured by a relay device management table as shown in FIG. 8 is constructed. In this relay device management table, for each relay device ID of each relay device 30, the operating status of each relay device 30, the reception date and time when the status information indicating the operating status is received by the remote communication management system 50, the relay device 30 The IP address and the maximum data transmission rate (Mbps) in the relay device 30 are associated and managed. For example, in the relay device management table shown in FIG. 7, the relay device 30 a with the relay device ID “111a” has the operating state “ON line” and the date and time when the status information was received by the remote communication management system 50. Is “November 10, 2009, 13:00”, the IP address of the relay device 30a is “1.2.1.2”, and the maximum data transmission rate in the relay device 30a is 100 Mbps. It is shown.

(Terminal authentication management table)
Furthermore, in the storage unit 5000, a terminal authentication management DB 5002 configured by a terminal authentication management table as shown in FIG. In this terminal authentication management table, each password is associated with each terminal ID of all remote communication terminals 10 managed by the remote communication management system 50 and managed. For example, the terminal authentication management table shown in FIG. 8 indicates that the terminal ID of the remote communication terminal 10aa is “01aa” and the password is “aaaa”.

(Terminal management table)
Further, in the storage unit 5000, a terminal management DB 5003 configured by a terminal management table as shown in FIG. 10 is constructed. In this terminal management table, for each terminal ID of each remote communication terminal 10, the operating state of each remote communication terminal 10, the reception date and time when login request information described later is received by the remote communication management system 50, and the remote communication terminal 10 An IP address is associated and managed. For example, in the terminal management table shown in FIG. 9, the remote communication terminal 10aa with the terminal ID “01aa” has the operating state “ON line” and the date and time when the remote communication management system 50 received the login request information. Is “November 10, 2009, 13:40” and the IP address of the remote communication terminal 10aa is “1.2.1.3”.

(Destination list management table)
Further, in the storage unit 5000, a destination list management DB 5004 configured by a destination list management table as shown in FIG. In this destination list management table, all terminal IDs of destination terminals 10B registered as candidates for destination terminal 10B are managed in association with terminal IDs of request source terminal 10A requesting the start of a video conference. For example, in the destination list management table shown in FIG. 11, a candidate for the destination terminal 10B that can request the start of the video conference from the request source terminal 10aa having the terminal ID “01aa” has the terminal ID “01ab”. The remote communication terminal 10ab with the terminal ID "01ba" and the remote communication terminal 10db with the terminal ID "01db" are shown. The destination terminal 10B candidates are updated by being added or deleted in response to a request for addition or deletion from the request source terminal 10A to the remote communication management system 50.

(Session management table)
Further, in this storage unit 5000, a session management DB 5005 configured by a session management table as shown in FIG. 12 is constructed. In this session management table, for each selection session ID used to execute a session for selecting the relay device 30, the relay device ID and request of the relay device 30 used for relaying image data, audio data, and display data. The terminal ID of the original terminal 10A, the terminal ID of the destination terminal 10B, the reception delay time (ms) when image data is received at the destination terminal 10B, and the delay information indicated by this delay time are sent from the destination terminal 10B. The reception date and time received and received by the remote communication management system 50 is associated and managed. For example, in the session management table shown in FIG. 11, the relay device 30a (relay device ID “111a”) selected in the session executed using the selection session ID “se1” has the terminal ID “01aa”. "And the destination terminal 10db with the terminal ID" 01db "are relayed image data and audio data, and the destination terminal 10db" November 10, 2009 14:00 It is shown that the delay time of the image data at the time is 200 (ms). When performing a video conference between the two remote communication terminals 10, the reception date and time of the delay information may be managed based on the delay information transmitted from the request source terminal 10A instead of the destination terminal 10B. Good. However, when a video conference is performed between three or more remote communication terminals 10, the delay information is based on the delay information transmitted from the remote communication terminal 10 on the receiving side of the image data, audio data, and display data. Manage the reception date and time.

(Address priority management table)
Further, in the storage unit 5000, a priority management DB 5006 configured by an address priority management table as shown in FIG. 13 is constructed. In this address priority management table, in accordance with the difference in the four dot address (Dot Address) portions of the general IPv4 IP addresses, the address priority points are associated and managed so as to increase. The For example, in the address priority management table shown in FIG. 13, when the three values are the same IP address from the top to the bottom of the dot address, the address priority point is “5”. When the two values are the same IP address from the top to the bottom of the dot address, the address priority point is “3”. In this case, whether the value of the lowest dot address is the same or not is not related to the priority. In the case of IP addresses having the same highest value of dot addresses and different second values from the top, the address priority point is “1”. In this case, whether or not the values of the third and lowest dot addresses from the top are the same has no relation to the priority. In the case of IP addresses having different uppermost values of dot addresses, the address priority point is “0”. In this case, whether or not the values of the second, third, and lowest dot addresses from the top are the same has no relation to the priority.

(Transmission speed priority management table)
The priority management DB 5006 built in the storage unit 5000 also includes a transmission speed priority management table as shown in FIG. In this transmission rate priority management table, the transmission rate priority points are associated and managed in accordance with the value of the maximum data transmission rate (Mbps) in the relay device 30 so as to increase. For example, in the transmission rate priority management table shown in FIG. 14, when the maximum data transmission rate in the relay device 30 is 1000 Mbps or more, the transmission rate priority point is “5”. When the maximum data transmission rate in the relay device 30 is 100 Mbps or more and less than 1000 Mbps, the transmission rate priority point is “3”. When the maximum data transmission rate in the relay device 30 is 10 Mbps or more and less than 100 Mbps, the transmission rate priority point is “1”. When the maximum data transmission rate in the relay device 30 is less than 10 Mbps, the transmission rate priority point is “0”.

(Quality control table)
Furthermore, in the storage unit 5000, a quality management DB 5007 configured by a quality management table as shown in FIG. In this quality management table, the image quality (image quality) of the image data relayed by the relay device 30 is associated and managed according to the delay time (ms) of the image data at the request source terminal 10A or the destination terminal 10B.

(Each functional part of the remote communication management system)
Next, each functional unit of the remote communication management system 50 will be described in detail. In the following, in describing each part of the remote communication management system 50, among the components shown in FIG. 3, the relationship with the main components for realizing each part of the remote communication management system 50 is also shown. explain.

  The transmission / reception unit 51 is executed by the network I / F 209 illustrated in FIG. 3, and transmits / receives various data (information) to / from other terminals, devices, or systems via the communication network 2. The terminal authentication unit 52 searches the terminal authentication management DB 5002 in the storage unit 5000 using the terminal ID and password included in the login request information received via the transmission / reception unit 51 as search keys, and is identical to the terminal authentication management DB 5002. Terminal authentication is performed by determining whether or not the terminal ID and password are managed. The state management unit 53 stores the terminal ID of the request source terminal 10A, the operation status of the request source terminal 10A, the remote status in the terminal management table (see FIG. 10) in order to manage the operation status of the request source terminal 10A that has requested login. The communication management system 50 stores and manages the reception date and time when the login request information is received and the IP address of the request source terminal 10A in association with each other.

  The terminal extraction unit 54 searches the destination list management table (see FIG. 11) using the terminal ID of the request source terminal 10A requested to log in as a key, and can select candidate destination terminals 10B that can remotely communicate with the request source terminal 10A. The terminal ID is extracted by reading the terminal ID. Further, the terminal extraction unit 54 searches the destination list management table (see FIG. 11) using the terminal ID of the request source terminal 10A that requested the login as a key, and sets the terminal ID of the request source terminal 10A as a candidate for the destination terminal 10B. As well as the terminal IDs of the other request source terminals 10A registered.

  The terminal status acquisition unit 55 searches the terminal management DB table (see FIG. 10) using the candidate terminal ID of the destination terminal 10B extracted by the terminal extraction unit 54 as a search key, and is extracted by the terminal extraction unit 54. The operating state is read for each terminal ID. Thereby, the terminal state acquisition unit 55 can acquire the operation state of the candidate of the destination terminal 10B that can remotely communicate with the request source terminal 10A that has requested the login. Further, the terminal state acquisition unit 55 searches the terminal management table (see FIG. 10) using the terminal ID extracted by the terminal extraction unit 54 as a search key, and also acquires the operating state of the request source terminal 10A that requested the login. To do.

  The narrowing unit 56 performs a primary narrowing process before the final narrowing process in order to support a final narrowing process that finally narrows down from a plurality of relay apparatuses 30 to one relay apparatus 30. A terminal IP address extraction unit 56b, a primary selection unit 56c, and a priority determination unit 56d are included. Among these, the selection session ID generation unit 56 a generates a selection session ID used for executing a session for selecting the relay device 30. Based on the terminal ID of the request source terminal 10A and the terminal ID of the destination terminal 10B included in the start request information sent from the request source terminal 10A, the terminal IP address extracting unit 56b The IP address of each corresponding remote communication terminal 10 is extracted. The primary selection unit 56c selects the relay device ID of the relay device 30 whose operation state is “ON line” from among the relay devices 30 managed in the relay device management table (see FIG. 7). The relay device 30 is selected.

  Further, the primary selection unit 56c searches the relay device management table (see FIG. 8) based on the IP address of the request source terminal 10A and the IP address of the destination terminal 10B extracted by the terminal IP address extraction unit 56b. By doing so, it is investigated whether each dot address of the IP address of the selected relay device 30 is the same as or different from each dot address in each IP address of the request source terminal 10A and the destination terminal 10B. Further, for each relay device, the primary selection unit 56c has the higher two points with the higher points of the integrated points obtained by integrating the higher points with respect to the remote communication terminal 10 at the address priority points and the transmission speed priority points. By selecting the relay device 30, the relay device 30 is further selected.

  In the present embodiment, the top two relay devices 30 with high points are selected. However, the present invention is not limited to this. If one relay device 30 can be narrowed down as many as possible, the top points with high points are selected. Three or more relay devices 30 may be selected.

  The priority determination unit 56d refers to the priority management table (see FIG. 13) and determines an address priority point for each relay device 30 investigated by the primary selection unit 56c. Further, the priority determination unit 56d searches the transmission rate priority management table (see FIG. 14) based on the maximum data transmission rate of each relay device 30 managed in the relay device management table (see FIG. 8). Thus, the transmission speed priority point is determined for each relay device 30 narrowed down by the primary selection unit 56c.

  The session management unit 57 stores the selection session ID generated by the selection session ID generation unit 56a, the terminal ID of the request source terminal, and the terminal ID of the destination terminal in the session management table (see FIG. 12) of the storage unit 5000. Store and manage in association. Further, the session management unit 57 relays the relay device 30 finally selected by the selection unit 16c of the remote communication terminal 10 for each session ID for selection with respect to the session management table (see FIG. 11). Store and manage device IDs.

  The quality determining unit 58 searches the quality management table (see FIG. 15) using the delay time as a search key, and extracts the image quality of the corresponding image data, thereby determining the image quality of the image data to be relayed to the relay device 30. To do. The storage / reading processing unit 59 is executed by the HDD 205 shown in FIG. 3, and stores various data in the storage unit 5000 or reads various data stored in the storage unit 5000. The delay time management unit 60 extracts a corresponding terminal ID by searching the terminal management table (see FIG. 10) using the IP address of the destination terminal 10B as a search key, and further extracts a session management table (see FIG. 12). In the session management table (1), the delay time indicated by the delay information is stored and managed in the field of delay time in the record including the extracted terminal ID.

<< Operation procedure >>
FIG. 16 is an example of a sequence diagram illustrating processing for transmitting and receiving image data and audio data between the remote communication terminals 10. Processing for transmitting and receiving image data and audio data in order to perform a video conference between the request source terminal 10aa and the destination terminal 10db will be described.

  First, the request source terminal 10aa transmits the image data of the subject imaged by the imaging unit 14a and the audio data of the audio input by the audio input unit 15a from the transmission / reception unit 11 to the relay device 30a via the communication network 2. (Step S81). In the present embodiment, high-quality image data including three resolutions of low resolution, medium resolution, and high resolution, and audio data are transmitted. Thereby, in the relay device 30a, the transmission / reception unit 31 receives the image data and the audio data of the three resolutions. Then, the data quality confirmation unit 33 searches the change quality management table (see FIG. 7) using the IP address “1.3.2.4” of the destination terminal 10db as a search key, and the image quality of the corresponding relayed image data Is extracted to check the quality of the image data to be relayed (step S82). In the present embodiment, the image quality of the confirmed image data is “high quality” and is the same as the image quality of the image data received by the transmission / reception unit 31. The voice data is transferred to the destination terminal 10db (step S83). As a result, the destination terminal 10db receives the image data and the audio data at the transmission / reception unit 11, the image display control unit 14b displays an image based on the image data on the display 100, and the audio output unit 15b is based on the audio data. Audio can be output.

  Next, the delay detection unit 17 of the remote communication terminal 10db detects the delay time of reception of the image data received by the transmission / reception unit 11 at regular time intervals (for example, every second) (step S84). In the present embodiment, the following description is continued for a case where the delay time is 200 (ms).

  The transmission / reception unit 11 of the destination terminal 10db transmits delay information indicating the delay time “200 (ms)” to the remote communication management system 50 via the communication network 2 (step S85).

  Thereby, the remote communication management system 50 can grasp the delay time and the IP address “1.3.2.4” of the remote communication terminal 10db that is the transmission source of the delay information.

  Next, the delay time management unit 60 of the remote communication management system 50 searches the terminal management table (see FIG. 10) using the IP address “1.3.2.4” of the destination terminal 10db as a search key. The corresponding terminal ID “01db” is extracted, and further, in the session management table (see FIG. 12) of the session management DB 5005, the delay time field portion of the record of the terminal ID “01db” is indicated by the delay information. The stored delay time “200 (ms)” is stored and managed (step S86).

  Next, the quality determination unit 58 searches the quality management table (see FIG. 14) using the delay time “200 (ms)” as a search key, and extracts the image quality “medium image quality” of the corresponding image data. The image quality is determined as “medium image quality” (step S87).

  Next, the transmission / reception unit 51 uses the relay device ID “111a” associated with the terminal ID “01db” in the session management table (see FIG. 12) as a search key, and the relay device management DB table (see FIG. 8). And the IP address “1.2.1.2” of the corresponding relay device 30a is extracted (step S88). Then, the transmission / reception unit 51 transmits quality information indicating the image quality “medium image quality” of the image data determined in step S87 to the relay device 30a via the communication network 2 (step S89). This quality information includes the IP address “1.3.2.4” of the destination terminal 10db used as the search key in step S86. As a result, in the relay device 30a, the change quality management unit 34 adds the IP address “1.3.2” of the remote communication terminal 10 (destination terminal 10db in this case) to the change quality management table (see FIG. 7). .4 ”and the image quality“ medium image quality ”of the image data to be relayed are stored and managed in association with each other (step S90).

  Next, the remote communication terminal 10aa continues to transmit the high-quality image data including the low image quality, the medium image quality, and the high image quality, and the audio data to the relay device 30a in the same manner as in the above step S81 ( Step S91). As a result, in the relay device 30a, as in step S82, the data quality confirmation unit 33 uses the IP address “1.3.2.4” of the destination terminal 10db as a search key to change the quality management table (see FIG. 7). ) And the image quality “medium image quality” of the corresponding image data to be relayed is extracted to check the image quality of the image data to be relayed (step S92). In this embodiment, since the image quality of the confirmed image data is “medium image quality” and lower than the image quality “high image quality” of the image data received by the transmission / reception unit 31, the data quality changing unit 35 The image quality of the image data is changed by suppressing the image quality from “high image quality” to “medium image quality” (step S93). Then, the transmission / reception unit 31 transmits the image data in which the image quality of the image data is changed to “medium image quality” and the audio data in which the sound quality of the audio is not changed to the remote communication terminal 10db via the communication network 2. (Step S94). In this way, when reception delay occurs at the destination terminal 10db that receives the image data, the relay device 30a changes the quality of the image so that the person participating in the video conference does not feel uncomfortable. can do.

  Next, a procedure for displaying the material data stored in the HDD 205 by the external input device 40 after the relay device 30 is determined and sharing the entire screen will be described with reference to FIG. Here, an example will be described in which information displayed by the external input device 40aa connected to the remote communication terminal 10aa is displayed on the remote communication terminal 10db which is the destination terminal.

  As described above, when the relay device 30 is determined, the relay device ID “111a” and the IP address “1.3.2.4” of the destination terminal 10db transmitted by the remote communication management system 50 in S67-21 are set. The transmission / reception unit 11 of the remote communication terminal 10aa receives the received relay device ID “111a” and the IP address “1.3.2.4”, and the storage / reading processing unit 19 stores it in the storage unit 1000 (step S67-). 22).

  When the external input device 40aa and the remote communication terminal 10aa are connected to each other, the connection detection unit 42 of the external input device 40aa detects this (step S70). When the connection detection unit 42 detects that the external input device 40aa and the remote communication terminal 10aa are connected to each other, the installation determination unit 43 determines whether or not the display data transmission unit 452 has already been installed (step S71). ). If it is determined in step S71 that the display data transmission unit 452 is not installed, the program acquisition unit 44 acquires and installs the data transmission unit 452 stored in the remote communication terminal 10aa (step S72). .

  When the display data transmission unit 452 is installed in step S72, the external input device 40aa requests the remote communication terminal 10aa to allow the display data acquisition unit 451 to execute processing (step S73). When the remote communication terminal 10aa permits the external input device 40aa to execute the processing of the display data acquisition unit 451, the display data acquisition unit 451 acquires display data (step S74). Next, the display data transmission unit 452 transmits the display data acquired by the display data acquisition unit 451 to the remote communication terminal 10aa (step S75).

  If it is determined in step S71 that the display data transmission unit 452 is installed, the process proceeds to step S73 and subsequent steps.

  When the external information transmitting / receiving unit 18 of the remote communication terminal 10db as the transmission destination receives the display data, the storage / reading processing unit 19 sends the relay device ID “111a” stored in the storage unit 1000 and the remote communication terminal 10db as the destination. IP address “1.3.2.4” is acquired (step S77). Then, for the relay device 30 indicated by the relay device ID “111a” acquired in step S77, the transmission / reception unit 11 displays the display data and the IP address “1.3.2” of the remote communication terminal 10db as the destination. .4 "is transmitted (step S78). When the display data transmitted from the remote communication terminal 10aa is received in step S78, the relay device 30 changes the quality of the display data based on the IP address “1.3.2.4” of the remote communication terminal 10db (step S78). In step S79, display data is transmitted to the remote communication terminal 10db (step S80). Details of the process in step S77 are the same as the process for changing the quality of the audio data and the image data (steps S81 to S94) described above, and thus the description thereof is omitted.

  When the transmission / reception unit 11 of the remote communication terminal 10db receives the display data transmitted from the relay device 30, the image display control unit 14b displays the display data.

  FIG. 18 shows a screen example of a screen displayed by the image display control unit 14b of the remote communication terminal 10db. In the example shown in FIG. 18, display data displayed by the external input device 40aa is displayed on the left side of the screen. As shown in FIG. 18, one of the features is that the display data includes a mouse cursor 505. Note that windows other than those shown are other display data and operation menus.

  In the upper right part of the screen, image data captured by the imaging unit 14a of the remote communication terminal 10aa and transmitted by the transmission / reception unit 11 is displayed. In the lower right part of the screen, image data captured by the imaging unit 14a of the remote communication terminal 10db is displayed.

[Display instruction information]
Next, acquisition of instruction information will be described. A user who operates the external input device 40aa operates the mouse 212 and the keyboard 211 to move the mouse cursor within the display device 400. Since the document is displayed on the display device 400, the position of the mouse cursor indicates the character of the document, one line, a figure, or a certain range.

  In the present embodiment, the coordinate data in the display data of the mouse cursor is instruction information. Note that the user may move the mouse cursor with a track pointer without using the mouse 212, or move the mouse cursor by touching the touch panel with a fingertip. In this embodiment, the type of the pointing device is not limited. The mouse cursor may have any shape or color.

  In addition, it is preferable to include an image of a mouse cursor in the instruction information. By doing so, it is possible to display a mouse cursor having the same shape and color on the remote communication terminals 10aa and 10db. However, if there is only coordinate data, the image of the icon for the mouse cursor that the remote communication terminal 10aa has can be combined with the display data. Similarly, the remote communication terminal 10db can synthesize the display data using an image (icon) of the mouse cursor of the own device.

  Further, if the external input device 40, not the remote communication terminal 10, stores the mouse cursor icon image data in advance, the external input device 40 can synthesize the mouse cursor image data with the display data.

  FIG. 19 is an example of a sequence diagram illustrating a procedure in which the external input device 40aa takes in the image data and instruction information of the material at the time of “display data acquisition in step S75” in FIG. “75: Display data acquisition” is a step corresponding to S75. In FIG. 19, “75-1: acquisition of image data”, “75-2: acquisition of instruction information”, and “75-3: combination of instruction information with image data” are performed in step S75.

  First, the screen information acquisition unit 4513 acquires image data of a material displayed on the external input device (for example, a notebook PC) 10aa (S75-1). The image data of the material is temporarily stored in the RAM 203 or the like.

  Next, the instruction information acquisition unit 4512 acquires the image data of the mouse cursor in “75-2: Instruction information acquisition” (S75-21). The image data of the mouse cursor is temporarily stored in the RAM 203 or the like. Also, the instruction information acquisition unit 4512 acquires coordinate data of the mouse cursor (S75-22).

  Next, the display data creation unit 4514 creates display data by synthesizing the image data of the mouse cursor at the position indicated by the coordinate data of the image data of the material (S75-3).

  FIG. 20 is an example of a diagram schematically illustrating the synthesis of two images. The image data 501 in FIG. 20 is the entire screen displayed on the display device 400 by the external input device 40aa, in which a window 5011 for displaying materials is displayed. The mouse cursor can move the entire image data 501, and the window 5011 can also occupy the entire image data 501. The screen information acquisition unit 4513 can capture the entire image data of the image data 501 and the image data of the window 5011 alone. In the present embodiment, this difference is referred to as image data of a material without distinction.

  The mouse cursor image data 502 is acquired in S75-2, and the mouse cursor coordinate data 503 is acquired in S75-3. The coordinate data is a coordinate indicating the two-dimensional position of the mouse cursor in pixel units with the upper left vertex of the image data of the material as the origin.

  Therefore, the display data creation unit 4514 can synthesize the mouse cursor with the image data of the material, for example, by arranging the tip of the mouse cursor at the position of the material image data 501 or 5011 indicated by the coordinate data. The composition means that image data of the mouse cursor is arranged on the image data of the material (replaces pixel values of overlapping pixels) to form one image data.

  Display data 504 in the lower part of FIG. 20 created in this way is transmitted to the remote communication terminal 10aa in step S76 of FIG. 17 or FIG. 19, and is transmitted from the remote communication terminal 10aa to the relay device 30.

  As described above, the remote communication system 1 according to the present embodiment can share the coordinate data and image data of the mouse cursor indicating the position on the screen in the display data such as materials among the remote communication terminals.

  In the first embodiment, the frequency at which the screen information acquisition unit 4513 acquires the image data of the material is the same as the frequency at which the instruction information acquisition unit 4512 acquires the instruction information. In the present embodiment, a description will be given of a remote communication system 1 that reduces the acquisition frequency of image data of a material than the acquisition frequency of instruction information. By doing so, it is possible to reduce the load on the external input device 40 and smooth the movement of the mouse cursor.

  For example, in a video conference, when each remote communication terminal 10 shares presentation material, the position of the mouse cursor changes relatively frequently compared to the screen of the presentation material. That is, in many cases, only the mouse cursor moves while the screen of the material remains stationary (the same).

  For this reason, even if the instruction information acquisition unit 4512 acquires the instruction information n times and only once when the screen information acquisition unit 4513 acquires the image data of the material, it is considered that the user hardly feels uncomfortable. Using this concept, the display data creation unit 4514 of this embodiment uses the image data of the material last acquired by the screen information acquisition unit 4513 n times (uses it again) and acquires it every cycle. Display data is created using the instruction information.

  FIG. 21 is an example of a flowchart showing a procedure in which the external input device 40aa takes in the image data and instruction information of the material at the time of “display data acquisition in step S75” in FIG. For convenience of explanation, FIG. 21 is not a sequence diagram but a flowchart.

  The screen information acquisition unit 4513 controls whether or not to acquire the image data of the material using the variable i. Here, the information acquisition frequency determination unit 4511 determines the acquisition frequency of the image data of the material and the instruction information. However, in this embodiment, the information acquisition frequency determination unit 4511 acquires the acquisition frequency of the instruction information and the acquisition of the instruction information. The acquisition frequency of the image data of the material with respect to the frequency is determined. For example, the acquisition frequency of the image data of the material is determined as once per n times of the acquisition frequency of the instruction information. Therefore, the information acquisition frequency determination unit 4511 determines this “n” times (for example, a natural number of 2 or more and the upper limit is 100). There are various determination methods. For example, there are methods according to the specifications of the external input device 40aa or accepting user settings.

  First, the screen information acquisition unit 4513 is initialized by setting 1 to the variable i when the external input device 40aa is activated to acquire the image data of the material for the first time. Then, the screen information acquisition unit 4513 determines whether i = 1 (S75-0), and acquires the image data of the material only when i = 1 (S75-1). A screen information acquisition unit 4513 updates the image data of the last material stored in the RAM 203.

  If i is not 1, the instruction information acquisition unit 4512 acquires the image data of the mouse cursor without acquiring the image data of the material (S75-21). Next, the instruction information acquisition unit 4512 acquires coordinate data of the mouse cursor (S75-22).

  Next, the display data creation unit 4514 combines the image data of the mouse cursor with the image data of the material (S75-3). When a series of processing for creating the display data is completed, the display data creating unit 4514 increments the variable i by one.

  Then, the display data creation unit 4514 determines whether i = n (S75-6). If i = n, the variable i is initialized to “1” (S75-7).

  Through the processing as described above, the external input device 40aa can acquire the image data of the material once every time the instruction information is acquired n times.

  Therefore, it is possible to reduce the acquisition frequency of image data of a material with little movement, increase the acquisition frequency of a mouse cursor with a lot of movement, and transmit a large amount of display data while keeping the processing load low. Therefore, display data can be transmitted to the other party of the remote communication terminal 10 efficiently (low load) and effectively (smooth cursor operation).

  In Example 1 and Example 2, although there is a difference in time interval, the external input device 40 acquires the image data and material information of the material at a fixed timing. However, the screens shared between the plurality of remote communication terminals 10 (documents and mouse cursor positions) are not determined in advance, but change as the user operates them.

  Therefore, in this embodiment, the external input device 40 can change the acquisition frequency of the image data of the material and the acquisition frequency of the instruction information in accordance with the change of the image data of the displayed material or the change amount of the position of the mouse cursor. Will be described. If there is no change in the position of the document or the mouse cursor, the external input device 40 does not acquire the image data or instruction information of the document, so that display data can be efficiently created and transmitted.

  22 and 23 are examples of a diagram illustrating a procedure in which the external input device 40aa takes in instruction information at the time of “display data acquisition in step S75” in FIG. Among these, FIG. 22 is a sequence diagram, and FIG. 23 is a flowchart for explaining in detail a part of the processing of FIG.

  22 and 23, the display data creation unit 4514 creates display data only when there is a change in the image data of the material or when the mouse cursor has moved a distance m or more. This is a procedure for transmitting to the terminal 10db. Therefore, if either the image data of the material or the instruction information is updated, the external input device 40aa synthesizes the image data of the material and the instruction information and transmits the display data, but neither of them has been updated. Repeats the same cycle without any composition or transmission.

  First, the instruction information acquisition unit 4512 sets “y” to the variable m when the external input device 40aa is activated. “y” is a constant and is a threshold for determining the amount of movement of the mouse cursor, such as several tens of pixels.

  First, the screen information acquisition unit 4513 acquires the image data of the material displayed on the external input device (for example, notebook PC) 40aa (S75-1). The image data of the material is temporarily stored in the RAM 203 or the like.

  Then, the screen information acquisition unit 4513 replaces the previous image data with the current image data only when the image data of the material stored in the RAM 203 is different from the previous image data this time (S75-11).

  The process of step S75-11 will be described with reference to the flowchart of FIG. The screen information acquisition unit 4513 determines whether the image data of the material stored in the RAM 203 this time is different from the previous image data (S75-111). Then, the previous image data is replaced with the current image data (S75-112).

  Here, “the image data of the material is different” means that the previous image data stored in the RAM 203 and the current image data are compared in pixel units, and the pixel value of one pixel is also different. By making such a determination, it can be accurately determined whether or not the image data of the current material and the previous material are different. Also, it can be determined that “the image data of the material is different” when the pixel values of 10 to 100 pixels or more are different, regardless of whether the pixel values of one pixel are different. By making such a determination, it can be determined that “the image data of the material is different” only when the change in the image data of the material is large.

  Further, the determination may be made by sampling the pixel value. That is, the screen information acquisition unit 4513 extracts and compares the pixel values at the same position in the image data of the current and previous materials, for example, every 8 to 16 pixels in the vertical and horizontal directions. It can also be determined that the image data of the materials are different. By doing so, it is possible to speed up the comparison process while maintaining a certain degree of accuracy.

  If the image data of the material is different, the user has operated the keyboard 211 or the mouse 212 to cause a change in the image data of the material. It is possible to determine whether or not “different”. The screen information acquisition unit 4513 performs an operation that causes a change in image data of a currently displayed document that has been registered in advance (a page-up operation from the page-up menu, page-down of the keyboard 211, a page-turning menu of the mouse 212, or a page-turning operation from the right-click menu). When such an operation is detected, it is determined that “the image data of the material is different”. In this case, since it is not necessary to capture the image data of the material for comparison, the processing load on the external input device 40aa can be reduced.

  Returning to FIG. 22, next, the instruction information acquisition unit 4512 acquires the coordinate data of the mouse cursor (S75-22). The coordinate data of the mouse cursor is temporarily stored in the RAM 203 or the like.

  The instruction information acquisition unit 4512 compares the coordinate data of the mouse cursor acquired this time with the coordinate data of the mouse cursor acquired this time, and the image data of the previous mouse cursor is only displayed when the distance is m or more. Replace with the coordinate data of the current mouse cursor (S75-21).

  The process of step S75-21 will be described with reference to the flowchart of FIG. The instruction information acquisition unit 4512 determines whether the distance is greater than or equal to m by comparing the coordinate data of the mouse cursor acquired this time with the coordinate data of the mouse cursor acquired last time (S75-211).

  “Distance is m or more” means that the Euclidean distance between the coordinates of two mouse cursors is m or more. Instead of the Euclidean distance between the two coordinates, the determination criterion may be that the amount of change in either the x coordinate or the y coordinate is m or more.

  Further, in addition to “distance is greater than or equal to m”, or regardless of whether or not “distance is greater than or equal to m”, instruction information acquisition unit 4512 determines whether the movement speed of the mouse cursor is greater than or equal to v and You may determine whether an acceleration is more than (alpha). Assuming that the procedure of FIG. 23 is executed at almost constant intervals, the speed is calculated by dividing the difference in the coordinate data of the mouse cursor acquired by the instruction information acquisition unit 4512 by the cycle time. Further, since the acceleration is a change speed of the speed per unit time, the acceleration is calculated by dividing the true change of the cursor speed by the cycle time.

  The instruction information acquisition unit 4512 calculates a speed and an acceleration from a plurality of past coordinate data (performs an averaging process if necessary), and the speed is a speed v or higher and / or an acceleration is α or higher Coordinate data can be updated.

  If the distance is greater than or equal to m, the instruction information acquisition unit 4512 replaces the previous coordinate data of the mouse cursor with the current coordinate data (S75-212).

  Since the position of the mouse cursor has moved greatly, the instruction information acquisition unit 4512 acquires the image data of the mouse cursor (S75-213). The image data of the mouse cursor is temporarily stored in the RAM 203 or the like.

  Returning to FIG. 22, the display data creation unit 4514 then moves the mouse cursor to the position indicated by the coordinate data of the image data of the material when one or more of the image data of the material or the instruction information is updated. Display data is created by combining the image data (S75-3).

  The process of step S75-3 will be described with reference to the flowchart of FIG. The display data creation unit 4514 determines whether one or more of the image data of the material or the instruction information has been updated (S75-31). The presence / absence of update is determined, for example, by monitoring an update flag for image data of a material and an update flag for coordinate data of a mouse cursor.

  When at least one of the document image data and the mouse cursor coordinate data is updated, the display data creation unit 4514 synthesizes the mouse cursor image data with the document image data (S75-32).

  In this way, by creating display data only when there is a change in either the document or the mouse cursor, the processing load and communication load can be reduced while the change in the screen is scarce.

  In addition to comparing the movement distance, speed, or acceleration of the coordinate data of the mouse cursor with a threshold value, the information acquisition frequency determination unit 4511 may increase or decrease the acquisition frequency of the instruction information according to the movement distance, speed, or acceleration. . For example, after the execution of FIG. 23, the information acquisition frequency determination unit 4511 executes only steps S75-22, S75-21, and S75-3 of FIG. 23 k (a natural number of 1 or more) times. In this way, it is possible to increase the acquisition frequency of the instruction information while maintaining the acquisition frequency of the image data of the material. If this “k” is increased according to the magnitude of the moving distance, speed or acceleration, the acquisition frequency of the instruction information can be increased as the moving distance, speed or acceleration is increased.

  By doing so, the instruction information can be obtained more frequently as the amount of movement of the mouse cursor increases, and the movement of the mouse cursor can be smoothed even if the mouse cursor moves faster.

  In the third embodiment, the display data creation frequency can be reduced, but the comparison of the image data of the materials is performed for each cycle. In the present embodiment, as in the second embodiment, a remote communication system 1 in which the screen information acquisition unit 4513 reduces the comparison frequency of image data of materials will be described. By doing so, it is possible to reduce the communication load while further reducing the load of the external input device 40aa than in the third embodiment.

  FIG. 24 is an example of a flowchart illustrating a procedure in which the external input device 40aa takes in the image data and instruction information of the material at the time of “display data acquisition in step S75” in FIG. The procedure of FIG. 24 is the same as that of FIG. 23, but in step S75-0, the screen information acquisition unit 4513 determines whether i = 1 (S75-0), and only when i = 1. The screen information acquisition unit 4513 acquires the image data of the material (S75-1). The processing (S75-11, 75-22, 75-21, 75-3) when the image data of the material is acquired is the same as that of the third embodiment.

  If i = 1 is not satisfied in step S75-0, the screen information acquisition unit 4513 does not acquire the image data of the material, and the instruction information acquisition unit 4512 acquires the image data of the mouse cursor (S75-22).

  The subsequent processing is the same as in the third embodiment, but when the display data creation unit 4514 combines the image data of the mouse cursor with the image data of the material (S75-32), the display data creation unit 4514 increases the variable i by one. .

  Then, the display data creation unit 4514 determines whether i = n (S75-6). If i = n, the variable i is initialized to “1” (S75-7).

  Through the above processing, the image data of the material may be acquired once every time display data is created n times and compared with the image data of the past material, and the processing of the external input device 40aa The load can be reduced.

  Accordingly, the acquisition frequency of the image data of the material can be reduced below the acquisition frequency of the instruction information, and if the image data of the current material and the previous material are not different, there is no need to update the image data in the RAM 203. If there is no change in either the document or the mouse cursor, there is no need to create display data. For this reason, the processing load / communication load can be reduced as compared with the third embodiment.

  Also in this embodiment, the information acquisition frequency determination unit 4511 can increase or decrease the acquisition frequency of the instruction information according to the moving distance, speed, or acceleration of the mouse cursor, and the acquisition frequency of the image data of the material is suppressed. The remote communication terminal 10db can display a mouse cursor that moves smoothly.

DESCRIPTION OF SYMBOLS 1 Remote communication system 10 Remote communication terminal 11 Transmission / reception part (an example of a display data transmission means)
DESCRIPTION OF SYMBOLS 12 Operation input reception part 13 Login request part 14a Imaging part 14b Image display control part 15a Audio | voice input part 15b Audio | voice output part 16 Selection processing part 16a Measurement part 16b Calculation part 16c Selection part 17 Delay detection part 18 External information transmission / reception part (display data) An example of receiving means)
19 Storage / Read Processing Unit 30 Relay Device 40 External Input Device 41 Transmission / Reception Unit 42 Connection Detection Unit 43 Installation Determination Unit 44 Program Acquisition Unit 451 Display Data Acquisition Unit 452 Display Data Transmission Unit 4511 Information Acquisition Frequency Determination Unit 4512 Instruction Information Acquisition Unit 4513 Screen information acquisition unit 4514 Display data creation unit 46 Operation input reception unit 47 Display control unit 48 Mount unit 49 Storage / read processing unit 50 Remote communication management system 400 Display device 1000 Storage unit (an example of destination information storage means)
3000 storage unit 3001 change quality management DB
5000 storage unit 5001 relay device management DB
5002 Terminal authentication management DB
5003 Terminal management DB
5004 Destination list management DB
5005 Session management DB
5006 Priority management DB
5007 Quality Management DB

US Pat. No. 6,760,749

Claims (25)

An external input device connected to the remote communication terminal that distributes data including video captured by the remote communication terminal to other devices,
Display means for displaying an image;
Cursor operation means for receiving an operation of a cursor displayed on the image by the display means;
Image data acquisition means for acquiring image data of the image displayed by the display means;
Cursor information acquisition means for acquiring cursor information including at least coordinate data of a cursor displayed on the image;
Display data creating means for creating display data obtained by combining the cursor image data of the cursor with the image data obtained by the image data obtaining means based on the cursor information;
In response to detecting a connection with the remote communication terminal, a program for transmitting the display data to the remote communication terminal is acquired from the connected remote communication terminal, and the remote communication is performed by the acquired program. when the terminal the telecommunications terminal is delivering data including a video imaged with respect to the other devices, separately from the data that contains the image, distributing the display data to the other device Transmitting means for transmitting the display data to the remote communication terminal,
By transmitting the display data by the transmission means, the data including the video and the display data are displayed on the same screen of the other device.
An external input device characterized by that. The cursor information includes the cursor image data of the cursor in addition to the coordinate data,
The cursor information acquisition means acquires the coordinate data and the cursor image data.
The external input device according to claim 1.   The external input device according to claim 1, wherein the cursor is a mouse cursor. The display data creation means creates the display data by synthesizing the cursor image data at a position indicated by the coordinate data of the image data.
The external input device according to claim 2.   The external input device according to claim 1, further comprising an information acquisition frequency determining unit that determines a first frequency with which the cursor information acquiring unit acquires the cursor information. The information acquisition frequency determining means determines a frequency at which the image data acquiring means acquires the image data as a second frequency different from the first frequency.
The external input device according to claim 5.   The external input device according to claim 6, wherein the first frequency is greater than the second frequency. Change detection means for detecting that the image data of the image displayed on the display means has changed, or that the coordinate data has changed more than a threshold,
The display data creation means creates the display data when the change detection means detects a change in the image data or detects that the coordinate data has changed more than a threshold value.
The external input device according to claim 5. The change detecting means detects that the image data has changed when the amount of change in the image data is equal to or greater than a threshold;
The external input device according to claim 8. The information acquisition frequency determining means determines the frequency at which the image data acquiring means acquires the image data as a second frequency smaller than the first frequency.
The external input device according to claim 8. The information acquisition frequency determination means increases the first frequency as the change amount, change speed, or acceleration of the coordinate data increases.
10. The external input device according to claim 8 or 9, wherein   The external input device according to claim 1, wherein the data including the video includes voice acquired by the remote communication terminal.   The remote communication terminal and the other device distribute data including the video via a relay device, and the display data is transmitted from the remote communication terminal to the other device via the relay device. The external input device according to claim 1, wherein the external input device is an external input device. A display data creation method for an external input device connected to the remote communication terminal that distributes data including video captured by the remote communication terminal to other devices,
A display means displaying an image;
A step of accepting an operation of a cursor displayed on the image by the display means;
An image data obtaining unit obtaining image data of the image displayed by the display unit;
Cursor information acquisition means acquires cursor information including at least coordinate data of a cursor displayed on the image;
Display data creation means, based on the cursor information, creating display data obtained by combining the cursor image data of the cursor with the image data acquired by the image data acquisition means;
In response to detecting the connection with the remote communication terminal , the transmission means acquires a program for transmitting the display data to the remote communication terminal from the connected remote communication terminal, and the acquired program , when the telecommunication terminal is the remote communication terminal to said another device is delivering data including a video imaged separately from the data that contains the image, the other said display data Transmitting the display data to the remote communication terminal so as to be distributed to a device of
Transmitting the display data by the transmitting step to display the data including the video and the display data on the same screen of the other device;
A display data creation method characterized by comprising:   15. The display data creation method according to claim 14, wherein the information acquisition frequency determination means includes a step of determining a first frequency at which the cursor information acquisition means acquires the cursor information.   16. The display data creation method according to claim 14, wherein the data including the video includes sound acquired by the remote communication terminal.   The remote communication terminal and the other device distribute data including the video via a relay device, and the display data is transmitted from the remote communication terminal to the other device via the relay device. The display data creation method according to claim 14, wherein the display data creation method is a display data creation method. To an external input device connected to a remote communication terminal that distributes data including video captured by the remote communication terminal to other devices,
A display step for displaying an image on the display means;
A cursor operation step for receiving an operation of a cursor displayed on the image by the display step;
An image data acquisition step of acquiring image data of the image displayed by the display step;
A cursor information acquisition step of acquiring cursor information including at least coordinate data of a cursor displayed on the image;
Based on the cursor information, a display data creating step for creating display data obtained by combining the cursor image data of the cursor with the image data obtained by the image data obtaining step;
In response to detecting a connection with the remote communication terminal, a program for transmitting the display data to the remote communication terminal is acquired from the connected remote communication terminal, and the remote communication is performed by the acquired program. when the terminal the telecommunications terminal is delivering data including a video imaged with respect to the other devices, separately from the data that contains the image, distributing the display data to the other device Transmitting so that the display data is transmitted to the remote communication terminal;
Transmitting the display data in the transmitting step, and displaying the data including the video and the display data on the same screen of the other device;
A program that executes   19. The program according to claim 18, further comprising an information acquisition frequency determination step for determining a first frequency at which the cursor information acquisition step is executed and the cursor information is acquired.   20. The program according to claim 18 or 19, wherein the data including the video includes audio acquired by the remote communication terminal.   The remote communication terminal and the other device distribute data including the video via a relay device, and the display data is transmitted from the remote communication terminal to the other device via the relay device. The program according to any one of claims 18 to 20, characterized in that: A communication system having an external input device, a first remote communication terminal connected to the external input device, and a second remote communication terminal connected to the first remote communication terminal,
Display means for displaying an image on an external input device;
Cursor operation means for receiving an operation of a cursor displayed on the image by the display means;
Image data acquisition means for acquiring image data of the image displayed by the display means;
Cursor information acquisition means for acquiring cursor information including at least coordinate data of a cursor displayed on the image;
Display data creating means for creating display data obtained by combining the cursor image data of the cursor with the image data obtained by the image data obtaining means based on the cursor information;
Transmitting means for transmitting the display data to the first remote communication terminal;
In response to detecting a connection with the first remote communication terminal, a program for transmitting the display data to the first remote communication terminal is acquired from the connected first remote communication terminal, Means for delivering data including video captured by the first remote communication terminal from the first remote communication terminal to the second remote communication terminal by the acquired program ;
Separately from the data that contains the image, to the second telecommunications terminal from said first telecommunications terminal, and means for delivering the display data,
Means for displaying the data including the video received from the first remote communication terminal and the display data on the same screen of the second remote communication terminal;
A communication system comprising:   23. The communication system according to claim 22, further comprising an information acquisition frequency determining unit that determines a first frequency with which the cursor information acquiring unit acquires the cursor information.   The communication system according to claim 22 or 23, wherein the data including the video includes voice acquired by the first remote communication terminal. The first remote communication terminal and the second remote communication terminal distribute data including the video via a relay device, and the display data is transmitted to the first remote communication via the relay device. The communication system according to any one of claims 22 to 24, wherein the communication system is transmitted from a terminal to the second remote communication terminal .

Images