JP2021174173A - Communication terminal, communication system, and communication method and program - Google Patents

Abstract

To provide a communication terminal capable of displaying a final result even when respective operations are not processed in order.SOLUTION: A communication terminal 1 is configured to display display data, and has: a reception unit 12 which receives an operation targeting the communication terminal; an operation data storage unit 1001 which stores operation data associated with the operation that the reception unit has received as an object of predetermined processing together with identification information on the display data as an object of operation; and a transmission information management unit 21 which replaces operation data in the operation data storage unit for which the predetermined processing has not been performed with the operation data that the reception unit has received when the display data on operation data for which the predetermined processing has not been performed is the same as the display data for which the reception unit has received the operation.SELECTED DRAWING: Figure 8

Description

The present invention relates to communication terminals, communication systems, communication methods, and programs.

Communication systems that communicate with remote locations via communication networks such as the Internet are widespread. Examples of the communication system include a conference system for conducting a remote conference. In a general conference system, a communication terminal used by one of the parties conducting a remote conference converts an image or sound of a subject in a conference room into digital data and transmits the image or sound to the communication terminal used by the other party. Next, the communication terminal used by the other party causes the display to display an image and the speaker to output sound. As a result, it is possible to hold a conference between remote locations in a state close to an actual conference. Further, the image data can be shared by transmitting the image data indicating the material or the like being held or displayed on one communication terminal side to the other communication terminal via the communication network (Patent Document 1). reference).

In addition, at meetings at companies, educational institutions, government agencies, etc., communication terminals such as electronic blackboards that display images on displays and allow users to draw stroke images such as letters, numbers, and figures on these images. It is used (see Patent Document 2). This stroke image is drawn by electronically converting the content drawn by the communication terminal by the user touching and moving it on the display with an electronic pen or hand to generate stroke data such as coordinate data. NS. Further, when a stroke image is drawn on one communication terminal, the stroke data for reproducing the stroke image is transmitted to the other communication terminal via the communication network, so that the same stroke image is also drawn on the other communication terminal. Is displayed.

From the above, by transmitting image data such as materials being displayed or stored by one communication terminal and stroke data for reproducing a stroke image to the other communication terminal via a communication network, the other communication terminal can be used. The communication terminal can display a material image such as a material or a screen background and a stroke image.

However, in a conventional communication terminal, even if the user continuously performs operations such as movement and scaling on display data such as strokes, each operation must be processed in order until the final result is displayed. There was a problem.

In view of the above problems, an object of the present invention is to provide a communication terminal capable of displaying a final result without processing each operation in order.

In view of the above problems, the present invention is a communication terminal that displays display data, and the reception unit that accepts operations on the communication terminal and the operation data related to the operations received by the reception unit are subject to predetermined processing. When the operation data storage unit that stores the identification information of the display data to be operated, the display data of the operation data that has not been subjected to the predetermined processing, and the display data that the reception unit has received the operation are the same. It is characterized by having a transmission information management unit that replaces the operation data of the operation data storage unit that has not been subjected to the predetermined processing with the operation data received by the reception unit.

It is possible to provide a communication terminal capable of displaying the final result without processing each operation in order.

It is a hierarchical diagram of data handled by a communication system. It is a figure explaining the reason why the electronic blackboard transmits the next operation data to a communication management system after the main reflection of the previous operation is completed. It is a figure explaining the flow of data when a stroke is shared by a plurality of bases. It is a figure explaining the flow of data when a stroke is operated with the other base as a starting point. It is a figure explaining the flow of data when the base is only own base. It is a figure explaining the flow of data when the operation of an electronic blackboard is rejected by a communication management system. It is a figure which shows the transmission waiting data when a user moves a stroke continuously. It is a figure which shows the operation data of waiting for transmission, the operation data of a user, and the like. It is the schematic which showed the communication route of this embodiment. It is a use image figure of an electronic blackboard. It is a hardware block diagram of the electronic blackboard which concerns on this embodiment. It is a hardware block diagram of the communication management system, the relay device, and the image storage device which concerns on this embodiment. It is an overall block diagram of the communication system which concerns on this embodiment. It is each function block diagram of the communication system which concerns on this embodiment. It is a conceptual diagram which shows the list table of operation data. It is a conceptual diagram which shows the authentication management table. It is a conceptual diagram which shows the terminal management table. It is a conceptual diagram which shows the destination list management table. It is a conceptual diagram which shows the session management table. It is a conceptual diagram which shows the relay device management table. It is a conceptual diagram which shows the operation data management table. It is a sequence diagram which showed the process of the preparatory stage to start the remote communication between electronic blackboards. This is a screen example of the destination list displayed on the electronic blackboard 1a. It is a sequence diagram which showed the process of starting a remote communication. It is a sequence diagram which showed the process of sharing the image data of a document image and the stroke data of a stroke image. It is a sequence diagram which showed the process of sharing the image data of a document image and the stroke data of a stroke image. It is a figure which shows (a) screen example of electronic blackboard 1a, (b) screen example of electronic blackboard 1b, (c) screen example of electronic blackboard 1a, (d) screen example of electronic blackboard 1b. This is an example of a flowchart illustrating a process in which an electronic blackboard registers operation data in a list table of operation data.

Hereinafter, as an example of the embodiment for carrying out the present invention, the communication system and the communication method performed by the communication system will be described with reference to the drawings.

<Terminology>
The display data is the data displayed on the display of the electronic blackboard. In the present embodiment, it is mainly a stroke, but an image (image) or the like read from a file may be displayed data. The display data includes not only the handwritten data itself, but also text data obtained by converting the handwritten data by character recognition, stamps displayed as fixed characters and marks such as "done" and "secret", and figures such as circles and stars. Data displayed based on the user's operation, such as a straight line, may also be included. Display data is described by the term object in this embodiment.

The handwritten data is data in which a coordinate point sequence in which the user continuously moves the input means on the display is displayed as a locus. Further, a series of operations in which the user presses the input means on the display, moves the input means continuously, and then releases the input means from the display is called a stroke, and the data handwritten by the stroke is called stroke data. The handwritten data has one or more stroke data. In the present embodiment, the stroke data will be described simply by the term stroke for simplification.

The predetermined process is a fixed process performed on the operation data stored in the operation data storage unit. The process of this embodiment means that the electronic blackboard transmits operation data to the communication management system and obtains a gid described later. In addition, a response from another electronic blackboard, waiting for processing of another electronic blackboard, or the like may be a predetermined process.

A communication terminal is a device that realizes remote communication by transmitting at least one of an image and a voice. In this embodiment, an electronic blackboard will be described as an example.

<Supplement to the communication system of this embodiment>
First, the communication system 100 of the present embodiment will be supplemented with reference to FIGS. 1 to 7. FIG. 1 is a hierarchical diagram of data handled by the communication system. In FIG. 1, a stroke handwritten as data to be handled will be described as an example.

Data is managed on a meeting-by-meeting basis. Also, the conference has one or more pages, and each page has one or more strokes. Each stroke is numbered for its own coordinate point sequence, the id (stroke identification information) numbered by the electronic blackboard, and the stroke operation transmitted from the electronic blackboard by the communication management system corresponding to the server. It has a gid (global ID). In FIG. 1, the coordinate point sequence of the stroke is omitted. The global ID is the identification information of the operation and is not changed even if it is transmitted to each electronic blackboard.

The pages of the electronic blackboard can be switched, and each stroke has "parent", which is the identification information (parent) of the page having the stroke, in order to identify which page the electronic blackboard has the line written on.

The information contained in these strokes and pages keeps the operation consistent. When the stroke is moved, scaled, changed in color / thickness, etc., the id is inherited as it is, but the gid is numbered (new) each time it is operated.

Subsequently, the provisional reflection and the main reflection will be described with reference to FIG. FIG. 2 is a diagram for explaining the reason why the electronic blackboard transmits the next operation data to the communication management system after the main reflection of the previous operation is completed.

As shown in FIG. 2A, the user may operate the same stroke 201 a plurality of times in succession. In FIG. 2A, the user moves the stroke 201 twice in I and II.

In this case, it is generally processed in the following order. FIG. 2B is a diagram illustrating a general processing sequence. The general term here means "slow operation" or "sufficient communication band".
1. 1. The user performs the move of I.

(A) The electronic blackboard temporarily reflects the operation data of I. Temporary reflection means to reflect the operation immediately. The movement itself is executed in real time by the provisional reflection. Temporary reflection is the state until the gid is notified from the communication management system. Therefore, the stroke 201 moves to the center of FIG. 2 (b).

(B) The electronic blackboard transmits the operation data of I to the communication management system.
2. The communication management system sends the response (gid) of the operation data of I to the electronic blackboard.

(A) The electronic blackboard reflects the operation data of the provisional reflection. This reflection means that the operation data is reflected in the stroke 201 in the order of gid. Even if the electronic blackboard reflects the operation data of I, the stroke 201 does not change (the operation of II has not been performed yet).
3. 3. The user performs the II move.

(A) The electronic blackboard temporarily reflects the operation data of II. Therefore, the stroke 201 moves to the right in FIG. 2 (b).

(B) The electronic blackboard transmits the operation data of II to the communication management system.
4. The communication management system sends the response (gid) of the operation data of II to the electronic blackboard.

(A) The electronic blackboard reflects the operation data of the provisional reflection. Even if the electronic blackboard reflects the operation data of II, the stroke 201 does not change. As described above, no particular inconvenience occurs in the general processing order.

Next, with reference to FIG. 2C, a case where the user operates the same stroke before the response from the communication management system is returned will be described.
1. 1. The user performs the move of I.

(A) The electronic blackboard temporarily reflects the operation data of I. Therefore, the stroke 201 moves to the center of FIG. 2 (c).

(B) The electronic blackboard transmits the operation data of I to the communication management system.
2. The user performs the II move.

(A) The electronic blackboard temporarily reflects the operation data of II. Therefore, the stroke 201 moves to the right in FIG. 2 (c).

(B) The electronic blackboard transmits the operation data of II to the communication management system.
3. 3. The communication management system sends the response of the operation data of I to the electronic blackboard.

(A) The electronic blackboard reflects the operation data of the provisional reflection. As a result, the stroke returns to the place where I was operated (center of FIG. 2C).
4. The communication management system sends the response of the operation data of II to the electronic blackboard.

(A) The electronic blackboard reflects the operation data of the provisional reflection. As a result, the stroke moves to the place operated in II (right of FIG. 2C).

If the response from the communication management system is delayed in this way (when the gid is operated continuously before returning), the stroke will not be displayed as intended by the user (after the stroke returns to its original position, it will be displayed again. , Seems to move to the destination).

In order to suppress such a phenomenon, the electronic blackboard confirms whether the target object of the operation data is included in the already transmitted operation data before transmitting the operation data temporarily reflected on the electronic blackboard (that is,). , Check if it is included in the operation data waiting for a response that has not received the gid). If it is included, the electronic blackboard does not send the temporarily reflected operation data to the communication management system (add it to the list waiting to be sent). When the gid of the temporarily reflected operation data is returned, the method of transmitting the next operation data to the communication management system in time is adopted. Therefore, it takes time for the final operation to be reflected.

Hereinafter, FIGS. 3 to 6 will explain the flow of data shared by the communication system 100 on the premise of provisional reflection and main reflection.

FIG. 3 is a diagram illustrating a data flow when strokes are shared by a plurality of bases.
(i) The user moves stroke A.
(ii) The electronic blackboard 1a generates operation data A (operation content, id of the object to be operated) and temporarily reflects it on itself. Stroke A moves due to the provisional reflection.
(iii) The electronic blackboard 1a transmits the operation data A to the communication management system 5.
(iv) The communication management system 5 assigns a gid to the received operation data A.
(v) The communication management system 5 transmits operation data A (including gid) to all the electronic blackboards 1b participating in the same conference, including the electronic blackboard 1a.
(vi) Each electronic blackboard 1 reflects the received operation data A in the order of gid. By this reflection, the operation data A is reflected on the electronic blackboard 1a again.

FIG. 4 is a diagram illustrating a data flow when a stroke is operated starting from another base.
(i) The electronic blackboard 1b generates operation data B (operation content, object to be operated) and temporarily reflects it on itself. The electronic blackboard 1b transmits the operation data B to the communication management system 5.
(ii) The communication management system 5 assigns a gid to the received operation data B.
(iii) The communication management system 5 transmits operation data B (including gid) to all the electronic blackboards 1a participating in the same conference, including the electronic blackboard 1b.
(iv) Each electronic blackboard 1 reflects the received operation data B in the order of gid. Since the electronic blackboard 1a does not temporarily reflect the operation data B, the stroke B is displayed by the operation data B for the first time after this reflection.

FIG. 5 is a diagram illustrating a data flow when the base is only the own base.
(i) The user moves stroke C.
(ii) The electronic blackboard 1a generates operation data C (operation content, object to be operated) and temporarily reflects it on itself.
(iii) The electronic blackboard 1a transmits the operation data C to the communication management system 5.
(iv) The communication management system 5 assigns a gid to the received operation data C.
(v) The communication management system 5 transmits the operation data C (including the gid) to the electronic blackboard 1a.
(vi) The electronic blackboard 1a reflects the received operation data C in the order of gid.

FIG. 6 is a diagram illustrating a data flow when the operation of the electronic blackboard is rejected by the communication management system 5.
(i) The user moves stroke D.
(ii) The electronic blackboard 1a generates operation data D (operation content, object to be operated) and temporarily reflects it on itself.
(iii) The electronic blackboard 1a transmits the operation data D to the communication management system 5.
(iv) The communication management system 5 rejects the received operation data D. The case of being rejected is, for example, a case where the operation data D is damaged.
(v) The communication management system 5 transmits to the electronic blackboard 1a that the operation data D has been rejected.
(vi) Since the operation of the electronic blackboard 1a is rejected, the received operation data D is restored (in this case, the stroke D is displayed by returning to the place before the movement).

<List of operation data waiting to be sent>
Subsequently, a comparative technique to be compared with the technique of the present embodiment will be described. It should be noted that the comparative technique is not limited to the known technique or the prior art. In the comparative technique, as described in FIG. 2, when the user continuously operates the same stroke, the electronic blackboard 1 transfers the next operation data to the communication management system 5 after the main reflection of the previous operation data is completed. Send. As a result, when the user operates the same stroke a plurality of times in succession by adjusting the position of the stroke, the operation data waiting to be transmitted is retained, and it takes time for the final operation to be reflected.

FIG. 7 shows transmission waiting data when the user continuously moves the stroke A. In FIG. 7, seven operation data are waiting for transmission. When the user continuously operates the stroke A as shown in FIG. 7, the strokes A are processed in the following order. The list of operation data waiting to be transmitted in FIG. 7 has a queue structure, and when the electronic blackboard 1 receives a gid of one operation data, the electronic blackboard 1 transmits the next data. This is to operate the electronic blackboard 1 as intended by the user, as described with reference to FIG.
1. 1. The electronic blackboard temporarily reflects the No. 1 operation data and transmits it to the communication management system 5.
2. The communication management system 5 numbers the gid of the No. 1 operation data and transmits it to the electronic blackboard 1.
3. 3. The electronic blackboard 1 reflects the No. 1 operation data.
4. The electronic blackboard 1 temporarily reflects the operation data of No. 2 and transmits it to the communication management system 5.
5. The communication management system 5 numbers the gid of the operation data of No. 2 and transmits it to the electronic blackboard 1.
6. The electronic blackboard 1 reflects the operation data of No. 2.
・ ・ ・
N-2. The electronic blackboard 1 temporarily reflects the operation data of No. 7 and transmits it to the communication management system 5.
N-1. The communication management system 5 numbers the gid of the operation data of No. 7 and transmits it to the electronic blackboard 1.
N. The electronic blackboard 1 reflects the operation data of No. 7.

As described above, when the network band between the electronic blackboard 1 and the communication management system 5 is narrow, it takes a lot of time for the operation data of No. 7 to be reflected on the electronic blackboard.

<Processing of the communication system of the present embodiment>
Next, with reference to FIG. 8, a method in which the electronic blackboard 1 of the present embodiment processes operation data will be described. The electronic blackboard 1 of the present embodiment reflects only the final operation on the same object among the operation data before transmission to the communication management system 5. Since the user wants to determine the final position and share it with a remote location, there is little inconvenience even if the operation in the middle is omitted.

Therefore, when the user inputs a new operation, the electronic blackboard 1 replaces the transmission waiting operation data with the latest operation data when the operation data for the same object is registered in the transmission waiting operation data. .. As an example, a case where the user operates the stroke A again when the operation data to the stroke A is waiting to be transmitted will be described.

FIG. 8 shows operation data waiting to be transmitted, user operation data, and the like. It is assumed that there is operation data waiting for transmission to move the stroke A as shown in FIG. 8A. It is assumed that the user newly operates the stroke A and the operation data as shown in FIG. 8B is added to the list waiting to be transmitted.

In the comparative technique, the operation data shown in FIG. 8B is added to No. 2 of the data waiting to be transmitted. In that case, the No. 1 operation data is transmitted to the communication management system 5, and after the gid is returned from the communication management system 5, the electronic blackboard 1 transmits the No. 2 operation data to the communication management system 5. Therefore, it takes time to reflect the No. 2 operation data.

In this embodiment, this problem is solved as follows. Since the user wants to reflect the operation data of the last added user on the electronic blackboard 1, the communication management system 5 reflects the operation data of No. 1 waiting to be transmitted on the other electronic blackboard (gid is numbered). ) No need.

Therefore, the electronic blackboard 1 replaces the contents of the No. 1 operation data waiting to be transmitted with the latest operation data added by the user. FIG. 8C shows operation data waiting for transmission in which the order has been changed. Since the No. 1 operation data waiting to be transmitted is not transmitted to the communication management system 5, the electronic blackboard 1 can reflect the user's operation earlier. It does not occur when there are a plurality of operation data waiting to be transmitted, but if there are, the electronic blackboard may register only the latest operation data in the list waiting to be transmitted, and delete the others.

The electronic blackboard 1 may replace the operation data waiting for gid by transmitting the operation data to the communication management system with the latest operation data. This is because the operation data is transmitted from the communication management system together with the gid to the electronic blackboard 1, so that the electronic blackboard 1 only needs to be reflected.

[Outline of communication system]
<Communication route>
First, with reference to FIG. 9, a communication system 100 for conducting a video conference while drawing between a plurality of electronic blackboards 1b and 1b will be described. FIG. 9 is a schematic view showing a communication route of the present embodiment. In addition, it may be called "video conference" instead of "video conference". Further, although the video conference will be described here as an example, it may be a meeting or a simple conversation.

The communication system 100 is constructed by a plurality of electronic blackboards 1a and 1b, a relay device 3, a communication management system 5, and an image storage device 7. The electronic blackboards 1a and 1b mutually communicate video data and sound data for a call, and content data such as image data and stroke data for sharing. The stroke data is data necessary for reproducing (reproducing) the stroke image, and includes coordinate data, line width data, line color data, vector data, and the like. The electronic blackboards 1a and 1b can make a remote video call by reproducing the video and sound of the other party by transmitting and receiving video data and sound data for a call.

By transmitting and receiving image data of the material image for sharing on the electronic blackboards 1a and 1b, participants using the communication system 100 can share the same material image. The material image is an image displayed on the display of the electronic blackboard 1, and is an image of a conference material, a background image displayed on the display, a capture screen when the display screen is captured, and the like. Further, the electronic blackboards 1a and 1b can share the same stroke image by the participants using the communication system 100 by transmitting and receiving the stroke data of the stroke image for sharing. The stroke image is an image showing a line or the like drawn by a user with a handwritten stroke with an electronic pen or the like. The stroke image is displayed by stroke data indicating points that specify coordinates on the display.

The communication system 100 is not limited to the two electronic blackboards 1a and 1b, and may be constructed by three or more electronic blackboards. Hereinafter, when the generic names of the electronic blackboards 1b and 1b are shown, they are referred to as "electronic blackboard 1".

In FIG. 9, an electronic blackboard equipped with a video conference function is shown as an example of the electronic blackboards 1a and 1b. The image of the video data may be a moving image or a still image.

Further, the electronic blackboard as the request source for requesting the start of the video conference is represented as the "start terminal", and the electronic blackboard as the destination (relay destination) as the request destination is represented as the "destination terminal". In FIG. 9, the electronic blackboard 1a is represented as a start terminal, and the electronic blackboard 1b is represented as a destination terminal. However, when the electronic blackboard 1b requests the start of the video conference, the electronic blackboard 1b serves as the start terminal and the electronic blackboard 1a serves as the destination terminal. The electronic blackboards 1b and 1b are used not only for communication between a plurality of business establishments and between different rooms in the same business establishment, but also for communication within the same room, outdoors and indoors, or outdoors and outdoors. It may be used in the communication of.

The relay device 3 is composed of a computer, and performs a process of relaying content data for a call between a plurality of electronic blackboards 1b and 1b.

The communication management system 5 is composed of a computer and centrally manages login authentication from the electronic blackboards 1b and 1b, management of the communication status of the electronic blackboards 1b and 1b, management of the destination list, communication status of the relay device 3, and the like. do. Further, the communication management system 5 relays stroke data for sharing between a plurality of electronic blackboards 1b and 1b.

The image storage device 7 is configured by a computer, stores image data of a material image for sharing uploaded from the electronic blackboard 1a, and downloads the image data to the electronic blackboard 1b. And vice versa. That is, the image storage device 7 saves the image data uploaded from the electronic blackboard 1b and downloads it to the electronic blackboard 1a.

The relay device 3, the communication management system 5, and the image storage device 7 may be constructed by a single computer, or may be constructed by a plurality of computers arbitrarily assigned by dividing each part (function or means). You may be.

Further, in the communication system 100, a management information session sei for transmitting and receiving various management information is established between the electronic blackboards 1a and 1b via the communication management system 5. Further, for transmitting and receiving four types of data, high-resolution image data, medium-resolution image data, low-resolution image data, and sound data, with and from the electronic blackboards 1a and 1b via the relay device 3. Four sessions are established. In FIG. 9, these four sessions are collectively shown as a session sed for image / sound data. The image / sound data session sed does not necessarily have to be four sessions, and the number of sessions may be less or more than the number of four sessions. Further, a communication session may be directly established between the start terminal and the destination terminal without going through the relay device 3.

Further, in the communication system 100, stroke data can be transmitted and received between the electronic blackboards 1a and 1b by using the management information session sei.

Here, the video resolution of the video data handled in the present embodiment will be described. The low-resolution video data is, for example, composed of 160 pixels in the horizontal direction and 120 pixels in the vertical direction, and serves as a base image. The medium-resolution video data consists of 320 pixels in the horizontal direction and 240 pixels in the vertical direction. The high-resolution video data includes, for example, 640 pixels in the horizontal direction and 480 pixels in the vertical direction. Of these, when passing through a narrow band path, low-quality video data consisting of only low-resolution video data as a base image is relayed. When the band is relatively wide, low-resolution video data as a base image and medium-quality video data including medium-resolution video data are relayed. When the band is very wide, high-quality video data including low-resolution video data as a base image, medium-resolution video data, and high-resolution video data is relayed. Since the amount of sound data is smaller than that of video data, it is relayed even in a narrow band path.

<Image of using electronic blackboard>
FIG. 10 is an image diagram of using an electronic blackboard. As shown in FIG. 10, the electronic blackboard 1 includes a leg 151 having a plurality of casters on the lower side, a support 152 provided on the upper side of the leg 151, and an upper part of the support 152. It is composed of a main body 153 of the electronic blackboard 1 provided on the side and a display 180 provided on the front surface of the main body 153. The main body 153 has a built-in CPU 101 and the like, which will be described later. Then, the user can input (draw) a stroke image such as characters on the display 180 by using the electronic pen 190.

[Hardware configuration]
Next, the hardware configuration of this embodiment will be described.

<Hardware configuration of electronic blackboard>
FIG. 11 is a hardware configuration diagram of the electronic blackboard. As shown in FIG. 11, the electronic blackboard 1 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, an SSD (Solid State Drive) 104, and a network I / It includes an F105 and an external device connection I / F (Interface) 106.

Of these, the CPU 101 controls the operation of the entire electronic blackboard 1. The ROM 102 stores a program used to drive the CPU 101 such as the CPU 101 and the IPL (Initial Program Loader). The RAM 103 is used as a work area of the CPU 101. The SSD 104 stores various data such as a program for an electronic blackboard. The network I / F 105 controls communication with the communication network N. The external device connection I / F 106 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory 130, an external device (microphone 140, speaker 150, camera 160).

Further, the electronic blackboard 1 includes a capture device 111, a GPU 112, a display controller 113, a contact sensor 114, a sensor controller 115, an electronic pen controller 116, a short-range communication circuit 119, an antenna 119a of the short-range communication circuit 119, a power switch 122, and the like. The selection switches 123 are provided.

Of these, the capture device 111 causes the display of the external PC (Personal Computer) 170 to display the video information as a still image or a moving image. The GPU (Graphics Processing Unit) 112 is a semiconductor chip that specializes in graphics. The display controller 113 controls and manages the screen display in order to output the output image from the GPU 112 to the display 180 or the like. The contact sensor 114 detects that the electronic pen 190, the user's hand H, or the like touches the display 180. The sensor controller 115 controls the processing of the contact sensor 114. The contact sensor 114 inputs and detects the coordinates by the infrared blocking method. In this method of inputting coordinates and detecting coordinates, two light receiving and emitting devices installed at both upper ends of the display 180 radiate a plurality of infrared rays in parallel with the display 180 and are provided around the display 180. This is a method of receiving light that is reflected by a reflecting member and returns on the same optical path as the light path emitted by the light receiving element. The contact sensor 114 outputs an infrared ID emitted by two light emitting / receiving devices blocked by the object to the sensor controller 115, and the sensor controller 115 specifies a coordinate position which is a contact position of the object. The electronic pen controller 116 communicates with the electronic pen 190 to determine whether or not there is a touch of the pen tip or a touch of the pen tail on the display 180. The short-range communication circuit 119 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 122 is a switch for switching the power ON / OFF of the electronic blackboard 1. The selection switches 123 are, for example, a group of switches for adjusting the brightness and color tone of the display 180.

Further, the electronic blackboard 1 is provided with a bus line 110. The bus line 110 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 101 shown in FIG.

The contact sensor 114 is not limited to the infrared blocking method, but is a capacitive touch panel that specifies a contact position by detecting a change in capacitance, and a contact position is specified by a voltage change between two opposing resistance films. Various detection means such as a resistance film type touch panel and an electromagnetic induction type touch panel that detects the electromagnetic induction generated when a contact object comes into contact with the display unit and specifies the contact position may be used. Further, the electronic pen controller 116 may determine whether or not there is a touch not only on the pen tip and pen tail of the electronic pen 190, but also on a portion gripped by the user of the electronic pen 190 and other electronic pen portions.

<Hardware configuration of communication management system, relay device, and image storage device>
FIG. 12 is a hardware configuration diagram of the communication management system 5 according to the present embodiment. Computers as an example of the communication management system 5 include CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I /. It includes an F509, a controller 511, a pointing device 512, a DVD-RW (Digital Versatile Disk ReWritable) drive 514, a media I / F 516, and a bus line 510.

Of these, the CPU 501 controls the operation of the entire communication management system 5. The ROM 502 stores a program used to drive the CPU 501 such as an IPL. The RAM 503 is used as a work area of the CPU 501. The HD504 stores various data such as a communication management program. The HDD controller 505 controls reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. The network I / F 509 is an interface for data communication using a communication network N such as the Internet. The keyboard 511 is a kind of input means including a plurality of keys for inputting characters, numerical values, various instructions, and the like. The pointing device 512 is a kind of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The DVD-RW drive 514 controls reading or writing (storage) of data to the DVD-RW. It should be noted that data reading or writing (storage) to a disc such as BD-RE (Blu-ray (registered trademark) Disc Rewritable) may be controlled instead of the DVD-RW drive. The media I / F 516 controls reading or writing (storage) of data to a recording medium 515 such as a flash memory.

Further, the bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG.

Since the hardware configuration of the relay device 3 and the image storage device 7 shown in FIG. 9 is the same as the hardware configuration of the communication management system 5, the description thereof will be omitted. However, in the case of the relay device 3, the relay program is stored in the HD504. Further, in the case of the image storage device 7, the image storage program is stored in the HD504.

[Overall configuration of communication system]
Subsequently, the overall configuration of the communication system 100 will be described with reference to FIG. FIG. 13 is an overall configuration diagram of the communication system 100 according to the embodiment of the present invention.

In FIG. 13, the electronic blackboard 1a is installed at the base A, and the electronic blackboard 1b is installed at the base B. For example, base A is the Tokyo office in Japan, and base B is the Osaka office in Japan. At the base A, the user A1 uses the electronic blackboard 1a, and at the base B, the users B1 and B2 use the electronic blackboard 1b.

Further, the electronic blackboards 1a and 1b, the relay device 3, the communication management system 5, and the image storage device 7 can perform mutual communication of data via a communication network N such as the Internet. The communication network N may include a wireless communication portion.

In FIG. 13, the electronic blackboards 1a and 1b are both electronic blackboards capable of video communication.

[Functional configuration of communication system]
Next, the functional configuration of the present embodiment will be described with reference to FIGS. 14 to 20. FIG. 14 is a functional block diagram of the communication system 100 according to the present embodiment.

<Functional configuration of electronic blackboard 1a>
As shown in FIG. 14, the electronic blackboard 1a includes a transmission / reception unit 11a, a reception unit 12a, a video / sound processing unit 13a, a display control unit 14a, a judgment unit 15a, an image processing unit 17a, and a short-range communication unit 18a. It has a transmission information management unit 21a and a storage / reading processing unit 19a. Each of these parts is a function realized by operating any of the components shown in FIG. 11 by an instruction from the CPU 101 according to a program developed on the RAM 103 from the SSD 104, or a means for the function. be. Further, the electronic blackboard 1a has a storage unit 1000a constructed by the RAM 103 shown in FIG. 11 and the flash memory 104 shown in FIG.

(List table of operation data)
FIG. 15 is a conceptual diagram showing a list table of operation data. The storage unit 1000a is constructed with an operation data storage unit 1001 configured by a list table of operation data as shown in FIG. In this operation data list table, No, the target object, the operation content, the position after the operation, and the gid of the operation are associated and managed. No indicates the transmission order, and the operation data is transmitted to the communication management system 5 in this order. The target object is the identification information of the object to be operated. The operation content specifically indicates what kind of operation was performed. The position after the operation indicates the position of the object. The operation gid is the global ID of the operation notified from the communication management system 5. Global means that it is common to the electronic blackboard 1 of each base. Except for the operation gid, it is registered when the reception unit 12a accepts the operation on the object. The operation gid is registered when the gid is transmitted from the communication management system 5. Therefore, the temporary reflection state is from the registration of each item except the operation gid to the registration of the operation gid. When the operation gid is registered, it will be in the state of this reflection.

Strictly speaking, each operation data registered in the operation data list table is deleted when the gid is notified (execution of a predetermined process). Since the reflected operation data is posted to another operation history table, the operation content is maintained. However, in FIG. 15, for the sake of explanation, the state in which the gid of the operation is registered is displayed. Therefore, the electronic blackboard 1 may determine whether or not the same target object as the latest operation data is registered in the operation data storage unit 1001.

(Each functional configuration of the electronic blackboard 1a)
Next, each component of the electronic blackboard 1a will be described. The transmission / reception unit 11a transmits / receives various data (or information) to / from another terminal, device, or system via the communication network N. In addition, the transmission / reception unit 11a also serves as a start unit, and performs a process of starting communication with another communication terminal such as the electronic blackboard 1b. The reception unit 12a receives various inputs from the user using the electronic pen 190 or the like.

The video / sound processing unit 13a performs the main processing of the video conference function. For example, the video / sound processing unit 13a performs digital processing such as encoding of video data and sound data together with the output signal of the microphone 140 and the output signal of the camera 160. Further, the video / sound processing unit 13a generates a video signal or a sound signal based on the video data and the sound data received by the transmission / reception unit 11a. The video / sound processing unit 13a performs a process of combining video data having different resolutions.

The display control unit 14a controls to output a video signal (image signal) or the like to the display 180. The determination unit 15a makes various determinations.

The image processing unit 17a performs the main processing of the electronic blackboard function. For example, the image processing unit 17a creates a stroke image and stroke data based on the stroke of the electronic pen 190 or the like received by the reception unit 12a, or creates a stroke image based on the stroke data received by the transmission / reception unit 11a. To do. Further, the image processing unit 17a generates an image signal based on the image data of the material image received by the transmission / reception unit 11a.

The transmission information management unit 21a manages a list table of operation data to be transmitted to the communication management system 5 in the storage unit 1000a. In addition, it is determined whether or not the operation data registered in the operation data list table and the operation data newly operated by the user are for the same object. In the same case, the operation data in the operation data list table is replaced with the operation data newly operated by the user (latest operation data).

The short-range communication unit 18a acquires and provides data by short-range wireless communication with each terminal having the short-range communication unit.

The storage / reading processing unit 19 performs a process of storing various data in a recording medium 1010 such as a storage unit 1000 or a USB memory 130, and reading various data stored in the storage unit 1000 or the recording medium 1010.

Further, the storage unit 1000 overwrites and stores the video data and the sound data received when communicating with another terminal each time it is received. Of these, the image is displayed on the display 180 by the video data before being overwritten, and the sound is output from the speaker 150 by the sound data before being overwritten.

<Functional configuration of electronic blackboard 1b>
As shown in FIG. 14, the electronic blackboard 1b includes a transmission / reception unit 11b, a reception unit 12b, a video / sound processing unit 13b, a display control unit 14b, a judgment unit 15b, an image processing unit 17b, and a short-range communication unit 18b. It has a transmission information management unit 21b and a storage / reading processing unit 19b. The transmission / reception unit 11b, the reception unit 12b, the video / sound processing unit 13b, the display control unit 14b, the judgment unit 15b, the image processing unit 17b, the short-range communication unit 18b, and the storage / reading processing unit 19b are the transmission / reception units 11a, respectively. It has the same functions as the reception unit 12a, video / sound processing unit 13a, display control unit 14a, judgment unit 15a, image processing unit 17a, short-range communication unit 18a, transmission information management unit 21a, and storage / reading processing unit 19a. Therefore, these explanations will be omitted.

<Functional configuration of relay device>
As shown in FIG. 14, the relay device 3 has a transmission / reception unit 31 that also serves as a transfer unit, a determination unit 35, and a storage / reading processing unit 39. Each of these parts has a function or a functioning means realized by operating any of the components shown in FIG. 12 by an instruction from the CPU 501 according to a relay program developed on the RAM 503 from the HD 504. Is. Further, the relay device 3 has a storage unit 3000 constructed by the RAM 503 and the HD 504 shown in FIG.

(Each function configuration of the relay device)
Next, each functional configuration of the relay device 3 will be described in detail. The transmission / reception unit 31 of the relay device 3 shown in FIG. 14 transmits / receives various data (or information) to / from another terminal, device, or system via the communication network N. The transmission / reception unit 31 also serves as a transfer unit, and transfers video data and sound data transmitted from a predetermined terminal to another terminal. The determination unit 35 makes various determinations such as determination of a data delay state and the like.

The storage / reading processing unit 39 performs a process of storing various data in the storage unit 3000 and reading various data stored in the storage unit 3000.

<Functional configuration of communication management system>
As shown in FIG. 14, the communication management system 5 includes a transmission / reception unit 51, an authentication unit 52, a generation unit 53, a selection unit 54, a judgment unit 55, an operation data management unit 56, and a storage / reading processing unit 59. Have. Each of these parts functions or functions as any of the components shown in FIG. 14 operates by an instruction from the CPU 501 according to a communication management program developed on the RAM 503 from the HD 504. It is a means. Further, the communication management system 5 has a storage unit 5000 constructed by the HD 504 shown in FIG.

(Authentication management table)
FIG. 16 is a conceptual diagram showing an authentication management table. In the storage unit 5000, an authentication management DB 5001 configured by an authentication management table as shown in FIG. 16 is constructed. In this authentication management table, each password is associated and managed for each terminal ID of all the electronic blackboards 1 managed by the communication management system 5. For example, in the authentication management table shown in FIG. 16, it is shown that the terminal ID of the electronic blackboard 1a is "01aa" and the password is "aaaa". The password is an example of authentication information, and the authentication information also includes an access token. The terminal ID is identification information for identifying a communication terminal, but also has a function of identifying a user or the like. It may also be called a communication ID. An email address may be used instead.

(Terminal management table)
FIG. 17 is a conceptual diagram showing a terminal management table. In the storage unit 5000, a terminal management DB 5002 configured by a terminal management table as shown in FIG. 17 is constructed. In this terminal management table, for each terminal ID for identifying each electronic blackboard 1, the destination name when each electronic blackboard 1 is set as the destination, the operating state of each electronic blackboard 1, and the login request information described later are the communication management system. The reception date and time received in 5 and the IP address of each electronic blackboard 1 are associated and managed. For example, in the terminal management table shown in FIG. 17, the electronic blackboard 1a having the terminal ID "01aa" has the terminal name "Japan Tokyo Office AA terminal" and the operating state is "ON line (communication possible)". Then, the date and time when the login request information was received by the communication management system 5 is "13:40 on April 10, 2015", and the IP address of this terminal 1aa is "1.2.1.3". It is shown. The terminal ID, the destination name, and the IP address of the terminal are stored when each electronic blackboard 1 is pre-registered to receive the service provided by the communication management system 5.

(Destination list management table)
FIG. 18 is a conceptual diagram showing a destination list management table. In the storage unit 5000, a destination list management DB 5003 configured by a destination list management table as shown in FIG. 18 is constructed. In this destination list management table, all the terminal IDs of the destination terminals registered as candidates for the electronic blackboard 1 (destination terminal) are associated with the terminal ID of the electronic blackboard 1 (starting terminal) requesting the start of communication. Is managed. For example, in the destination list management table shown in FIG. 18, a candidate for a destination terminal that can request the start of communication from a start terminal (electronic blackboard 1a) having a terminal ID of "01aa" has a terminal ID of "01aa". It is shown that it is an electronic blackboard 1b or the like of "01ba". The candidate of the destination terminal is updated by being added or deleted at the request of addition or deletion to the communication management system 5 from an arbitrary starting terminal.

The destination list is an example of destination information, and the destination information may not be in a list format and may include information related to the destination such as a terminal ID.

(Session management table)
FIG. 19 is a conceptual diagram showing a session management table. In the storage unit 5000, a session management DB 5004 configured by a session management table as shown in FIG. 19 is constructed. In this session management table, the relay device ID of the relay device 3 used, the electronic blackboard 1 ( The terminal ID of the start terminal), the terminal ID of the electronic whiteboard 1 (destination terminal), the delay time (ms) of reception when video data is received at the destination terminal, and the delay information indicating this delay time are addressed to the destination. The reception date and time sent from the terminal and received by the communication management system 5 is associated and managed. For example, in the session management table shown in FIG. 19, in the communication session executed by using the session ID "se01", the relay device (relay device ID "111a") is an electronic blackboard whose terminal ID is "01aa". And the electronic blackboard with the terminal ID "01db", the video data and sound data are relayed, and the video as of "13:41 on April 10, 2015" on the electronic whiteboard (destination terminal). The data delay time is shown to be 200 (ms).

(Relay device management table)
FIG. 20 is a conceptual diagram showing a relay device management table. In the storage unit 5000, a relay device management DB 5005 configured by a relay device management table as shown in FIG. 20 is constructed. In this relay device management table, for each of the plurality of relay device 3 relay device IDs, the operating status of each relay device 3, the reception date and time when the status information indicating the operating status is received by the communication management system 5, and the IP of the relay device 3 The address and the maximum data transmission rate (Mbps) in the relay device 3 are associated and managed. For example, in the relay device management table shown in FIG. 20, the relay device 3 having the relay device ID "111a" has an operating state of "ON line" and the date and time when the status information is received by the communication management system 5. At "13:30 on April 10, 2014", it is shown that the IP address of the relay device 3 is "1.2.1.2" and the maximum data transmission speed of the relay device 3 is 100 Mbps. Has been done.

(Operation data management table)
FIG. 21 is a conceptual diagram showing an operation data management table. In the storage unit 5000, an operation data management DB 5006 configured by an operation data management table as shown in FIG. 21 is constructed. In this operation data management table, the operation gid, the operation content, the position after the operation, and the target object (id) are associated and managed. The contents of each item may be the same as the list table of the operation data shown in FIG. On the other hand, the operation data of each electronic blackboard is registered in the operation data management table. Therefore, the target object (id) may be duplicated in the operation data management table. By transmitting this operation data to each electronic blackboard 1, each electronic blackboard 1 can share the operation for the same object.

(Each function configuration of communication management system)
The transmission / reception unit 51 of the communication management system 5 shown in FIG. 14 transmits / receives various data (or information) to / from another terminal, device, or system via the communication network N.

The authentication unit 52 uses the terminal ID and password included in the login request received via the transmission / reception unit 51 as a search key, searches the authentication management DB 5001 of the storage unit 5000, and uses the same set of terminals in the authentication management DB 5001. Authentication is performed by determining whether the ID and password are managed. The generation unit 53 generates a session ID for identifying the communication session based on the communication start request (see S62) from the electronic blackboard 1. The selection unit 54 performs a process of finally selecting one relay device 3 from the plurality of relay devices 3. The determination unit 55 makes various determinations. The operation data management unit 56 numbers the gid of the operation data transmitted from each electronic blackboard 1, and transmits the operation data to all the electronic blackboards 1 participating in the same conference. The storage / reading processing unit 59 performs a process of storing various data in the storage unit 5000 and reading various data stored in the storage unit 5000.

<Functional configuration of image storage device>
As shown in FIG. 14, the image storage device 7 includes a transmission / reception unit 71 and a storage / reading processing unit 79. Each of these parts functions or functions as any of the components shown in FIG. 14 operates by a command from the CPU 501 according to an image storage program developed on the RAM 503 from the HD 504. It is a means. Further, the image storage device 7 has a storage unit 7000 constructed by the RAM 503 and the HD 504 shown in FIG.

(Each function configuration of the image storage device)
Next, each functional configuration of the image storage device 7 will be described in detail. The transmission / reception unit 71 of the image storage device 7 shown in FIG. 14 transmits / receives various data (or information) to / from another terminal, device, or system via the communication network N. The storage / reading processing unit 79 performs a process of storing various data in the storage unit 7000 and reading various data stored in the storage unit 7000.

[Processing or operation of the embodiment]
Next, processing or operation in the communication system 100 according to the present embodiment will be described with reference to FIGS. 22 to 27.

<Processing in the preparatory stage of remote communication>
First, the preparation process for communication performed by the electronic blackboard 1a as the login request terminal will be described with reference to FIGS. 22 and 23. Note that FIG. 22 is a sequence diagram showing a process in the preparatory stage for starting remote communication between electronic blackboards. FIG. 23 is a screen example of the destination list displayed on the electronic blackboard 1a. Since the process of logging in and sharing by the electronic blackboard 1b is the same as the process of the electronic blackboard 1a, the description thereof will be omitted.

First, when the power switch 122 is turned on on the electronic blackboard 1a, the reception unit 12a receives the power ON (step S22).

Next, the transmission / reception unit 11a transmits login request information indicating a login authentication request to the communication management system 5 via the communication network N (step S23). The login request information includes the terminal ID and password of the electronic blackboard 1a.

Next, the storage / reading processing unit 59 of the communication management system 5 uses the terminal ID and password included in the login request information received via the transmission / reception unit 51 as search keys to use the authentication management table (see FIG. 16). The terminal is authenticated by searching and determining whether the same terminal ID and the same password are managed (step S24). Here, it will be described continuously assuming that the same terminal ID and the same password are managed by the storage / reading processing unit 59.

If it is determined by the storage / reading processing unit 59 that the same terminal ID and the same password are managed and the login request is from a terminal having a legitimate usage authority, the storage / reading processing unit 59 In the terminal management table (see FIG. 17), 59 changes the field part of the operating state of the record of the terminal ID received in step S23 to "ON line (communicable)" and sets it to the field part of the reception date and time. The reception date and time when the login request information is received in step S23 is stored (step S25). As a result, in the terminal management table, the terminal ID "01aa", the operating status "online (communication possible)", the reception date and time "2015.4.10.13:40", and the IP address "1.2.1.3" are displayed. Will be associated and managed. The IP address of the terminal may not be registered in advance, but may be transmitted from the electronic blackboard 1a in step S23.

Next, the storage / reading processing unit 59 adds and manages a new record including the terminal ID of the electronic blackboard 1a received in step S23 in the session management table (see FIG. 19) (step S26). Then, the transmission / reception unit 51 of the communication management system 5 transmits the authentication result information indicating the authentication result obtained by the process of step 24 to the electronic blackboard 1a that has requested the login via the communication network N. (Step S27).

When the transmission / reception unit 11a of the login request terminal (electronic blackboard 1a) receives the authentication result information indicating the result of being determined to be a terminal having legitimate usage authority, the transmission / reception unit 11a communicates via the communication network N. The destination list request information indicating that the destination list is requested is transmitted to the management system 5 (step S28). As a result, the transmission / reception unit 51 of the communication management system 5 receives the destination list request information.

Next, the storage / reading processing unit 59 searches the destination list management table (see FIG. 18) using the terminal ID "01aa" of the login request terminal (electronic blackboard 1a) as a search key, and the login request terminal (electronic whiteboard 1a). ), And the destination name corresponding to this terminal ID is read from the terminal management table (see FIG. 17) (step S29). Here, the terminal IDs of the destination candidates corresponding to the terminal ID "01aa" of the login request terminal (electronic blackboard 1a) and the destination names corresponding to these are extracted.

Next, the transmission / reception unit 51 of the communication management system 5 reads the data of the destination list frame and the data of the icon indicating the operating state from the storage unit 5000 via the storage / reading processing unit 59 (step S30), and this destination list. "Destination list information (destination list frame, icon, terminal ID, destination name)" including the frame and icon and the terminal ID and destination name read by the storage / reading processing unit 59 can be input to the login request terminal (electronic blackboard 1a). ) (Step S31). As a result, in the login request terminal (electronic blackboard 1a), the transmission / reception unit 11a receives the destination list information, and the storage / reading processing unit 19a stores the destination list information in the storage unit 1000a (step S32). As described above, in the present embodiment, the communication management system 5 centrally manages the destination list information of all the terminals, instead of managing the destination list information in each terminal. Therefore, the communication system 100 will include a new electronic blackboard, a new model terminal will be included in place of the terminal already included, and the appearance of the destination list frame will be changed. Even in such a case, since the communication management system 5 handles all of them collectively, it is possible to save the trouble of changing the destination list information on each terminal side.

Further, the storage / reading processing unit 59 of the communication management system 5 searches the terminal management table (see FIG. 17) using the terminal ID of the read destination candidate as a search key, and operates correspondingly for each terminal ID. By reading the state, each operating state of the electronic blackboard as a destination candidate is acquired (step S33).

Next, the transmission / reception unit 51 transmits "terminal status information" including the terminal ID as the search key used in step S33 and the operating status of each corresponding destination terminal via the communication network N. It is transmitted to the login request terminal (electronic blackboard 1a) (step S34).

Next, the storage / reading processing unit 19a of the login sharing terminal (electronic blackboard 1a) sequentially stores the state information of the terminal received from the communication management system 5 in the storage unit 1000a (step S35). Therefore, the login request terminal (electronic blackboard 1a) can communicate with the login request terminal (electronic blackboard 1a) by receiving the status information of each of the above electronic blackboards, such as the electronic blackboard 1b which is a destination candidate at the present time. Each operating status can be acquired.

Next, the display control unit 14a of the login request terminal (electronic blackboard 1a) reflects the state of the terminal as a destination candidate based on the destination list information stored in the storage unit 1000a and the state information of the terminal. Along with creating the destination list, the display control unit 14a displays the destination list screen 1100 shown in FIG. 23 on the display 180 of the electronic blackboard 1a (step S36). On the destination list screen 1100, an icon indicating an operating state, a terminal ID, and a destination name are displayed for each destination candidate. In FIG. 23, icons showing the operating state of each terminal are displayed as “offline” and “online (communicable)” from the top.

On the other hand, the storage / reading processing unit 59 of the communication management system 5 searches the destination list management table (see FIG. 18) based on the terminal ID "01aa" of the login request terminal (electronic blackboard 1a) to log in the login request terminal. The terminal ID "01aa" of the (electronic blackboard 1a) is registered as a destination candidate, and the terminal IDs of other terminals are extracted (step S37). In the destination list management table shown in FIG. 18, the terminal IDs of the other terminals to be read are "01ab", "01ba", "01da" and the like.

Next, the storage / reading processing unit 59 of the communication management system 5 searches the terminal management table (see FIG. 17) based on the terminal ID "01aa" of the login request terminal (electronic blackboard 1a), and the login request terminal (electronic). Acquire the operating state of the blackboard 1a) (step S38).

Then, the transmission / reception unit 51 is acquired in step S38 by the terminal whose operating state is "online" in the terminal management table (see FIG. 17) among the terminals related to the terminal ID extracted in step S37. The terminal ID "01aa" of the login request terminal (electronic blackboard 1a) and the "terminal status information" including the operating status "online" are transmitted (step S39). When the transmission / reception unit 51 transmits the state information of the terminal to the electronic blackboard 1b, the IP address of the electronic blackboard managed in the terminal management table (see FIG. 17) is referred to based on each terminal ID. As a result, the terminal ID "01aa" of the login request terminal (electronic blackboard 1a) and the operating state "online" are sent to each of the other destination terminals capable of communicating with the login request terminal (electronic blackboard 1a) as a destination candidate. Can be conveyed. Therefore, the status of the destination candidate can be displayed even on the destination candidate (electronic blackboard 1b or the like) (step S40).

<Start processing of remote communication>
Subsequently, with reference to FIG. 24, a process in which the electronic blackboard 1a starts remote communication with the electronic blackboard 1b will be described. FIG. 24 is a sequence diagram showing a process of starting remote communication.

First, when the user of the requesting terminal (electronic blackboard 1a) presses the destination candidate (terminal ID "01ba") shown in FIG. 17 and selects the electronic blackboard 1b, the reception unit shown in FIG. 14 12a receives a request to start a call with the destination terminal (electronic blackboard 1b) (step S61). Then, the transmission / reception unit 11a of the request source terminal (electronic blackboard 1a) transmits start request information indicating that the call is to be started to the communication management system 5 (step S62). The start request information includes the terminal ID "01aa" of the request source terminal (electronic blackboard 1a) and the terminal ID "01ba" of the destination terminal (electronic blackboard 1b). As a result, the transmission / reception unit 51 of the communication management system 5 receives the start request information and also receives the IP address of the request source terminal (electronic blackboard 1a) which is the transmission source.

Then, the storage / reading processing unit 59 bases the terminal management table based on the terminal ID "01aa" of the request source terminal (electronic whiteboard 1a) and the terminal ID "01ba" of the destination terminal (electronic blackboard 1b) included in the start request information. In (see FIG. 17), the field portion of the operating state of the record including the terminal ID “01aa” and the terminal ID “01ba” is changed to “in a call” (step S63). In this state, the requesting source terminal (electronic blackboard 1a) and the destination terminal (electronic blackboard 1b) have not started the video conference, but are in a busy state, and the third electronic blackboard terminal is the requesting terminal (electronic blackboard 1b). When trying to make a call with the electronic blackboard 1a) or the destination terminal (electronic blackboard 1b), a notification sound or display indicating a so-called busy state is output.

Next, a process of executing a session for selecting the relay device 3 to be actually used will be described. First, the generation unit 53 of the communication management system 5 generates a session ID used for executing a session for selecting the relay device 3 (step S64). Here, the case where the session ID “se1” is generated will be described.

Then, the storage / reading processing unit 59 displays the session ID “se1” generated in step S64, the terminal ID “01aa” of the requesting terminal (electronic blackboard 1a), and the destination in the session management table (see FIG. 19). The terminal ID "01ba" of the terminal (electronic blackboard 1b) is associated, stored and managed (step S65).

Next, the selection unit 54 of the communication management system 5 shown in FIG. 14 is a requesting terminal (electronic blackboard 1a) based on the relay device management table (see FIG. 20) and the terminal management table (see FIG. 17). And the relay device 3 for relaying the call between the two bases of the destination terminal (electronic blackboard 1b) is selected (step S66). Specifically, among the relay device IDs related to the relay devices whose operating state is "ON line" in the relay device management table (see FIG. 20), the requesting terminal (electronic blackboard 1a) in the terminal management table (see FIG. 17). The relay device ID related to the relay device 3 having an IP address close to the IP address of the above is selected. Here, the case where the relay device 3 (relay device ID “111a”) is selected will be continuously described below.

When the process of selecting the relay device in step S66 is completed, the transmission / reception unit 51 of the communication management system 5 transmits the relay device selection information to the requesting terminal (electronic blackboard 1a) (step S67-1). .. The relay device selection information includes the IP address of the relay device 3 selected in step S66 and the session ID “se1” generated in step S64. As a result, the requesting terminal (electronic blackboard 1a) can acquire the IP address of the communication management system 5 which is the source of the relay device selection information.

Further, the transmission / reception unit 51 of the communication management system 5 transmits the relay device selection information to the destination terminal (electronic blackboard 1b) (step S67-2). The relay device selection information includes the IP address of the relay device 3 selected in step S66, the terminal ID "01aa" of the requesting terminal (electronic blackboard 1a), and the session ID "se1" generated in step S64. It is included. As a result, the destination terminal (electronic blackboard 1b) can acquire the IP address of the communication management system 5 which is the source of the relay device selection information in the execution of the session with the session ID “se1”.

Next, in response to the process of step S67-1, the transmission / reception unit 11 of the requesting terminal (electronic blackboard 1a) receives the relay device selection information from the communication management system 5 by the process of step S67-1. Is transmitted (step S68-1) indicating that the reception completion information has been completed. The reception completion information includes the session ID transmitted / received in the process of step S67-1. As a result, the communication management system 5 acquires that the transmission of the relay device selection information executed by the specific session ID "se1" is completed.

Further, in response to the process of step S67-2, the destination terminal (electronic blackboard 1b) similarly indicates to the communication management system 5 that the reception of the relay device selection information has been completed by the process of step S67-2. The reception completion information is transmitted (step S68-2). Also in this case, the communication management system 5 acquires that the transmission of the relay device selection information executed by the specific session ID “se1” is completed.

As described above, the electronic blackboards 1a and 1b can perform a video conference by transmitting and receiving video data and sound data via the relay device 3 selected in step S66.

<Communication processing of document images and stroke images>
Subsequently, the communication processing of the material image displayed on the electronic blackboard 1 and the input stroke image will be described with reference to FIGS. 25 to 27. 25 and 26 are sequence diagrams showing a process of sharing the image data of the document image and the stroke data of the stroke image. Here, in the case where the image data of the material image and the stroke data of the stroke image are shared by the electronic whiteboards 1a and 1b, the material image displayed on the electronic whiteboard 1a and the input stroke image are also displayed on the electronic whiteboard 1b. The case where it is done will be described. In FIG. 27, (a) is a screen example of the electronic blackboard 1a, (b) is a screen example of the electronic blackboard 1b, (c) is a screen example of the electronic blackboard 1a, and (d) is a screen example of the electronic blackboard 1b. ..

First, on the electronic blackboard 1a of the base A, the screen shown in FIG. 27A is displayed on the display 180 by the display control unit 14a. Here, the "share" button b1 that is pressed when sharing the material image d1, the image v2 of the base B, and the material image d1 is displayed. In this state, when the user a of the electronic blackboard 1a presses the "share" button b1, the reception unit 12a accepts the material image sharing process from the user a (step S101). Next, the transmission / reception unit 11a transmits request information indicating that a URL (Uniform Resource Locator) indicating a storage position of the image data of the material image is requested to the communication management system 5 (step S102). As a result, the transmission / reception unit 51 of the communication management system 5 receives the request information. The URL is an example of the storage location information, and the storage location information also includes a URI (Uniform Resource Identifier).

Next, the generation unit 53 of the communication management system 5 generates a URL indicating the storage position of the image data of the material image (S103). Then, the transmission / reception unit 51 transmits the URL of the image data generated by the generation unit 53 to the electronic blackboard 1a. As a result, the transmission / reception unit 11a of the electronic blackboard 1a receives the URL of the image data.

Next, the transmission / reception unit 11a of the electronic blackboard 1a transmits a notification of the start of uploading the image data of the material image to the communication management system 5 (step S105). As a result, the transmission / reception unit 51 of the communication management system 5 receives the start notification.

Next, the transmission / reception unit 51 forwards the start notification to the electronic blackboard 1b of the other party during the remote conference (step S106). As a result, the transmission / reception unit 11b of the electronic blackboard 1b receives the start notification. The display control unit 14d of the electronic blackboard 1b displays a screen as shown in FIG. 27B on the display 180 of the electronic blackboard 1b (step S107). Here, the video v1 of the base A and the "share" button b2 that is pressed when sharing are originally displayed, and the start notification of step S106 visually indicates the passage of time for downloading the image data of the material image. Output is performed to display the hourglass icon c2.

In addition, the passage of time can be indicated by an hourglass of a still image or an hourglass of a moving image with sand movement. Further, the hourglass icon c2 is an example of schedule information indicating that the image data is scheduled to be downloaded. Other examples of appointment information may be icons other than hourglasses (eg, clock icons), characters (and / or "symbols") instead of icons, and icons and characters (eg, clock icons). And / or a combination of "symbols"). Further, the schedule information may be a notification by sound. In this case, the sound is output from the speaker 150 by the video / sound processing unit 13a instead of the display control unit 14d.

The electronic blackboard 1a may be subjected to the process of step S102 after the process of step S105. Further, the communication management system 5 may transmit the URL of the image data in the step S104 after receiving the image data upload start notification in the step S105.

Subsequently, the transmission / reception unit 11a of the electronic blackboard 1a uploads the image data of the material image to the URL related to the image storage device 7 received in step S104 (step S108). As a result, the transmission / reception unit 71 of the image storage device 7 receives the image data. Then, the storage / reading processing unit 79 of the image storage device 7 stores the image data for the URL in the storage unit 7000 (step S109).

Subsequently, at the base A, the user A1 touches and moves the electronic pen 190 or the hand H on the display 180 of the electronic blackboard 1a, so that the reception unit 12a receives the input of the movement stroke (trajectory). (Step S110). Then, the image processing unit 17a creates stroke data (for example, coordinate data (x, y)) for displaying the stroke image on the two-dimensional display 180 based on the stroke, and the display control unit 14a electronically creates the stroke data (for example, coordinate data (x, y)). A stroke image is displayed on the display 180 of the blackboard 1a (step S111). As a result, the stroke image st1 is displayed as shown in FIG. 27 (c).

The transmission information management unit 21a registers the operation data received by the reception unit 12a in the operation data list table, or replaces the operation data with the operation data in the list table (step S111-2). Details will be described with reference to FIG. At this point, No, the target object, the operation content, the position after the operation, and the gid of the operation are registered.

Next, the transmission / reception unit 11a transmits stroke data (operation data) for reproducing the stroke image created in step S111 to the communication management system 5 (step S112).

If the operation data in the provisionally reflected state is registered in the operation data list table, the transmission / reception unit 11a does not transmit the next operation data to the communication management system 5 until the actual reflection is achieved. In this embodiment as well, in order to suppress the inconvenience described with reference to FIG. 2, when the operation data in the provisionally reflected state is registered in the operation data list table, it is not replaced in step S112-2, so that this determination is made.

The transmission / reception unit 51 of the communication management system 5 receives the stroke data of the stroke image. Then, in the communication management system 5, the storage / reading processing unit 59 stores the stroke data (step S113). The operation data management unit 56 assigns a gid.

The transmission / reception unit 51 transmits the stored operation data to the transmission source electronic blackboard 1a (step S113-2). As a result, the electronic blackboard 1a receives the operation data including the gid, the transmission information management unit 21a searches the list table by the id, associates it with the same id, and registers the gid.

Next, the transmission / reception unit 51 transfers the stored stroke data to the electronic blackboard 1b on the other side (step S114). As a result, the transmission / reception unit 11b of the electronic blackboard 1b receives the stroke data. Then, in the electronic blackboard 1b, the image processing unit 17b creates a stroke image based on the stroke data, and the display control unit 14d displays the stroke image on the display 180 of the electronic blackboard 1b (step S115). As a result, as shown in FIG. 27 (d), the same stroke image st1 as the electronic blackboard 1a on the other side is displayed.

Subsequently, as shown in FIG. 26, the transmission / reception unit 71 of the image storage device 7 transmits a completion notification indicating that the upload of the image data of the material image is completed to the electronic blackboard 1a (step). S121). As a result, the transmission / reception unit 11a of the electronic blackboard 1a receives the completion notification.

Next, the transmission / reception unit 11a of the electronic blackboard 1a transmits a completion notification indicating that the upload of the image data of the material image has been completed to the communication management system 5 (step S122). The completion notification includes the URL of the image data received in step S104. As a result, the transmission / reception unit 51 of the communication management system 5 receives the completion notification. Then, the transmission / reception unit 51 forwards the completion notification including the URL to the electronic blackboard 1b of the other party (step S123). As a result, the transmission / reception unit 11b of the electronic blackboard 1b receives the completion notification.

Next, the transmission / reception unit 11b of the electronic blackboard 1b transmits request information indicating that the image data of the material image is requested to be downloaded to the URL related to the image storage device 7 received in step S123 (step S124). .. As a result, the transmission / reception unit 71 of the image storage device 7 receives the request information.

Next, in the image storage device 7, the storage / reading processing unit 79 reads out the image data of the material image to be requested from the storage unit 7000 based on the URL (step S125). Then, the transmission / reception unit 71 transmits the image data of the material image that is the request target to the electronic blackboard 1b that is the request source (step S126). As a result, the transmission / reception unit 11b of the electronic blackboard 1b downloads (receives) the image data. Then, on the electronic blackboard 1b side, the display control unit 14b displays the same material image as the material image d1 shown in FIG. 27 (c) on the screen shown in FIG. 27 (d). Hide the hourglass icon c2 that was displayed until then.

<Registration process of operation data in list management table>
The process of step S111-2 of FIG. 25 will be described in detail. FIG. 28 is an example of a flowchart illustrating a process in which the electronic blackboard 1a registers operation data in a list table of operation data.

First, the reception unit 12a of the electronic blackboard 1a accepts the input of operation data by the user (S201).

Next, the transmission information management unit 21a determines whether or not the operation data for the same object is registered in the operation data list table (S202).

If the determination in step S202 is No, the transmission information management unit 21a registers the operation data at the end of the operation data list table (S203).

If the determination in step S202 is Yes, the transmission information management unit 21a replaces the operation data of the same object in the operation data list table with the latest operation data (S204). When the latest operation data is at the top of the operation data list table, the electronic blackboard 1 can transmit the operation data to the communication management system 5 immediately after the latest operation data is registered in the operation data list table. When the replaced operation data is waiting for gid, the transmission information management unit 21a adds the latest operation data instead of replacing the operation data in the list table with the latest operation data. When the transmission / reception unit 11a receives the gid, the operation data related to the operation received by the reception unit 12a is transmitted to the communication management system 5. Note that "replace" may be "replace", "overwrite", or the like.

By doing so, only the final operation data is added to the operation data list table. Therefore, since the final operation data is transmitted to the communication management system 5 immediately after the operation, the electronic blackboard 1 does not have to transmit the operation data in the middle, and the final result can be displayed at an early stage.

[Main effects of the embodiment]
As described above, when there is operation data waiting to be transmitted for the same object as the latest operation data, the electronic blackboard 1 replaces the operation data waiting to be transmitted with the latest operation data added by the user. As a result, the operation data in the middle of the operation, which is not the final operation, is not transmitted to the communication management system 5, so that the electronic blackboard 1 can reflect the user's operation earlier.

<Other application examples>
In the above embodiment, an office device which is an electronic blackboard has been described as an example of a communication terminal, but the present invention is not limited to this. Other examples of communication terminals include PCs, smartphones, smart watches, car navigation terminals and the like. Further, the communication terminal includes a medical device. In the case of medical equipment, the document image is the patient's image.

Further, in the above embodiment, the case where the video conference is performed by the communication system 100 has been described, but the present invention is not limited to this, and the meeting, general conversation between family members and friends, remote diagnosis, or one-way operation is performed. It may be used to present the information of.

Further, the configuration examples shown in FIG. 14 and the like are divided according to the main functions in order to facilitate understanding of the processing by the electronic blackboard 1, the communication management system 5, the relay device 3, and the image storage device 7. be. The present invention is not limited by the method of dividing the processing unit or the name. The processing of the electronic blackboard 1, the communication management system 5, the relay device 3, and the image storage device 7 can be divided into more processing units according to the processing content. It is also possible to divide one processing unit so as to include more processing.

Also, the group of devices described in the examples merely represents one of a plurality of computing environments for implementing the embodiments disclosed herein. In certain embodiments, the communication management system 5 includes a plurality of computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including networks, shared memory, and the like, and perform the processes disclosed herein.

Further, the communication management system 5 can be configured to share the disclosed processing steps, such as FIGS. 25 and 26, in various combinations. For example, a process executed by a predetermined unit can be executed by a plurality of information processing devices included in the communication management system 5. Further, the communication management system 5 may be integrated into one server device or may be divided into a plurality of devices.

Each function of the embodiment described above can be realized by one or more processing circuits. Here, the "processing circuit" in the present specification is a processor programmed to execute each function by software such as a processor implemented by an electronic circuit, or a processor designed to execute each function described above. It shall include devices such as ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array) and conventional circuit modules.

1 Electronic blackboard 3 Relay device 5 Communication management system 7 Image storage device 100 Communication network

Japanese Unexamined Patent Publication No. 2011-254453 Japanese Unexamined Patent Publication No. 2015-070543

Claims (8)

A communication terminal that displays display data
A reception unit that accepts operations on the communication terminal and
An operation data storage unit that stores operation data related to an operation received by the reception unit together with identification information of the display data to be operated as a target of predetermined processing.
When the display data of the operation data for which the predetermined processing has not been performed and the display data for which the reception unit has received the operation are the same, the operation data of the operation data storage unit for which the predetermined processing has not been performed is used. The transmission information management unit that replaces the operation data received by the reception unit, and
A communication terminal characterized by having. When the display data of the operation data for which the predetermined processing has not been performed and the display data for which the reception unit has received the operation are not the same, the transmission information management unit has the operation data related to the operation received by the reception unit. The communication terminal according to claim 1, wherein is stored in the operation data storage unit together with the identification information of the display data. By transmitting the operation data stored in the operation data storage unit to the communication management system, the operation for the display data can be shared with other communication terminals.
Claim 1 is characterized in that, as the predetermined process, the operation data stored in the operation data storage unit is transmitted to the communication management system, and the communication unit receives the identification information of the operation data. Or the communication terminal according to 2. The communication unit receives the identification information of the operation data although the operation data for the same display data as the display data of the operation data related to the operation received by the reception unit has been transmitted to the communication management system. If not,
The communication terminal according to claim 3, wherein the operation data related to the operation received by the reception unit is not transmitted to the communication management system. When the communication unit receives the identification information of the operation data for the same display data as the display data of the operation data related to the operation received by the reception unit, the communication unit receives the identification information.
The communication terminal according to claim 4, wherein the operation data related to the operation received by the reception unit is transmitted to the communication management system. A communication system in which other communication terminals and communication terminals communicate with each other via a network and share display data.
The communication terminal is
A reception unit that accepts operations on the communication terminal and
An operation data storage unit that stores operation data related to an operation received by the reception unit together with identification information of the display data to be operated as a target of predetermined processing.
When the display data of the operation data for which the predetermined processing has not been performed and the display data for which the reception unit has received the operation are the same, the operation data of the operation data storage unit for which the predetermined processing has not been performed is used. It has a transmission information management unit that replaces the operation data received by the reception unit, and has.
The other communication terminal
A communication system characterized in that the display data is displayed based on the operation data received from the communication terminal. It is a communication method performed by a communication terminal that displays display data.
A step in which the reception unit accepts an operation on the communication terminal,
A step of storing the operation data related to the operation received by the reception unit in the operation data storage unit together with the identification information of the display data of the operation target as a predetermined processing target.
When the display data of the operation data for which the predetermined processing has not been performed and the display data for which the reception unit has received the operation are the same, the transmission information management unit has the operation data for which the predetermined processing has not been performed. A communication method comprising: a step of replacing operation data of a storage unit with operation data received by the reception unit. For communication terminals that display display data,
A step of accepting an operation on the communication terminal and
A step of storing the operation data related to the operation received by the reception unit in the operation data storage unit together with the identification information of the display data of the operation target as a predetermined processing target.
When the display data of the operation data for which the predetermined processing has not been performed and the display data for which the reception unit has received the operation are the same, the operation data of the operation data storage unit for which the predetermined processing has not been performed is used. The step of replacing with the operation data received by the reception unit,
A program that executes.

Images