JP2006031359A - Screen sharing method and conference support system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a screen sharing method or the like that allows people to debate while sharing the same screen and writing in the shared screen. <P>SOLUTION: The screen sharing method that provides common contents for display on a screen includes sending to a server the contents displayed on the screens of a plurality of clients; integrating the contents at the server receiving them; distributing the contents to the plurality of clients; and displaying them by superimposing on the contents of the each clients on the screens. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a screen sharing technique in which a plurality of computers including a client and a server are connected to a network, and the client and the server display common contents on the screen.

  Today, technologies for digitizing various contents such as video, text, graphics, music, etc. and distributing them using broadband have been developed. However, since the amount of such information is large, the method of distributing all at once with the address of the distribution destination has a bottleneck that the amount of transmitted information becomes enormous and the server load on the distribution side becomes extremely heavy. .

  Therefore, for example, a method of delivering and distributing contents one after another by duplicating content at a node on a transmission path has attracted attention.

  On the other hand, systems that handle multimedia are also being actively developed. For example, an invention related to an electronic conference system using a plurality of personal computers (hereinafter referred to as “PCs”), and communication between a plurality of handwriting input devices. Many inventions have been made to synchronize writing contents related to a plurality of PCs and a plurality of handwriting input devices.

  Among them, there is a so-called white board in which a screen sharing technique is used and a plurality of participants write characters and figures with pointers on the screen. In addition, tools that allow multiple clients connected to a single server to view the same screen and perform remote operations (for example, Microsoft NetMeeting, AT & T VNC, etc.) have been developed. Has been.

  Therefore, the whiteboard application is started, and the server PC itself writes to the whiteboard locally, while the client PC remotely operates the server PC by the screen sharing application and writes to the whiteboard. is there.

  However, since this method has only one logical input means for writing on the whiteboard, a plurality of people cannot write at the same time.

  Therefore, by performing exclusive control, the operator is limited to one to avoid writing conflict.

  In addition, a number of techniques for sharing screen contents between a plurality of PCs have been developed.

  For example, when a plurality of pointers are displayed in a video conference system, only the right person's pointer is displayed, and the person is identified by the color or shape of the pointer (see Patent Document 1).

  However, with this technique, it is not possible to operate the screen or perform writing with each displayed pointer.

  In addition, operations by applications are exclusively controlled, but there are some operations that are given by a pointer (see Patent Document 2).

  However, this method is basically exclusive control, and simultaneous writing from a plurality of users is not possible. In addition, it is not a screen sharing but a network remote mouse, and the response speed becomes slow and drawing as expected is not possible.

Therefore, there is a technique in which drawing data is made translucent, data of a certain client is distributed to other clients via a server, and received data is superimposed (see Patent Document 3).
JP 7-298235 A JP 2000-181846 A JP 2003-143349 A

  However, the method described in Patent Document 3 or the like has a large amount of processing because the write data is transmitted to all the clients via the server, and the received client needs to draw and superimpose the respective data. There is a problem when participating from the middle.

  In addition, since drawing data is sent to everyone and received data is drawn on each PC, when multiple people write at the same time, there is a problem that the order of writing differs depending on the communication status and the processing capability of the PC. And there is no guarantee that everyone will see the same drawing screen.

  In view of the above circumstances, the present invention writes on the shared screen while sharing the screen displayed on each PC constituting the group, that is, the conference participants can display the screen of the document of the presentation or report, etc. The purpose is to provide a screen sharing method and a conference support system for supporting a conference by allowing each participant to make a necessary writing while sharing and allowing all participants to discuss while viewing the same screen. To do.

A screen sharing method of the present invention that achieves the above object is a screen sharing method in which a plurality of computers including a client and a server are connected to a network, and content common to the client and the server is displayed on the screen. ,
Send the content displayed on each client screen to the server,
The server that has received the content from each of the plurality of clients integrates the received content, displays the integrated content on the shared screen of the server, and displays the displayed integrated content to each of the plurality of clients. Deliver,
The integrated content distributed from the server is superimposed on the content of each client transmitted to the server and displayed on the screen of the client.

  In this way, after the content displayed on the screen of each PC constituting the group is sent to the server and integrated with the server, the integrated content to be distributed is displayed in real time superimposed on the content displayed on the screen. Therefore, the members of the group can share, for example, the contents that are simultaneously written on the screen while viewing the common screen.

The conference support system of the present invention that achieves the above object is a conference support system in which a plurality of computers including a client and a server are connected to a network, and content common to the client and the server is displayed on the screen. ,
The server
A first communication means for transmitting and receiving data to and from each of a plurality of clients;
First display means for merging data received from each of the plurality of clients and displaying integrated content on a screen;
Each of the multiple clients
A second communication means for transmitting / receiving data to / from the server;
An input means for inputting data;
A second display means for displaying the integrated content based on the merged data received from the server on the screen, and displaying the content based on the data input from the input means on the integrated content.
The second communication means transmits data representing the content displayed on the screen superimposed on the second display means to the server,
The first communication unit distributes merged data representing the integrated content displayed on the screen by the first display unit to each of the plurality of clients.

  For example, the contents displayed by being written are transmitted to the server in each PC constituting the group, and the server merges them and distributes them to each PC. Each PC can view a common screen at the same time, and can also share content written simultaneously from each PC over the content displayed on the screen. Therefore, all participating members can write and discuss while viewing the same screen.

  According to the screen sharing method and the conference support system of the present invention, it is possible to write on the shared screen displayed in common on each PC constituting the group, so that all the participants can see the same screen for discussion. Is possible.

Embodiments of the conference support system of the present invention using the screen sharing method of the present invention will be described below.
(First embodiment)
As a general rule, participants can view the same screen when they write and discuss characters and figures on the display screens of multiple PCs connected to the network. is there. Therefore, in the conference support system of the present embodiment, each of a plurality of PCs has a whiteboard function that can draw characters and figures by moving a pointer displayed on the screen, and each PC is drawn. Combined with the screen sharing function, each of the plurality of PCs shares the displayed screen by superimposing and displaying the integrated drawing image obtained by integrating the drawing images on the drawing images drawn on each of the plurality of PCs in real time. .
Thereby, the conference participants can simultaneously write using the mouse or the like while sharing the same screen in each PC.

  1 and 2 are schematic diagrams showing an embodiment of the conference support system of the present invention. FIG. 1 is a diagram for explaining a screen sharing function, and FIG. 2 is a diagram for explaining a whiteboard function. .

  The conference support system of the present embodiment is configured by a plurality of personal computers (hereinafter referred to as “PCs”) connected to a network, one of the plurality of PCs being a server, and the other PC being a client. However, for the sake of convenience, here, a conference support system constituted by one server and one client will be described.

  The conference support system according to the present embodiment shown in FIGS. 1 and 2 includes a server PC and a client PC. The server PC 10 and the client PC 20 are connected by a network 30 such as a LAN, for example. Communication is performed according to the IP protocol.

  As shown in FIG. 1, the server PC 10 includes an OS communication unit serving as an interface with the network 30 and a screen sharing server 12 that displays an integrated drawing image on the screen. The client PC 20 serves as an interface with the network 30. An OS communication unit 21 and a screen sharing client 22 that displays an integrated drawing image on a screen are provided.

  The screen sharing server 12 includes a receiving unit 13 that receives an operation event from the client PC 20 via the OS communication unit 11, and an execution unit 14 that causes the server PC 10 to generate an operation event based on the operation event received by the receiving unit 13. ing. Moreover, the acquisition part 15 which acquires the screen data displayed on the desktop screen of server PC10, and the transmission part 16 which transmits the screen data acquired to client PC via OS communication part in real time are provided.

  The screen sharing client 22 includes a receiving unit 23 that receives screen data from the server PC 10 via the OS communication unit 21, a display unit 24 that displays an integrated drawing image based on the screen data received by the receiving unit 23, and a mouse. An acquisition unit 25 that acquires an operation event input by the movement of the computer, and a transmission unit 26 that transmits the operation event acquired by the acquisition unit 25 to the server PC 10 via the OS communication unit 21.

  The display unit 24 makes a display request to the OS of the client PC 20, and the acquisition unit 25 acquires an operation event according to an instruction from the OS.

  In this embodiment, a mouse is used as the vector data input means. However, a keyboard or a trackball can be used.

  As shown in FIG. 2, the server PC 10 merges the OS communication unit 11 serving as an interface with the network 30, the rendered image based on the vector data input by the mouse of the server PC 10 itself, and the vector data received from the client PC 20. The whiteboard server 1 displays the integrated drawing image superimposed on the screen, and the client PC 20 draws the OS communication unit 21 as an interface with the network 30 and the vector data input by the mouse of the client PC 20 itself. And a whiteboard client 31 that superimposes and displays an image and an integrated drawing image based on the merged vector data received from the server PC 10.

  The whiteboard server 1 includes an acquisition unit 4 that acquires vector data based on mouse events written by the mouse of the server PC 10 itself, a reception unit 2 that receives vector data of the client PC 20 received via the OS communication unit 11, The client management unit 3 that manages the client PC 20, the data storage unit 5 that stores the vector data acquired by the acquisition unit 4 and received by the reception unit 2, and the vector data stored in the data storage unit 5 are merged. And a drawing unit 6 displayed on the screen. The acquisition unit 4 acquires vector data in response to an instruction from the OS of the server PC 10.

  The whiteboard client 31 acquires an acquisition unit 35 that acquires vector data based on a mouse event written by the mouse of the client PC 20 itself, a data storage that stores the vector data acquired by the acquisition unit 35 and the vector data received from the server PC 10 Unit 32, transmission unit 33 that transmits vector data acquired by acquisition unit 35 stored in data storage unit 32 to server PC 10 via OS communication unit 21, and vector data stored in data storage unit 32 And a drawing unit 34 that displays the integrated drawing image and the drawing image written by the client PC 20 on the screen.

  Note that the acquisition unit 35 acquires vector data in response to an OS command from the client PC 20, and the drawing unit 34 issues a display request to the OS.

  Here, the whiteboard is a function for drawing a line segment based on coordinates obtained from an input device such as a mouse. For example, many application programs such as Microsoft MS Paint are known. .

  FIG. 3 is a diagram showing an embodiment of the conference support system of the present invention.

  The conference support system of this embodiment shown in FIG. 3 has one server and two clients A and B.

  Each of the server PC 10 and the client PC 20 has a white board server 1, a white board client 31, a screen sharing server 12, a screen sharing client 22, and a white board function depending on whether it functions as a server or a client. A connection control unit 40 that switches between a board function and a screen sharing function is provided.

  Here, the server and the client of this embodiment are both PCs, and can be connected by, for example, Ethernet (registered trademark).

  On each PC, the whiteboard application of the present embodiment is activated, and the whiteboard activated as a server has a screen sharing function.

  With this screen sharing function, the server desktop data or part of the screen data is transmitted to the client, and the client can share the server screen data by displaying the received screen data. In these software, it is possible to remotely operate the server PC from the client by transmitting input events such as a mouse and a keyboard from the client to the server.

  As software having a screen sharing function, for example, VNC (Virtual Network Computing) of AT & T or NetMeeting of Microsoft Corporation can be used.

  In the conference support system of the present embodiment, the client browses the screen of the whiteboard server 1 by the screen sharing function, so that the same whiteboard screen can be browsed by a plurality of PCs. Therefore, the write data can be merged and integrated, and problems such as a different drawing screen being displayed on any PC can be avoided.

  In addition, the whiteboard server 1 in the conference support system according to the present embodiment has a function for drawing input from the server's own mouse and the like, and vector data sent from the client via the network, for example, the coordinates of the start point and the end point. The configured vector information is received, and drawing can be performed on the screen based on the vector information.

  Therefore, in the whiteboard of this embodiment, instead of transmitting a mouse input event, vector data is transmitted, and the server sequentially draws the received vector data as line segments. For this reason, even if writing is performed simultaneously from a plurality of clients, drawing is possible, and further drawing is possible with the mouse of the server PC.

  In the conference support system of the present embodiment, the server PC 10 starts up the whiteboard server 1 and the screen sharing server 12 by the connection control unit 40, and the client PCs 20 </ b> A and B are connected to the whiteboard client 31 by the connection control unit 40. The screen sharing client 22 is activated, the server PC 10 and the client PC 20 are connected to the whiteboard server 1 and the whiteboard client 31, and the screen sharing server 12 and the screen sharing client 22 are connected.

  As described above, since the functions of both the server PC 10 and the client PC 20 are provided, and the function can be switched according to the role by the connection control unit 40, for example, the chairperson can be arbitrarily set according to the state of the conference. it can.

  FIG. 4 is a diagram illustrating an example of a whiteboard client and a whiteboard server.

  The three screens on the left side shown in FIG. 4 represent the screen of the client PC 10, and the three screens on the right side represent the screen of the server PC 20.

  The screen of the client PC 10 includes a local whiteboard (upper part of the client PC screen) and a shared screen (middle part) window. In practice, the local whiteboard and the shared screen are displayed in an overlapping manner (lower part). ).

  On the other hand, the server PC 20 draws the vector data received from the client (upper, middle, lower).

  When the client PC 20 moves a pointer (not shown) displayed on the local whiteboard while clicking the mouse, a drawing image is displayed on the local whiteboard, and the vector data is sent to the server PC 10 at the same time.

  When the screen display is updated by the vector data received from the client PC 20, the server PC 10 detects the screen change and transmits the vector data displayed on the screen to the client PC 20. . When receiving the vector data, the client PC 20 updates the shared screen.

  At this time, in the client PC 20 that performed the previous drawing, the written content is drawn not only on the local whiteboard but also on the shared screen.

  These operations are shown in a flowchart.

  FIG. 5 is a flowchart showing the operation of the shared screen in the client whiteboard.

  As shown in FIG. 5, on the shared screen of the client PC, when new screen data is received from the server (S1), the shared screen is updated (S2), and the same screen as the server whiteboard is displayed.

  FIG. 6 is a flowchart showing the operation of the local whiteboard among the client whiteboards.

  As shown in FIG. 6, when the user moves the pointer on the client whiteboard screen while clicking the mouse (S5), coordinates are read in units of 10 ms, for example, and vector data is acquired (S6). Then, a line segment connecting the two coordinates is drawn on the local whiteboard (S7). Then, the vector data drawn on the local whiteboard is transmitted to the server whiteboard in units of 0.5 seconds, for example (S8). That is, the drawn content is temporarily stored as vector data from the start point coordinates to the end point coordinates, and when 0.5 seconds have elapsed, the stored vector data string is transmitted to the server.

  Here, the acquired data is not necessarily vector data, but may be coordinate data, bitmap data, or vector data, coordinate data, or bitmap data in ZLIB format or ZIP format. The data may be compressed by an arbitrary compression method.

  The time interval for reading the coordinates is determined according to the basic performance of the OS and CPU of the PC, the CPU load, and the like. In addition, the time interval for transmitting data to the server does not necessarily need to be in units of 0.5 seconds, and may be set to, for example, 0.2 seconds in order to improve responsiveness. If the communication environment is not good, The interval may be 1 second. Alternatively, it may be transmitted in units of one stroke each time the mouse button is detected to be UP.

  The reason for this configuration is to improve the usability when the user writes to the local whiteboard. The acquired vector data is sent directly to the server, and the rendered image displayed on the server whiteboard is displayed on the screen. This is to avoid a situation in which, when the procedure of browsing by sharing is taken, the time lag until it is displayed on the client screen increases, and the display of the drawn image does not follow the pointer trajectory of the client PC.

  As a result, the drawing image displayed on the local whiteboard of the client and the vector data transmitted from the client to the server are displayed on the shared screen on the server, and are received and displayed on the shared screen of the client. The drawn image overlaps and is displayed redundantly, but these overlappingly displayed drawn images are exactly overlapped by making the coordinates and display magnification the same, for example by echo etc. A double drawn image is not displayed.

  As a result, the client that performed the writing can write at a relatively high speed without being aware of any change in the screen.

In the conference support system of this embodiment, since the client has a local whiteboard, the client can save or reuse the vector data written on the local whiteboard.
(Second Embodiment)
The second embodiment is different from the first embodiment in that the client does not have a local whiteboard, so that writing to the shared screen is different, but the other points are common. Therefore, the difference will be described.

  7 and 8 are flowcharts showing the operation of the shared screen of the client.

  As shown in FIG. 7, when the user moves the pointer on the shared screen while clicking the mouse (S11), the coordinates are read and vector data is acquired (S12). Then, the line segment connecting the two coordinates is overwritten on the shared screen by superimposing the drawn image already displayed (S13). Then, the read vector data is transmitted to the server whiteboard (S14).

  As shown in FIG. 8, when the drawing image written on the shared screen while the user clicks the mouse is temporarily displayed superimposed on the already displayed drawing image (S13), from the server to the server When new vector data displayed on the whiteboard is received (S15), the shared screen that was temporarily displayed is updated, and the same drawing image displayed on the whiteboard of the server is displayed ( S16).

  A specific description will be given by focusing on one pixel to which the client has written.

  Before writing, this pixel is white, but it is assumed that it is painted black by writing, for example. At the time of writing, the coordinates on the whiteboard of the server or other client remain white. When the vector data including the pixel is transmitted from the client and drawn on the whiteboard on the server, the target pixel at the corresponding coordinates on the server whiteboard is repainted in black. Therefore, when a change in the screen area including the pixel is detected by the screen sharing function, information indicating that the pixel of interest is black is transmitted to all clients. The target pixel of the client that performed the writing was temporarily black, but when the latest shared screen vector data is received from the server, it is updated to the vector data, and the pixel is overwritten with black. As a result, the display of the client becomes the same as the server from a temporary one.

  FIG. 9 is a diagram schematically illustrating a drawn image displayed on the shared screen of the client as time elapses.

  The shared screen shown in FIG. 9 transitions from the left side to the right side of the diagram as time passes, the upper part is the overwritten part that is written on the shared screen using a mouse or the like, and the middle part is the same as the server whiteboard. The shared screen, the lower row, shows a screen that is actually displayed on the PC.

  It is assumed that a white client is in a completely white state and a client writes an X mark on the left side of the screen (1) and then writes a diagonal line at the bottom center of the screen (2).

  The left column in the figure shows the state immediately after writing locally. The shared screen with the server (middle row) remains pure white, and only the client that performed the writing displays the cross mark locally on the pure white shared screen. And diagonal lines are drawn (upper and lower).

  In the middle column of the drawing, the vector data (3) on the left written on the left side is transmitted to the server (upper row) and drawn on the server whiteboard. The vector data drawn on the whiteboard is transmitted from the server to each client. The client that has performed the writing receives the vector data (4) of the x mark from the server, and updates the screen data of the screen area surrounded by the dotted line including the x mark (middle stage). The cross marked by the client locally has been completely rewritten by the screen data (5) received from the server.

  On the other hand, the slanted line still remains as it is when the client that performed the writing rewrites the shared screen data locally (upper row).

  Even if it is received from the server, unless another client is writing to the same area at the same timing, the client who has written does not feel a change in the screen (lower row). It should be noted that only the crosses are displayed on the shared screen of other clients and the whiteboard of the server, and there is no diagonal line.

  In the right column of the figure, the hatched vector data (6) is transmitted to the server whiteboard (upper stage), and the screen data of the screen area including the hatched line is drawn on the server.

  In the client that performed the writing, a cross mark and a diagonal line are drawn on the shared screen, and the diagonal line that was displayed by rewriting the data of the shared screen locally is overwritten by the data of the latest shared screen completely. Yes (middle, lower).

  In this way, it is possible to perform remote writing without delay by temporarily rewriting the shared screen.

  Here, the case where the drawing image displayed on the shared screen of the client according to the present embodiment is written from one client has been described as an example, but the same applies to the case where writing is performed simultaneously from a plurality of clients.

Compared with the first embodiment, the present embodiment does not need to permanently store vector data drawn locally by the client, thus saving memory. Further, since only one screen data is required and there is no need for superposition, the load on the CPU can be reduced.
(Third embodiment)
In the third embodiment, the server PC and the client PC have a plurality of screen buffers as compared with the first embodiment, and a plurality of desktop screens can be displayed on other PCs via the network. Is different.

  FIG. 10 is a diagram showing a third embodiment of the conference support system of the present invention.

  The conference support system of this embodiment shown in FIG. 10 has one server and two clients A and B.

  Each of the server PC 10 and the client PC 20 has a white board server 1, a white board client 31, a screen sharing server 12, a screen sharing client 22, and a white board function depending on whether it functions as a server or a client. A connection control unit 40 that switches between a board function and a screen sharing function is provided.

  In the conference support system of the present embodiment, in the server PC 10, the whiteboard server 1, the whiteboard client 31, and the screen sharing server 12 are activated by the connection control unit 40, and the client PCs 20 </ b> A and B are connected to the connection control unit 40. As a result, the whiteboard client 31 and the screen sharing client 22 are activated. The server PC 10 and the client PC 20 are connected to the whiteboard server 1 and the whiteboard client 31, and are connected to the screen sharing server 12 and the screen sharing client 22.

  In addition, unlike the server PC of the first embodiment, the server PC 10 of the present embodiment uses the function of the whiteboard client 31 activated simultaneously when writing on the screen of the server PC 10 itself.

  Here, a UNIX (registered trademark) OS is used for the server PC 10 and the client PC 20, but any OS having a function having a plurality of screen buffers may be used, and the OS is not necessarily limited to the UNIX (registered trademark).

  FIG. 11 is a diagram illustrating the whiteboard function of the server of the conference support system according to the third embodiment.

  As shown in FIG. 11, the whiteboard function of the server according to the present embodiment is compared with the whiteboard function of the server according to the first embodiment described in FIG. It is provided and not provided in the whiteboard server. However, since other functions are shared with those described in the first embodiment, the same functions are denoted by the same reference numerals and description thereof is omitted.

  FIG. 12 is a diagram illustrating an example of the screen buffer of the server.

  As shown in FIG. 12, the whiteboard server 1 activates two screen buffers.

  The whiteboard server 1 is activated on the first screen buffer 100, but the first screen buffer 100 is not displayed on the display but is not displayed. On the other hand, the whiteboard client 31 is activated on the second screen buffer 200 displayed on the display, and the activated whiteboard client 31 is connected to the whiteboard server 1.

  Since the whiteboard server 1 is not displayed, it is drawn at a higher speed than when displayed on the display. Further, the screen change of the whiteboard server 1 is detected by using the screen sharing function, but since the actual drawing is not performed, the detection of the screen update becomes faster and the followability of the entire system is improved.

Here, since it is not necessary for the whiteboard server 1 to perform drawing directly with the mouse, only the vector data from the client may be received and drawn, or writing may be performed from the mouse of the server PC 10. Even if it exists, since it is performed by the whiteboard client, it can be handled in the same manner as the client PC 20, and the data processing is simplified.
(Fourth embodiment)
The fourth embodiment is different from the first to third embodiments in that the server whiteboard has a plurality of windows, and the drawn images displayed in the plurality of windows are superimposed and displayed. However, other points are common. Therefore, the difference will be described.

  FIG. 13 is a diagram schematically illustrating a whiteboard of a server according to the fourth embodiment.

  As shown in FIG. 13, in the whiteboard server 1, a client A window 51 and a client B window 52 are arranged on a desktop 50, and a server window 53 for drawing vector data acquired by a mouse event is the most. Arranged on the top surface.

  The whiteboard server 1 manages a drawing surface for each user, has a socket (not shown) that waits for a connection from a client, and has a function of displaying a new drawing surface on a shared screen and a function of deleting.

  Each window has a separate socket, and the whiteboard client 31 can use the socket to connect to the server PC by, for example, TCP / IP.

  FIG. 14 is a flowchart illustrating the operation of the whiteboard of the server according to the fourth embodiment.

  As shown in FIG. 14, when the server whiteboard starts waiting for connection (S21), when a connection from the client whiteboard is accepted (S22), a new window is created for the client (S23). A new socket is created (S24). Then, the client connected through the standby socket is connected to the socket of this new window, and the window and the client whiteboard are connected (S25). Further, the new window is superimposed and displayed on the screen of the server PC in the same manner as other windows (S26).

  Even if a client disconnects, the client window, the vector data drawn in the window, and the network address of the client are retained, and when the client reconnects, the same window is displayed. By adding a function to reassign, it is possible to make a system that can withstand a network trouble.

  In the conference support system according to the present embodiment, a separate socket is assigned to each window. Therefore, by designing each window to have a communication function and a drawing function, a plurality of clients can write simultaneously. When this is done, the processing is simplified and drawing can be performed at high speed. Also, by assigning colors for each window, the user can be easily identified and the processing can be speeded up.

  Next, color allocation for each window will be described.

  FIG. 15 is a diagram illustrating an example when colors are assigned to each window.

  As shown in FIG. 15, the client 51's window 51 has black as the writing color, the client B's window 52 has red as the writing color, and the server window 53 has blue as the writing color. ing. In that case, the vector data transmitted from the client to the server does not include color information. Further, when the server performs color display on each window, it is not necessary to determine which client has drawn the drawing and add color to the vector data.

  The drawing image of the color designated in this way is displayed in a superimposed manner as shown in the upper left of the figure. Therefore, regardless of the PC processing capacity, communication volume, and data volume, it is possible to automatically color each user and display the written contents.

  Further, when the server assigns a color to the client, the server can instruct the client of the color, and the drawing image color displayed locally on the client can also be changed to follow the designated color.

  FIG. 16 and FIG. 17 are diagrams illustrating an example when colors are assigned to each client.

  The server whiteboard manages the color information assigned to each client and each client.

  As shown in FIG. 16, when a client is connected (S31), information such as a network address and a client name is acquired (S32). A color to be assigned to the client is determined based on the acquired information (S33).

  For example, when the colors assigned in the order of connection are determined to be black, red, blue, green, etc., the address and color information can be managed in pairs.

  Then, as shown in FIG. 17, when writing is performed by the client and the server receives the vector data (S35), an IP address is acquired from the client (S36). Which color is assigned to the acquired IP address is collated (S37), and the vector data is displayed on the server whiteboard with the assigned color (S38).

  Here, the color assigned by the server can be freely changed at an arbitrary timing.

  Next, storage of data when colors are assigned to each window described in FIG. 15 will be described.

  As shown in FIG. 15, when writing is performed by the three participants as shown in the upper left of the drawing, management data is added to the drawing image data displayed in the window for each participant and saved separately. can do.

For example, the conference material image (background image) is bg-0001. bmp, the rendered image of the server is svr-0001. bmp, the drawing image of client A is cl1-0001. bmp, the drawing image of client B is cl2-0001. If the file is saved with the file name bmp, wbd-0001. The management data file saved with the file name wbd
[Background]
file = bg-0001. bmp
[Server]
file = svr-0001. bmp
[Client1]
file = cl1-0001. bmp
[Client2]
file = cl2-0001. bmp is described.
If you want to reproduce the saved whiteboard screen later, load the management data file with the application that browses the whiteboard, load the background image based on the data described there, use it as the background, and place it on it. The rendered images of the server and each client are displayed superimposed.

  At this time, if only the writing of a specific user is displayed, it is easy to understand who wrote which writing.

  For example, the screen data of the whiteboard shown in FIG. 18 is stored, and the user name is recorded in the management data file as follows.

In FIG. 18, drawn images of Taro, Hanako, A field, and B river (respectively, “this”, “?”, “Hand mark”, “× and diagonal line”) are displayed together with the respective marks.
[Background]
file = bg-0001. bmp
[Server]
file = svr-0001. bmp
color = BLACK
name = Taro
[Client1]
file = cl1-0001. bmp
color = RED
name = Hanako
[Client2]
file = cl2-0001. bmp
color = BLUE
name = A field
[Client3]
file = cl3-0001. bmp
color = GREEN
When browsing the name = B River whiteboard screen, for example, if “Taro” and “Hanako” are designated and displayed by the “display user selection dialog” 55, a screen as shown in FIG. 19 is displayed. , A field and B river writing (× and diagonal lines) is not displayed. In this case, when the user name is recorded in the management data, it is possible to display in the “display user selection dialog” 55 which color the user has used in an easily understandable manner.

Here, the same function can be realized even when colors are assigned by addresses as follows.
[Background]
file = bg-0001. bmp
[Drawing]
file = drw−0001. bmp
[Server]
name = Taro color = BLACK
[Client1]
name = Hanako color = RED
[Client2]
name = A field color = BLUE
[Client3]
name = B river color = GREEN
Here, when writing from the client, writing on the shared screen without any advance notice, and only the lines are drawn, which may give an unnatural impression to the person who saw the screen. As a countermeasure, the mark 56 can be drawn as an image at the position of the latest vector data on the server whiteboard. That is, since this mark is an image, unlike the pointer displayed on the server whiteboard by the coordinates of the mouse, the whiteboard cannot be operated by this mark 56.

  For example, by changing the shape of the mark 56 for each client or displaying the user name together, it is possible to make it easy to understand who is writing what.

  For example, as shown in FIG. 18, it is understood that three users Taro, Hanako, and B River are writing, and A is only pointing.

  If each client has a window, the mark shape and the user name to be displayed can be registered for each window, and the mark can be displayed for each window.

However, if the burden on the screen sharing function becomes a problem, the mark is not drawn on the server whiteboard, the coordinates are sent to all clients, and the mark of each participant is drawn locally on each client. You can also.
(Fifth embodiment)
Unlike the first to fourth embodiments, the fifth embodiment is written on the whiteboard on each PC over the desktop screen while all the participants in the conference see the common desktop screen (for example, conference material). The drawn images are displayed in a superimposed manner so that discussion can be performed.

  FIG. 20 is a diagram illustrating a conference support system according to the fifth embodiment.

  The conference support system of this embodiment shown in FIG. 20 has one server and two clients A and B.

  The server PC 10 has a desktop screen 50 on the bottom surface, and an application can be executed on the desktop screen 50 to open a document. On the desktop screen 50, there is a window 54 in which the windows 51 to 52 of each client and the window 53 of the server are superimposed and displayed with a transparent background.

  The client PC 20 has a shared screen 60 with the window 54 of the server PC 10 on the bottom surface. On the shared screen, a local window 61 in which each client writes, a local window 61 and a shared screen 60 There is a window 62 in which is superimposed and displayed.

  Here, the background can be made transparent and superimposed, for example, by using a Windows (registered trademark) 2000 or higher OS of Microsoft Corporation by using an API function group called Layered Window.

  On the screen of the server PC 10, a window 54 in which whiteboard writing is superimposed and displayed on the screen of an application running on the desktop screen 50 is shared with each client PC 20 by the screen sharing function.

  In addition, when the drawing image written on the whiteboard is stored, the entire display screen of the server PC 10 or the screen of the application running on the desktop of the server PC 10 is stored at the same time. Can be stored in an easy-to-understand form when viewed at a later date.

  As a format of data to be saved, for example, a state in which a whiteboard screen is displayed superimposed on a desktop screen need not be limited to bitmap data, and is saved in a compression format such as JPEG format, GIF format, or PNG format. You can also.

  Here, when a document is opened on the desktop and the whiteboard is displayed on the desktop, the written content of the whiteboard can be pasted as data such as bitmap data on data such as conference materials.

  FIG. 21 is a diagram illustrating an example of a flowchart for performing processing for pasting the content written on the whiteboard to a conference material or the like.

  As shown in FIG. 21, when a server whiteboard is instructed to be pasted on a document, the whiteboard writing content is first acquired as image data (S41). For example, bitmap data of the displayed image is acquired.

  Then, the bitmap data is copied to the shared memory (S42).

  Next, the handle of the application that opens the document to be pasted is acquired (S43). For example, once the whiteboard is minimized and displayed, the document to be written is clicked and specified.

  Then, an instruction to paste the write bitmap data in the shared memory is issued to the application of the designated handle (S44), and the write bitmap data in the shared memory is pasted to the document data (S45). .

  Such an operation can be realized by creating an external program that operates software that opens a document by using software provided with a development environment, such as Microsoft Office. it can. Specifically, when a whiteboard is displayed over a document, the written image is displayed as if it was written directly on the document by matching the position of the image on the whiteboard with the position of the document. Can be pasted as bitmap data. When pasting bitmap data, the position and size are matched to the document, and the bitmap background is pasted transparently.

  Here, when acquiring the handle of a document, the topmost window below the whiteboard may be automatically specified, or a list of currently open windows is displayed and the user can be displayed. A dialog may be displayed that indicates which window is for the document to be pasted.

  Alternatively, bitmap data may be pasted on a document using an application macro. That is, a macro for pasting a bitmap image by matching the positions is created, installed in a document or an application environment for displaying the document, and the macro is called from the outside.

  Furthermore, bitmap data may be pasted on a document by transmitting a key event such as Ctrl + V to the application. However, this method may not be able to align or make the background transparent.

  The format of the image data need not be limited to the bitmap format. For example, when the document file is opened by an application capable of pasting a compressed image in JPEG format, GIG format, or PNG format, such format is used. The image data may be pasted on the document.

  Next, a data storage method will be described.

  FIG. 22 is a flowchart in the case of separately storing an image of a conference material or the like displayed on the desktop and a written image.

  As shown in FIG. 22, when saving is started in the whiteboard (S46), a screen shot image of the desktop screen is saved as image data in a bitmap format, for example (S47). Next, the drawing data of the integrated drawing image obtained by superimposing the drawing images displayed in the windows of the respective clients is stored as bitmap format image data (S48). Finally, management data for associating the two image data is stored (S49).

  Later, when browsing the whiteboard image, the management data is first read, the image data associated with the management data and the drawing data are read, and the latter is superimposed on the former and displayed. The drawing data may be saved as a vector data string.

  FIG. 23 is a flowchart for making a whiteboard transparent when the whiteboard is activated.

  As shown in FIG. 23, when the whiteboard is activated, a pure white screen is displayed, and processing is performed to avoid hiding images such as conference materials displayed on the desktop. When the whiteboard is activated, a screen shot image of the desktop screen is taken (S50), and the screen is set as the whiteboard background image (S51).

  When the client is connected (S52), the desktop screen image is transmitted to the client (S53).

In this embodiment, the desktop screen image is displayed on the whiteboard. However, the background color of the whiteboard may be transparent and displayed on the desktop screen.
(Sixth embodiment)
Since the sixth embodiment is different from the first to fifth embodiments in that the server has a function of managing the right to write to the whiteboard, the difference will be described.

  24 and 25 are flowcharts illustrating the operation of the client in the conference support system according to the sixth embodiment.

  The whiteboard of the server of this embodiment manages the write right, and when the write right given to a certain client is transferred to another client, in the example of FIG. 24, whether or not the client has the write right. In the example of FIG. 25, the server notifies the client that the write right is given.

  In the example shown in FIG. 24, when the client acquires vector data from the mouse (S55), the client determines whether or not the write right is given (S56), and when the write right is not given, the vector data Are stored (S57), and when the write right is given, the vector data is transmitted to the server (S58).

  Thus, when there is no write right, vector data is not transmitted even if local drawing is performed, but is temporarily stored in a local memory or the like. Further, when it is detected that the write right has been transferred, the stored vector data is sequentially transmitted to the server, and the written drawing image is drawn on the server.

  Here, it is possible for a client to which no write right is given not to display the local window and to prevent the client from drawing.

  In another example shown in FIG. 25, instead of handing the write right to the next person, the write right is released, and in the state where the write right is released, the write is made to the person who first started writing. Rights are granted.

  If the write right is released (S62), the write right is given to the user who performed the write earlier, in a first-come-first-served basis (S63). When the write right is obtained, a timer is activated on the server whiteboard (S64). This timer is set to 15 seconds, for example.

  When writing is performed (S66), the timer is reset (S67), and when writing is stopped, the timer starts again (S64). When 15 seconds have elapsed after the writing is stopped and the timer ends (S65), the write right is automatically released (S68).

  In the following, the write right is given to the user who has written earlier, with the first-come-first-served basis.

  It should be noted that other users cannot write while someone has write permission.

Here, the timer is set to 15 seconds, but is not necessarily 15 seconds, and may be set to 30 seconds, for example, or may be set to an arbitrary time according to the scene.
In addition, a client that does not have the write right can prevent local overwriting of the shared screen even if local drawing is performed.

  In addition to the method shown here, it is possible to manage permission / rejection of the write right on the server whiteboard.

It is a schematic diagram (screen sharing function) which shows embodiment of the meeting assistance system of this invention. It is a schematic diagram (whiteboard function) which shows embodiment of the meeting assistance system of this invention. It is a figure which shows embodiment of the meeting assistance system of this invention. It is a figure which shows an example of a client whiteboard and a server whiteboard. It is a flowchart which shows operation | movement of a shared screen among client whiteboards. It is a flowchart which shows operation | movement of a local whiteboard among client whiteboards. It is a flowchart which shows operation | movement of a shared screen of a client. It is a flowchart which shows operation | movement of a shared screen of a client. It is a figure which shows typically the drawing image displayed on a shared screen of a client as time passes. It is a figure which shows 3rd Embodiment of the meeting assistance system of this invention. It is a figure explaining the whiteboard function of the server of the meeting assistance system of 3rd Embodiment. It is a figure which shows an example of the screen buffer of a server. It is a figure which shows typically the whiteboard of the server of 4th Embodiment. It is a flowchart which shows operation | movement of the whiteboard of the server of 4th Embodiment. It is a figure which shows the example at the time of allocating a color for every window. It is a figure which shows the example at the time of allocating a color for every client. It is a figure which shows the example at the time of allocating a color for every client. It is a figure which shows a white board screen. It is a figure which shows a white board screen. It is a figure which shows the meeting assistance system of 5th Embodiment. It is a figure which shows an example of the flowchart which performs the process which pastes the content written in a whiteboard on a meeting material. It is a flowchart in the case of preserve | saving separately images, such as a meeting material displayed on the desktop, and a writing image. It is a flowchart in the case of making a whiteboard transparent when a whiteboard is started. It is a flowchart which illustrates the operation | movement of the client in the meeting assistance system of 6th Embodiment. It is a flowchart which illustrates the operation | movement of the client in the meeting assistance system of 6th Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Whiteboard server 2 Vector data receiving part 3 Client management part 4 Mouse event acquisition part 5 Data storage part 6 Drawing part 10 Server PC
DESCRIPTION OF SYMBOLS 11, 21 OS communication part 12 Screen sharing server 13 Operation event receiving part 14 Operation event execution part 15 Screen data acquisition part 16 Screen data transmission part 20 Client PC
DESCRIPTION OF SYMBOLS 22 Screen sharing client 23 Screen data receiving part 24 Screen data display part 25 Operation event acquisition part 26 Operation event transmission part 30 Network 31 Whiteboard client 32 Data storage part 33 Vector data transmission part 34 Vector data drawing part 35 Mouse event acquisition part 40 Connection control unit 50 Desktop screen 51 Client A window 52 Client B window 53 Server window 54, 62 Overlaid window 55 Display user selection dialog box 56 Mark 60 Shared screen 61 Local window 100 First screen buffer 200 Second screen buffer

Claims (19)

A screen sharing method in which a plurality of computers including a client and a server are connected to a network, and content common to the client and the server is displayed on the screen,
Send the content displayed on each client screen to the server,
The server that has received the content from each of the plurality of clients integrates the received content, displays the integrated content on the shared screen of the server, and displays the displayed integrated content to each of the plurality of clients. Deliver,
The screen sharing method, wherein the integrated content distributed from the server is displayed on the screen of the client so as to overlap the content of each client transmitted to the server. A conference support system in which a plurality of computers including a client and a server are connected to a network, and content common to the client and the server is displayed on the screen,
The server
A first communication means for transmitting and receiving data to and from each of a plurality of clients;
First display means for merging data received from each of the plurality of clients and displaying integrated content on a screen;
Each of the plurality of clients
Second communication means for transmitting and receiving data to and from the server;
An input means for inputting data;
A second display unit that displays the integrated content based on the merged data received from the server on the screen and displays the content based on the data input from the input unit on the integrated content.
The second communication means transmits data input from the input means and displayed on the screen by the second display means to the server,
The meeting support system, wherein the first communication unit distributes merged data representing the integrated content displayed on the screen by the first display unit to each of the plurality of clients. The first display means displays an integrated drawing image on the screen by merging vector data received from each of the plurality of clients,
The server includes server input means for inputting vector data, and displays a drawing image based on the vector data input from the server input means so as to overlap the integrated drawing image displayed by the first display means. The conference support system according to claim 2, wherein:   The second display means displays a drawing image based on vector data input from the input means in a window different from a window in which the integrated drawing image is displayed, or a window in which the integrated drawing image is displayed. 4. The conference support system according to claim 3, wherein the conference support system is displayed by being overwritten on the integrated drawing image.   The first display means has a plurality of windows different for each of the clients, and displays the integrated drawing image in which drawing images displayed in the plurality of windows are superimposed on a shared screen. The meeting support system according to Item 3.   The first display means displays a transparent window that transmits the integrated drawing image on the forefront of the screen, and displays the drawing image based on the vector data input from the server input means. The conference support system according to claim 3, wherein the conference support system is displayed.   4. The conference support system according to claim 3, wherein the first display unit simultaneously stores all or part of the integrated drawing image and vector data representing the drawing image. The second display means displays a transparent window that transmits the integrated drawing image on the forefront of the screen, and the drawing image based on the vector data input from the input means is displayed on the transparent window. Display
The first communication unit distributes merged data representing the integrated drawing image displayed behind the transparent window by the first display unit to each of the plurality of clients;
7. The conference support system according to claim 6, wherein the second communication means transmits vector data representing the drawing image displayed on the transparent window by the second display means to the server.   The first display means separates and stores vector data representing the drawing image displayed in the transparent window and vector data representing the integrated drawing image displayed behind the transparent window. The conference support system according to claim 6, wherein the integrated drawing image is displayed behind the drawing image based on the vector data.   The first display means separates and stores vector data representing the integrated drawing image for each of the plurality of clients, and displays the integrated drawing image based on the stored vector data. 6. The meeting support system according to 6.   The first display means separates and stores vector data representing the integrated drawing image for each of the plurality of clients, and stores the vector data of the predetermined client selected from the plurality of clients based on the stored vector data. The conference support system according to claim 6, wherein vector data is merged to display an integrated drawing image.   The first display means displays a document image based on predetermined document data, and a drawing image represented by vector data received from the plurality of clients is displayed at a position on the displayed document image at a designated position. 6. The conference support system according to claim 5, which has a pasting function.   The first display means attaches a color assigned according to the address to each of the portions displayed based on the vector data received from the plurality of clients when the integrated drawing image is displayed on the screen. The conference support system according to claim 3. The plurality of clients have different colors assigned to each;
The said 1st display means attaches the color allocated to each of these several clients to each component part of the said integrated content, when displaying the said integrated drawing image on a screen. Meeting support system.   The said 1st display means preserve | saves the information of the color allocated to each of these several clients collectively, when preserve | saving the merged vector data showing the said integrated drawing image. Or the meeting support system of 14. The second communication means transmits data representing the position of the pointer displayed on the screen by the second display means to the server,
The first display means predetermines the received position of the pointer in a window for displaying the integrated drawing image based on data representing the position of the pointer received from the plurality of clients by the first communication means. 6. The conference support system according to claim 5, wherein the conference support system is displayed with a mark.   17. The conference support system according to claim 16, wherein the predetermined mark has a different shape depending on each of the plurality of clients, or includes an identifier for identifying each of the plurality of clients. The first communication unit includes a write right management unit that determines whether or not each of the vector data received from the plurality of clients has a write right.
Each of the plurality of clients, when the vector data is input by the input unit, transmits the vector data from the second communication unit to the server according to the presence or absence of the write right,
6. The conference support system according to claim 5, wherein the first display unit displays a drawing image based on the vector data determined to have the write right by the write right management unit.   The write right management means has a function of canceling the write right of the client when the vector data received from the client having the write right is in a stopped state for a predetermined time. 19. The conference support system according to claim 18, wherein after the write right is released, the write right is given to the client that first transmitted the vector data to the first communication means. .

Images