JP2015060592A - Image processing system, and information processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system for enabling a user of each base to grasp operations of users of other bases in advance.SOLUTION: A first information processor includes operation reception means 26, operation contents notification information creation means 39 for creating operation contents notification information of an operation, first display means control means 37 for reflecting operation contents in display means of the first information processor, and transmission means 65 for transmitting the operation contents to second information processor. The second information processor makes display means of the second information processor display the operation contents notification information transmitted from the first information processor, and displays an image reflecting the operation contents in the display means of the second information processor on the basis of the operation contents transmitted from the first information processor.

Description

  The present invention relates to an image processing system in which a plurality of information processing apparatuses that share an image displayed on a display means are connected via a network.

  An electronic blackboard is used in which a background image is displayed on a large display at a meeting in a company, an educational institution, an administrative organization, etc., and a user can draw stroke images such as letters, numbers, and figures on the background image. . In such an electronic blackboard, a stroke image generated by a user on a touch panel or the like can be displayed on the touch panel. The stroke image is stored as page data together with the background for each page.

  In addition, with the spread of networks, a technology is also known in which electronic blackboards at each base are connected by a network so that each electronic blackboard shares a stroke image or the like (see, for example, Patent Document 1). Patent Document 1 discloses a mutual information sharing system that transmits base identification information together with input data, and displays the data by attaching base information such as an operator.

  However, as in the mutual information sharing system of Patent Document 1, the input data and the base information can be transmitted at the same time to display the base information in which the data is input, but when the base is known, the data is already drawn. There is a problem that has been. When a plurality of bases share a stroke image, users at a plurality of bases can write a stroke at an arbitrary timing, but often write in the same place. For this reason, if the base information is displayed together with the data, the user at each base cannot predict the operation of the user at the other base. For example, a plurality of users may write strokes at overlapping locations, and the strokes overlap. It may be difficult to interpret.

  In addition, it is not limited to drawing strokes, and if the operation of a user at another site cannot be predicted, a user at another site may perform an operation that contradicts the operation of a user at another site, and a good operational feeling is obtained. There may be no case.

  In view of the above problems, an object of the present invention is to provide an image processing system in which users at each site can grasp in advance the operations of users at other sites.

  The present invention is an image processing system in which a plurality of information processing apparatuses including a display unit for displaying an image are connected via a network, wherein the first information processing apparatus includes an operation receiving unit that receives an operation, and the operation Operation content notification information creating means for creating operation content notification information for notifying an operation accepted by the accepting means, and an image reflecting the operation contents of the operation are displayed on the display means of the first information processing apparatus. 1 display means control means, and transmission means for transmitting the operation content notification information and the operation content to the second information processing device, wherein the second information processing device is configured to transmit the first information. The operation content notification information transmitted from the processing device is displayed on the display means of the second information processing device, and based on the operation content transmitted from the first information processing device. And displaying an image in which the operation content is reflected on the display unit of the second information processing apparatus.

  It is possible to provide an image processing system in which users at each base can grasp in advance the operations of users at other bases.

1 is an example of an overall configuration diagram of an image processing system. It is an example of the hardware block diagram of an electronic blackboard. It is an example of the functional block diagram of an electronic blackboard. It is an example of a functional block diagram of a file processing unit. It is an example of the functional block diagram of a communication control part. It is an example of the figure which illustrates page data typically. It is an example of the figure which illustrates stroke sequence data typically. It is an example of the figure which illustrates coordinate arrangement data typically. It is an example of the figure which illustrates media data typically. It is an example of the figure which illustrates a remote license management table typically. It is an example of the figure which illustrates an address book management table typically. It is an example of the figure which illustrates backup data typically. It is an example of the figure which illustrates a connecting point management table typically. It is an example of the figure which illustrates a participating base management table typically. It is an example of the figure which illustrates operation data typically. It is an example of the figure explaining the superimposition process of each image layer. An example of the sequence diagram in which an electronic blackboard participates in a remote conference is shown. An example of the sequence diagram in which an electronic blackboard displays an image is shown. It is a figure which shows the example of a display of the list | wrist of the base displayed on the display of a participating apparatus. It is an example of the flowchart figure which shows the transmission procedure of a remote context. It is an example of the remote context which the base which received the remote context displays on a display. It is an example of the schematic block diagram of a contact sensor. It is an example of the flowchart figure which shows the procedure in which a finger discrimination | determination part discriminate | determines that it is drawing with a finger | toe and draws a marker. (Example 3) which is an example of the functional block diagram of an electronic blackboard. It is an example of the figure explaining an affine transformation matrix. It is an example of the flowchart figure which shows the procedure which shares a gesture in two bases.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

<System overview>
FIG. 1 is an overall configuration diagram of an image processing system according to the present embodiment. In FIG. 1, only two electronic blackboards 2a and 2b and associated electronic pens 4a and 4b are shown to simplify the description. A pen or the like may be used. The electronic blackboards 2a and 2b are examples of the information processing apparatus in the claims.

As shown in FIG. 1, the image processing system 1 includes a plurality of electronic blackboards 2a and 2b, a plurality of electronic pens 4a and 4b, USB memories 5a and 5b, notebook PCs (Personal Computers) 6a and 6b, a television ( Video) having conference terminals 7a, 7b and PC8. The electronic blackboards 2a and 2b and the PC 8 are connected via a communication network 9 so that they can communicate with each other. Further, the plurality of electronic blackboards 2a and 2b are respectively provided with displays 3a and 3b. Also, the electronic blackboard 2a has an event generated by the electronic pen 4a (the pen tip of the electronic pen 4a on the display 3a or An image drawn by touching the pen butt of the electronic pen 4a can be displayed on the display 3a. Note that the image displayed on the display 3a can be changed based not only on the electronic pen 4a but also on an event (a gesture such as enlargement, reduction, or page turning) generated by the user's hand Ha or the like.

  The electronic blackboard 2a can be connected to a USB memory 5a. The electronic blackboard 2a reads an electronic file such as PDF from the USB memory 5a, and the electronic blackboard 2a records an electronic file in the USB memory 5a. Can do. In addition, the electronic blackboard 2a is connected via a cable 10a1 capable of communication according to standards such as Display Port, DVI (Digital Visual Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), and VGA (Video Graphics Array). The notebook PC 6a is connected. The electronic blackboard 2a generates an event by touching the display 3a, and transmits event information indicating the event to the notebook PC 6a in the same manner as an event of an input device such as a mouse or a keyboard. Similarly, a television (video) conference terminal 7a is connected to the electronic blackboard 2a via a cable 10a2 capable of communication according to the above-mentioned standards. Note that the notebook PC 6a and the video conference terminal 7a may communicate with the electronic blackboard 2a by wireless communication complying with various wireless communication protocols such as Bluetooth (registered trademark).

  On the other hand, in the other bases where the electronic blackboard 2b is installed, the electronic blackboard 2b with the display 3b, the electronic pen 4b, the USB memory 5b, the notebook PC 6b, the video conference terminal 7b, the cable 10b1, and the cable 10b2 in the same manner as described above. Is used. Furthermore, the image displayed on the display 3b can be changed based on an event caused by the user's hand Hb or the like.

  Thereby, the image drawn on the display 3a of the electronic blackboard 2a at one base is also displayed on the display 3b of the electronic blackboard 2b at another base, and conversely on the display 3b of the electronic blackboard 2b at another base. The image drawn on is displayed on the display 3a of the electronic blackboard 2a at one base. As described above, the image processing system 1 can perform remote sharing processing for sharing the same image at a remote location, and thus is very convenient when used for a conference or the like at a remote location.

  In the following, when an arbitrary electronic blackboard among the plurality of electronic blackboards is indicated, it is indicated as “electronic blackboard 2”. When an arbitrary display among a plurality of displays is shown, “display 3” is shown. When an arbitrary electronic pen among a plurality of electronic pens is indicated, it is indicated as “electronic pen 4”. When an arbitrary USB memory is indicated among the plurality of USB memories, it is indicated as “USB memory 5”. When an arbitrary notebook PC among the plurality of notebook PCs is indicated, it is indicated as “notebook PC 6”. When an arbitrary video conference terminal is indicated among the plurality of video conference terminals, it is indicated as “video conference terminal 7”. In addition, “hand H” is used when an arbitrary hand among the hands of a plurality of users is shown. When an arbitrary cable is indicated among the plurality of cables, it is indicated as “cable 10”.

  In the present embodiment, an electronic blackboard is described as an example of an image processing device. However, the present invention is not limited to this, and other examples of the image processing device include electronic signboards (digital signage), sports and weather forecasts. It may be a telestrator to be used, a remote image (video) diagnostic device, or the like. The notebook PC 6 will be described as an example of the information processing terminal, but is not limited thereto, and other examples of the information processing terminal include a desktop PC, a tablet PC, a smartphone, a PDA, a digital video camera, and a digital camera. A terminal capable of supplying image frames such as a game machine may be used. Further, the communication network includes the Internet, a LAN (Local Area Network), a mobile phone communication network, and the like. In this embodiment, a USB memory is described as an example of a recording medium. However, the present invention is not limited to this, and various recording media such as an SD card may be used as another example of the recording medium.

<Hardware configuration of electronic blackboard>
Next, the hardware configuration of the electronic blackboard according to the present embodiment will be described with reference to FIG. FIG. 2 is a hardware configuration diagram of the electronic blackboard.

  As shown in FIG. 2, the electronic blackboard 2 includes a CPU 101 that controls the operation of the electronic blackboard 2 as a whole, a ROM 102 that stores a program used to drive the CPU 101 such as an IPL, and a RAM 103 that is used as a work area of the CPU 101. And an SSD 204 for storing various data such as a program for the electronic blackboard 2, a network controller 105 for controlling communication with the communication network 9, and an external storage controller 106 for controlling communication with the USB memory 5.

  The electronic blackboard 2 also includes a capture device 111 that captures still images or moving images displayed on the display of the notebook PC 6, a GPU (Graphics Processing Unit) 112 that specializes in graphics, and an output image from the GPU on the display 3 or A display controller 113 that controls and manages screen display for output to the video conference terminal 7 is provided.

  The electronic blackboard 2 further includes a sensor controller 114 that controls processing of the contact sensor 115 and a contact sensor 115 that detects that the electronic pen 4 or the hand H of the user touches the display 3. The contact sensor 115 performs coordinate input and coordinate detection by an infrared ray blocking method. In this coordinate input and coordinate detection method, two light emitting / receiving devices (not shown) installed at both upper ends of the display 3 emit a plurality of infrared rays in parallel with the display 3, This is a method of receiving light that is reflected by the reflecting member provided on the light source and returns on the same optical path as the light path of the light emitted by the light receiving element. The contact sensor 115 outputs infrared ID (Identification) emitted by the two light receiving and emitting devices blocked by the object to the sensor controller 114, and the sensor controller 114 identifies the coordinate position that is the contact position of the object. All IDs shown below are examples of identification information.

  In addition, the contact sensor 115 is not limited to the infrared blocking method, and a capacitive touch panel that identifies a contact position by detecting a change in capacitance, or a contact position determined by a voltage change between two opposing resistance films. Various detection means such as a resistance film type touch panel to be identified and an electromagnetic induction type touch panel to identify a contact position by detecting electromagnetic induction caused by contact of a contact object with the display unit may be used.

  The electronic blackboard 2 includes an electronic pen controller 116. This electronic pen controller 116 communicates with the electronic pen 4 to determine whether or not there is a pen tip touch or a pen butt touch on the display 3. It should be noted that the electronic pen controller 116 may determine whether or not the user grips the part of the electronic pen 4 as well as the other electronic pen part, as well as the pen tip and the bottom of the electronic pen 4.

  Further, the electronic blackboard 2 includes the CPU 101, ROM 102, RAM 103, SSD 104, network controller 105, external storage controller 106, capture device 111, GPU 112, sensor controller 114, and electronic pen controller 116 as shown in FIG. Are provided with a bus line 120 such as an address bus and a data bus.

  The program for the electronic blackboard 2 may be recorded and distributed on a computer-readable recording medium such as a CD-ROM.

<Functional configuration of electronic blackboard>
Next, the functional configuration of the electronic blackboard will be described with reference to FIGS. First, the overall functional configuration of the electronic blackboard 2 will be described with reference to FIG. FIG. 3 is a functional block diagram of the electronic blackboard.

  The electronic blackboard 2 has the functional configurations shown in FIG. 3 by the hardware configuration and program shown in FIG. The electronic blackboard 2 can be a “host device” that starts the remote sharing process first, and can also be a “participating device” that participates in the remote sharing process that has already started. The electronic blackboard 2 includes a client machine 20 and a server machine 90 as main functions. That is, the client machine 20 and the server machine 90 are included in one casing of the electronic blackboard 2. The client machine 20 and the server machine 90 are functions realized in one housing of the electronic blackboard 2. When the electronic blackboard 2 is a host device, the electronic blackboard 2 implements the client machine 20 and the server machine 90. When the electronic blackboard 2 is a participating device, the client machine 20 is realized in the electronic blackboard 2, but the server machine 90 is not realized. That is, in FIG. 1, when the electronic blackboard 2a is the hosting device and the electronic blackboard 2b is the participating device, the client machine 20 of the electronic blackboard 2a is connected via the server machine 90 implemented in the same electronic blackboard 2a. The electronic blackboard 2b communicates with the client machine 20. On the other hand, the client machine 20 of the electronic blackboard 2b communicates with the client machine of the other electronic blackboard 2a via the server machine 90 realized in the other electronic blackboard 2a.

[Functional configuration of client machine 20]
Subsequently, the functional configuration of the client machine 20 will be described mainly using FIGS. 3 to 5. The client machine 20 includes a video acquisition unit 21, a coordinate detection unit 22, an automatic adjustment unit 23, a contact detection unit 24, an event distribution unit 25, an operation processing unit 26, a gesture processing unit 27, a video superimposition unit 28, and a finger determination unit 29. -1, a finger input start standby timer 29-2, an image processing unit 30, and a communication control unit 60.

  Among these, the video acquisition unit 21 acquires an output video of a video output device such as a notebook PC 6 connected to the cable 10. When the video acquisition unit 21 receives an image signal from a video output device (such as a notebook PC 6), the video acquisition unit 21 analyzes the image signal, and the resolution of an image frame that is a display image of the video output device formed by the image signal, Image information such as the update frequency of the image frame is derived and output to the image acquisition unit 31.

  The coordinate detection unit 22 detects the coordinate position of an event (such as an operation in which the user's hand H is touched on the display 3) generated by the user on the display 3. The coordinate detection unit 22 also detects the touched area.

  The automatic adjustment unit 23 is activated when the electronic blackboard 2 is activated, and adjusts the image processing parameters of the sensor camera in the optical sensor method of the contact sensor 115 so that the contact sensor 115 can output an appropriate value to the coordinate detection unit 22. To do.

  The contact detection unit 24 detects an event (such as an operation in which the pen tip of the electronic pen 4 or the pen butt of the electronic pen 4 is pressed (touched) on the display 3).

  The event distribution unit 25 distributes the coordinate position of the event detected by the coordinate detection unit 22 and the detection result detected by the contact detection unit 24 to each event of stroke drawing, UI operation, and gesture operation.

  Here, “stroke drawing” is performed when the user presses the electronic pen 4 on the display 3 when a stroke image (B) described later shown in FIG. 16 is displayed on the display 3. In this state, the electronic pen 4 is moved and finally the electronic pen 4 is released from the display 3. By this stroke drawing, for example, alphabets “S” and “T” are drawn on the display 3. The “stroke drawing” includes not only drawing an image but also an event of deleting an already drawn image or editing a drawn image.

  The “UI operation” is an event in which the user presses a predetermined position with the electronic pen 4 or the hand H when a UI image (A) described later shown in FIG. 16 is displayed on the display 3. . By this UI operation, for example, the color or width of a line drawn by the electronic pen 4 is set.

  The “gesture operation” is an event in which the user touches or moves the display 3 with the hand H when the stroke image (B) described later shown in FIG. is there. By this gesture operation, for example, by moving the hand H while the user touches the display 3 with the hand H, the image can be enlarged (or reduced), the display area can be changed, or the page can be switched. it can.

  The operation processing unit 26 executes various operations according to the UI element in which the event is generated, from what is determined as the UI operation by the event distribution unit 25. Examples of the UI element include a button, a list, a check box, and a text box. The gesture processing unit 27 generates a gesture event by determining the content of the gesture from the history of coordinates and the like for the event determined to be a gesture operation by the event distribution unit 25.

  The finger determination unit 29-1 determines that the finger is touching the screen. Since the coordinate detection unit 22 detects coordinates using an infrared ray blocking method and triangulation, it is possible to detect contact with an indicator other than an electronic pen. For this reason, it is determined by the finger determination unit 29-1 whether the contact is made by the electronic pen or by an indicator other than the electronic pen. In addition, although a finger is described as an example of an instruction tool other than the electronic pen, infrared rays can be shielded without using a finger.

  The finger determination unit 29-1 acquires a touch detection image event and a pen-down event from the event distribution unit 25 for finger determination. The contact detection image event is an event in which light shielding is detected in the detection area of the contact sensor 115 (an area parallel to the screen that is several cm or less from the screen). The pen down event is an event in which the touch detection unit 24 detects a touch of the pen tip of the electronic pen 4 or the pen butt of the electronic pen 4. In addition, the finger determination unit 29-1 acquires an event during gesture from the gesture processing unit.

  The finger determination unit 29-1 measures the time by the finger input start waiting timer 29-2 according to the presence / absence of the touch detection image event, the pen down event, the gesture event, and the size of the image of the image event ( Count). In addition, when the finger input start standby timer 29-2 starts counting, the finger determination unit 29-1 accumulates (queues) the coordinate values in time series.

  When the finger input start standby timer 29-2 counts a time equal to or longer than a predetermined time, the finger determination unit 29-1 determines that the finger is being drawn and transmits the coordinates queued to the marker processing unit 38. . The marker processing unit 38 draws a series of coordinates as a marker, and thereafter continues drawing the marker using the acquired coordinates as the last coordinate of the marker.

  Therefore, when the finger determination unit 29-1 determines that it is a finger, the marker processing unit 38 draws with a marker, and when it determines that it is an electronic pen, the stroke processing unit 32 draws a stroke with the pen. Determining that the finger determination unit 29-1 is a finger may be referred to as detecting a marker event.

  In the present embodiment, since the marker event is included in the remote context described later, the finger determination unit 29-1, the finger input start standby timer 29-2, and the marker processing unit 38 have been described, but the details will be described in the second embodiment. explain.

  The video superimposing unit 28 displays the image superimposed by the display superimposing unit 36 described later as a video on the video output device (display 3). In addition, the video superimposing unit 28 performs picture-in-picture of the video sent from another video output device (such as the video conference terminal 7) with respect to the video from the video output device (such as the notebook PC 6). Further, the video superimposing unit 28 performs switching for displaying the video that is picture-in-picture and displayed on a part of the display 3 on the entire display 3.

  The image processing unit 30 performs superimposition processing of each image layer. Specific examples of superposition will be described with reference to FIG. The image processing unit 30 includes an image acquisition unit 31, a stroke processing unit 32, a UI image generation unit 33, a background generation unit 34, a layout management unit 35, a marker processing unit 38, a context processing unit 39, a display superimposing unit 36, and a page processing unit. 37, a file processing unit 40, a page data storage unit 300, and a remote license management table 310.

  Among these, the image acquisition unit 31 acquires each frame as an image from the video acquired by the video acquisition unit 21. This image corresponds to an output image (C) from the video output device (notebook PC 6 or the like) shown in FIG.

  The stroke processing unit 32 draws an image, deletes the drawn image, or edits the drawn image based on the event related to the stroke drawing assigned by the event distribution unit 25. The image by this stroke drawing corresponds to the stroke image (B) shown in FIG. In addition, the results of drawing, deleting, and editing an image based on the stroke and the like are stored in the operation data storage unit 840 as operation data described later.

  The UI image generation unit 33 generates a UI (user interface) image set in advance on the electronic blackboard 2. This UI image corresponds to the UI image (A) shown in FIG.

  The background generation unit 34 receives media data from the page data read from the page data storage unit 300 by the page processing unit 37 from the page processing unit 37. The background generation unit 34 outputs the received media data to the display superimposing unit 36. The image based on the media data corresponds to the background image (D) shown in FIG. The pattern of the background image (D) is plain or grid display.

  The layout management unit 35 provides the layout information indicating the layout of each image output from the image acquisition unit 31, the stroke processing unit 32, and the UI image generation unit 33 (or the background generation unit 34) to the display superimposing unit 36. Is managing. Thereby, the layout management unit 35 causes the display superimposing unit 36 to display the output image (C) and the stroke image (B) in the UI image (A) and the background image (D). It is possible to instruct whether to hide.

  The context processing unit 39 transmits a marker event, a gesture event, a page creation / deletion event, and a file read event to the server machine 90 via the remote context transmission unit 68 of the communication control unit 60. In addition, the context processing unit 39 acquires a marker event, a gesture event, a page creation / deletion event, and a file read event from the remote context receiving unit 67 of the communication control unit 60. Also, the other site participation event and the other site leaving event are acquired from the participating site management table 610 of the communication control unit 60. The context processing unit 39 generates a base image according to the acquired event and base and displays it on the display 3. Note that a page creation / deletion event occurs when a user performs a page creation or deletion operation on a UI image. The file read event occurs when the file input unit 42a reads a file. The other site participation event occurs when a new site is registered in the participating site management table 610. The other site exit event occurs when a site is deleted from the participating site management table 610.

  The display superimposing unit 36 is based on the layout information output from the layout management unit 35, and includes an image acquisition unit 31, a stroke processing unit 32, a UI image generation unit 33 (background generation unit 34), a marker processing unit 38, and a background generation unit. The layout of each image output from 34 is performed.

  The page processing unit 37 stores the data of the stroke image (B) and the data of the output image (C) in one page data and stores them in the page data storage unit 300. The data of the stroke image (B) forms part of page data as stroke arrangement data (each stroke data) indicated by the stroke arrangement data ID shown in FIG. The data of the output image (C) forms part of the page data as the media data indicated by the media data ID shown in FIG. Then, when this media data is read from the page data storage unit 300, it is handled as data of the background image (D).

  Further, the page processing unit 37 transmits the media data of the once stored page data to the display superimposing unit 36 via the background generating unit 34 so that the video superimposing unit 28 displays the background image (D). 3 can be displayed again. The page processing unit 37 can return the stroke arrangement data (each stroke data) in the page data to the stroke processing unit 32 so that the stroke can be re-edited. Further, the page processing unit 37 can delete or duplicate page data.

  That is, the data of the output image (C) displayed on the display 3 when the page processing unit 37 stores the page data in the page data storage unit 300 is temporarily stored in the page data storage unit 300 and then When read from the page data storage unit 300, it is read as media data indicating the background image (D). Then, the page processing unit 37 outputs stroke arrangement data indicating the stroke image (B) among the page data read from the page data storage unit 300 to the stroke processing unit 32. Further, the page processing unit 37 outputs media data indicating the background image (D) among the page data read from the page data storage unit 300 to the background generation unit 34.

  The display superimposing unit 36 includes an output image (C) from the image acquisition unit 31, a stroke image (B) from the stroke processing unit 32, a UI image (A) from the UI image generation unit 33, and a background generation unit 34. The background image (D) is superimposed according to the layout designated by the layout management unit 35. Accordingly, as shown in FIG. 16, each of the UI image (A), the stroke image (B), the output image (C), and the background image (D) is displayed in the order in which the user can see even if the images overlap. It has Leia composition.

  The display superimposing unit 36 can also superimpose the image (A) and the image (B) exclusively by switching the image (C) and the image (D) shown in FIG. For example, when the cable 10 between the electronic blackboard 2 and the video output device (notebook PC 6 or the like) is disconnected while the image (A), the image (B), and the image (C) are initially displayed. According to the designation of the layout management unit 35, the drawing (C) can be removed from the superimposition target and the image (D) can be displayed. The display superimposing unit 36 also performs display enlargement, display reduction, and display area movement processing.

  The page data storage unit 300 stores page data as shown in FIG. FIG. 6 is a conceptual diagram showing page data. The page data is data for one page displayed on the display 3 (stroke arrangement data (each stroke data) and media data). Since there are many types of parameters included in the page data, the contents of the page data will be described here separately in FIGS.

  As shown in FIG. 6, the page data includes a page data ID for identifying an arbitrary page, a disclosure time indicating the start time of display of this page, and rewriting of the contents of the page by a stroke, a gesture, or the like. Are associated with the end time indicating the time when the operation is stopped, the stroke arrangement data ID for identifying the stroke arrangement data generated by the stroke of the electronic pen 4 or the user's hand H, and the media data ID for identifying the media data. Is remembered. The stroke arrangement data is data for displaying a stroke image (B) shown in FIG. The media data is data for displaying a background image (D) shown in 16 described later on the display 3.

  With such page data, for example, when the user draws the alphabet “S” with the electronic pen 4, the stroke is written in one stroke, and therefore, one stroke data ID is displayed and one letter of the alphabet [S] is indicated. However, when the user draws the alphabet “T” with the electronic pen 4, it is written with two strokes, so that one stroke of the alphabet “T” is indicated by two stroke data IDs.

  The stroke arrangement data shows detailed information as shown in FIG. FIG. 7 is a conceptual diagram showing stroke arrangement data. As shown in FIG. 7, one stroke arrangement data is represented by a plurality of stroke data. One stroke data includes a stroke data ID for identifying the stroke data, a start time indicating the start time of writing one stroke, an end time indicating the end time of writing one stroke, the color of the stroke, A stroke width and coordinate array data ID for identifying an array of stroke passing points are shown.

  Further, the coordinate array data shows detailed information as shown in FIG. FIG. 8 is a conceptual diagram showing coordinate array data. As shown in FIG. 8, the coordinate array data includes one point (X coordinate value, Y coordinate value) on the display 3, and a time difference (ms from the disclosure time of the stroke when the one point is passed) ) And writing pressure information of the electronic pen 4 at this one point.

  That is, the collection of one point shown in FIG. 8 is shown by one coordinate array data shown in FIG. For example, when the user draws the alphabet “S” with the electronic pen 4, it is drawn with a single stroke, but passes through a plurality of passing points before the drawing of “S”. Point information is shown.

  Further, the media data in the page data shown in FIG. 6 shows detailed information as shown in FIG. FIG. 9 is a conceptual diagram showing media data. As shown in FIG. 9, the media data is stored in the page data storage unit 300 from the page processing unit 37, the media data ID in the page data shown in FIG. The recording time, the position of the image displayed on the display 3 by the page data (X coordinate value, Y coordinate value), the size of the image (width, height), and data indicating the contents of the media data are shown in association with each other. Has been. Among these, the position of the image displayed on the display 3 by the page data is displayed by the page data when the coordinates of the upper left corner of the display 3 are (X coordinate value, Y coordinate value) = (0, 0). The position of the upper left corner of the image to be displayed is shown.

  Returning to FIG. 3, the remote license management table 310 manages license data necessary for executing the remote sharing process. In the remote license management table 310, as shown in FIG. 10, the product ID of the electronic blackboard 2, the license ID used for authentication, and the expiration date of the license are managed in association with each other.

(Functional configuration of the file processing unit 40)
Next, the functional configuration of the file processing unit 40 shown in FIG. 3 will be described with reference to FIG. FIG. 4 is a functional block diagram of the file processing unit. Hereinafter, the electronic blackboard 2 that starts the remote sharing process first will be referred to as “host device”, and the electronic blackboard 2 that will participate in the remote sharing processing that has already been started will be referred to as “participating device”.

  The file processing unit 40 includes a recovery processing unit 41, a file input unit 42a, a file output unit 42b, a file conversion unit 43, a file transmission unit 44, an address book input unit 45, a backup processing unit 46, a backup output unit 47, and a setting management unit. 48, a setting file input unit 49a, and a setting file output unit 49b. Furthermore, the file processing unit 40 includes an address book management table 410, a backup data storage unit 420, a setting file storage unit 430, and a connection destination management table 440.

  Among these, the recovery processing unit 41 detects the abnormal end after the electronic blackboard 2 ends abnormally, and recovers the unsaved page data. For example, in the case of a normal end, the page data is recorded as a PDF file on the USB 5 via the file processing unit 40, but in the case of an abnormal end such as when the power is down, the page data is recorded in the page data storage unit 300 Has been done. Therefore, when the power is turned on again, the recovery processing unit 41 recovers by reading the page data from the page data storage unit 300.

  The file input unit 42a reads a PDF file from the USB memory 5 and stores each page in the page data storage unit 300 as page data.

  The file conversion unit 43 converts the page data stored in the page data storage unit 300 into a PDF format file.

  The file output unit 42 b records the PDF file output by the file conversion unit 43 in the USB memory 5.

  The file transmission unit 44 transmits the PDF file generated by the file conversion unit 43 by attaching it to an electronic mail. The transmission destination of the file is displayed by displaying the contents of the address book management table 410 on the display 3 by the display superimposing unit 36, and the file transmitting unit 44 accepts the selection of the destination by the operation of the input device such as a touch panel by the user. It is determined. In the address book management table 410, as shown in FIG. 11, the name of the destination and the mail address of the destination e-mail are managed in association with each other. Further, the file transmission unit 44 can accept an input of a mail address as a destination by an operation of an input device such as a touch panel by the user.

  The address book input unit 45 reads a list file of e-mail addresses from the USB memory 5 and manages it in the address book management table 410.

  The backup processing unit 46 performs backup by storing the file output by the file output unit 42 b and the file transmitted by the file transmission unit 44 in the backup data storage unit 420. If the user does not set backup, backup processing is not performed. The backup data is stored in the PDF format as shown in FIG.

  The backup output unit 47 stores the backed up file in the USB memory 5. At the time of storage, a password is input for security by the operation of an input device such as a touch panel by the user.

  The setting management unit 48 stores and reads various setting information of the electronic blackboard 2 in the setting file storage unit 430 for management. Examples of the various setting information include network settings, date and time settings, region and language settings, mail server settings, address book settings, connection destination list settings, and backup settings. The network setting is, for example, an IP address setting of the electronic blackboard 2, a net mask setting, a default gateway setting, a DNS (Domain Name System) setting, or the like.

  The setting file output unit 49b records various setting information of the electronic blackboard 2 in the USB memory 5 as a setting file. Note that the settings file cannot be viewed by the user due to security.

  The setting file input unit 49a reads a setting file stored in the USB memory 5 and reflects various setting information on various settings of the electronic blackboard.

  The address book input unit 50 reads a list file of connection destination IP addresses for remote sharing processing from the USB memory 5 and manages it in the connection destination management table 440. As shown in FIG. 13A, when the electronic blackboard 2 is a participating device that intends to participate in the remote sharing process, the connection destination management table 440 indicates that the user of the participating device is a host device. It is a table for managing beforehand, in order to reduce the effort which inputs the IP address of the electronic blackboard which plays a role. In the connection destination management table 440, the base name of the base where the electronic blackboard 2 as the hosting device that can participate is installed, the IP address of the electronic blackboard 2 as the hosting device, and the connection state are associated with each other. It is managed. This connection state is an example of “communication state information” in the claims. The connection destination management table 440 is an example of the “state storage unit” in the claims.

  FIG. 13B is an example for explaining the meaning of the connection state. There are five types of connection states, for example, offline, online, currently being held, being held, being held (compatible), and being held (different versions). The connection state is determined by a combination of whether communication with the hosting device is possible, whether the conference is held, and whether participation is possible. If the communication availability is “impossible”, the holding state is also “unknown” and the participation availability is “unknown”, so it is determined to be offline. In the online state, whether communication is possible is “possible”, but because the holding state is “not held”, the participation possibility is “unknown”. If the communication possibility is “possible”, the holding state is “held”, and the participation possibility is also “participable”, it is determined that the event is being held. Note that “hosting” means a state in which a user of the sponsoring device performs a predetermined operation and the participating devices can share page data and the like.

  “Currently (Compatible)” is a state in which the host device and the participating device have different software versions, but participation is possible in the compatibility mode. In compatibility mode, only the functions of the older software can be used.

  “Currently held (different version)” is a state in which participation is not possible because the software version of the sponsoring device is different from that of the participating device. This connection state is a state in which participation is not possible because, for example, the versions are greatly different.

  In this case, however, the user of the participating device needs to input the IP address of the host device using an input device such as a touch panel in order to start remote request processing with the host device. Therefore, the user of the participating device obtains the IP address of the sponsoring device from the user of the sponsoring device by telephone or electronic mail.

  The participation base management table 450 is a table for managing an electronic blackboard as a participating device that is currently participating in the remote sharing process when the electronic blackboard is a sponsor device. In the participating base management table 450, as shown in FIG. 14, the name of the base where the participating electronic blackboard 2 is installed, the IP address of the electronic blackboard 2, and the base color are managed in association with each other. . When there are a plurality of participating devices, a plurality of IP addresses are registered.

  The base color is determined, for example, in the order of participation in the conference. For example, for the base color, the host base (first) is magenta, the second is cyan, the third is orange, and the fourth is green. Also, if the second base leaves, the order of the second and subsequent bases is advanced, the third base is cyan, and the fourth base is orange.

  The participation base management table is transmitted from the remote connection processing unit 81 to the client machine via the remote connection result transmission unit 72 when the participating device joins and leaves. The remote start processing unit 61 or the remote participation processing unit 62 of the client machine receives the participation base management table. Of the bases managed in the connection destination management table of the client machine, if the IP addresses of the bases in the received participating base management table match and no name is set in the connection destination management table, the participating base management table Are registered in the connection destination management table.

(Functional configuration of communication control unit 60)
Next, the functional configuration of the communication control unit 60 will be described with reference to FIG. FIG. 5 is a functional block diagram of the communication control unit 60. The communication control unit 60 controls communication with other electronic blackboards 2 and communication with the later-described communication control unit 70 in the server machine 90 via the communication network 9. Therefore, the communication control unit 60 includes a remote start processing unit 61, a remote participation processing unit 62, a remote image transmission unit 63, a remote image reception unit 64, a remote operation transmission unit 65, a remote operation reception unit 66, a remote context transmission unit 68, A remote context receiving unit 67 and a participating site management table 610 are provided.

  Among these, the remote start processing unit 61 makes a request for starting a new remote sharing process to the server machine 90 of the same electronic blackboard 2, and receives the request result from the server machine 90. In this case, the remote start processing unit 61 refers to the remote license management table 310 and makes a request to start remote sharing processing when license information (product ID, license ID, and expiration date) is managed. be able to. However, if the license information is not managed, a request to start remote sharing processing cannot be made.

  The participation base management table 610 is a table for managing an electronic blackboard as a participating device currently participating in the remote sharing process when the electronic blackboard is a sponsor device. In the participating site management table 610, as shown in FIG. 14, the name of the site where the participating electronic blackboard 2 is installed and the IP address of the electronic blackboard 2 are managed in association with each other.

  The remote participation processing unit 62 transmits the remote connection processing to the remote sharing processing to the remote connection request receiving unit 71 in the server 90 of the electronic blackboard 2 as the hosting device that has already started the remote sharing processing via the communication network 9. Request participation. Also in this case, the remote participation processing unit 62 refers to the remote license management table 310. In addition, when the remote participation processing unit 62 participates in the remote sharing process that has already been started, the IP address of the electronic blackboard 2 of the participation destination is acquired with reference to the connection destination management table 440. Note that the remote participation processing unit 62 may not input the connection destination management table, and may input the IP address of the electronic blackboard 2 as a participation destination by an operation of an input device such as a touch panel by the user.

  The remote image transmission unit 63 transmits the output image (C) sent from the video acquisition unit 21 via the image acquisition unit 31 to the server machine 90.

  The remote image receiving unit 64 receives image data from a video output device connected to another electronic blackboard 2 from the server machine 90 and outputs it to the display superimposing unit 36, thereby enabling remote sharing processing.

  The remote operation transmission unit 65 transmits various operation data necessary for remote sharing processing to the server machine 90. Examples of the operation data include adding a stroke, deleting a stroke, editing a stroke (enlarging, reducing, moving), storing page data, creating page data, duplicating page data, deleting page data, and displaying. Data related to the switching of the current page. In addition, the remote operation receiving unit 66 receives the operation data input from the other electronic blackboard 2 from the server machine 90 and outputs it to the image processing unit 30 to perform remote sharing processing.

  The remote context transmission unit 68 acquires a marker event, a gesture event, a page creation / deletion event, and a file read event from the context processing unit 39 and transmits them to the server machine 90 as a remote context.

  The remote context receiving unit 67 acquires the event information generated on the other electronic blackboard from the server machine 90, and the context processing unit 39 of the image processing unit 30 displays the base image.

[Functional configuration of server machine]
Next, the functional configuration of the server machine 90 will be described with reference to FIG. The server machine 90 is provided in each electronic blackboard 2, and any electronic blackboard 2 can serve as a server machine. Therefore, the server machine 90 includes a communication control unit 70 and a data management unit 80.

(Functional configuration of communication control unit 70)
Next, the functional configuration of the communication control unit 70 will be described with reference to FIG.

  The communication control unit 70 controls communication with the communication control unit 60 in the client machine 20 in the same electronic blackboard 2 and the communication control unit 70 in the client machine 20 in another electronic blackboard 2 via the communication network 9. . The data management unit 80 manages operation data, image data, and the like.

  More specifically, the communication controller 70 includes a remote connection request receiver 71, a remote connection result transmitter 72, a remote image receiver 73, a remote image transmitter 74, a remote operation receiver 75, a remote operation transmitter 76, The remote context receiving unit 78 and the remote context transmitting unit 77 are provided.

  Among these, the remote connection request receiving unit 71 receives a remote sharing processing start request from the remote start processing unit 61 or receives a participation request for remote sharing processing from the remote participation processing unit 62. The remote connection result transmission unit 72 transmits the result of the remote sharing processing start request to the remote start processing unit 61, or transmits the result of the participation request for the remote sharing processing to the remote participation processing unit 62.

  The remote image receiving unit 73 receives the image data (data of the output image (C)) from the remote image transmitting unit 63 and transmits it to the remote image processing unit 82 described later. The remote image transmission unit 74 receives image data from the remote image processing unit 82 and transmits this image data to the remote image reception unit 64.

  The remote operation receiving unit 75 receives operation data (data such as a stroke image (B)) from the remote operation transmitting unit 65 and transmits it to the remote operation processing unit 83 described later. The remote operation transmission unit 76 receives operation data from the remote operation processing unit 83 and transmits this operation data to the remote operation reception unit 66.

  The remote context receiver 78 receives the remote context from the client machine and outputs it to the remote operation processor 83. The remote operation processing unit 83 adds identification information (for example, an IP address) of the local site and outputs it to the remote context transmission unit 77. The remote context transmission unit 77 transmits the remote context and identification information of its own base to each client machine.

(Functional configuration of the data management unit)
Next, the functional configuration of the data management unit 80 will be described with reference to FIG. The data management unit 80 includes a remote connection processing unit 81, a remote image processing unit 82, a remote operation processing unit 83, an operation composition processing unit 84, and a page processing unit 85. Further, the server 90 includes a passcode management unit 810, a participating site management table 820, an image data management unit 830, an operation data storage unit 840, and a page data storage unit 850.

  Among these, the remote connection processing unit 81 starts the remote sharing process and ends the remote sharing process. The remote connection processing unit 81 also receives the license information received by the remote connection request receiving unit 71 from the remote start processing unit 61 together with the remote sharing processing start request or the remote sharing processing unit 62 from the remote sharing processing participation request. Based on the received license information, it is confirmed whether there is a license and whether it is within the license period. Further, the remote connection processing unit 81 confirms whether or not the participation request from the other electronic blackboard 2 as the client machine exceeds the predetermined number of possible participations.

  Further, the remote connection processing unit 81 has the same passcode that is sent when another electronic blackboard 2 has requested to participate in remote sharing processing as the passcode managed by the passcode management unit 810. It is determined whether or not there is the same, and if they are the same, participation in the remote sharing process is permitted. This passcode is issued by the remote connection processing unit 81 when a new remote sharing process is started, and the user of the electronic blackboard 2 as a participating apparatus who intends to participate in the remote sharing process can It is transmitted from the user of the electronic blackboard 2 by telephone or e-mail. As a result, the user of the electronic blackboard 2 as a participating device who intends to participate in the remote sharing process inputs the pass code to the electronic blackboard 2 and requests participation, thereby permitting participation. Note that the user's usability is given priority over security, and the confirmation of the passcode may be omitted only by confirming the license status.

  In addition, the remote connection processing unit 81 uses the participation site information included in the participation request sent from the remote participation processing unit 62 of the electronic blackboard 2 as the participation device via the communication network 9 to the participation site of the server 90. Store in the management table 820. Then, the remote connection processing unit 81 reads out the remote site information stored in the participating site management table 820 and transmits it to the remote connection result transmission unit 72. The remote connection result transmission unit 72 transmits the remote base information to the remote start processing unit 61 in the client machine 20 of the same sponsor device. The remote start processing unit 61 stores the remote site information in 610 in the participating site management table. As a result, the host device manages the remote site information on both the client machine 20 and the server machine 90.

  The remote image processing unit 82 receives image data from a video output device (such as a notebook PC 6) connected to a client device of each electronic blackboard 2 (including a client device of its own electronic blackboard that is the hosting device) during remote sharing processing. (Output image (C)) is received and stored in 830, and the display order of the image data to be remotely shared is determined in the order of time of arrival at the server device 90 of the electronic blackboard 2 that is the host device. In addition, the remote image processing unit 82 refers to the participation base management table 820, and adds to all the client machines 20 of the electronic blackboard 2 participating in the remote sharing process (including the client machine of its own electronic blackboard that is the host device). The image data is transmitted in the determined order via the communication control unit 70 (remote image transmission unit 74).

  The remote operation processing unit 83 performs various types of operation data (stroke images (B, B, B) drawn on the client machine of each electronic blackboard 2 (including the client machine of its own electronic blackboard which is the hosting apparatus) during the remote sharing process. ) And the like, and the display order of the images to be remotely shared is determined in the order of time of arrival at the server 90 of the electronic blackboard 2 that is the host device. The various operation data is the same as the various operation data described above. Further, the remote operation processing unit 83 refers to the participating site management table 820 and operates the operation data on all the client machines 20 of the electronic blackboard 2 (including the client machine of its own electronic blackboard which is the hosting apparatus) during the remote sharing process. Send.

The operation synthesis processing unit 84 synthesizes the operation data of each electronic blackboard 2 output from the remote operation processing unit 83, stores the operation data as a result of this synthesis in the operation data storage unit 840 and the remote operation processing unit 83. Return to. The operation data is transmitted from the remote operation transmission unit 76 to each of the electronic blackboard client machine that is the hosting apparatus and the electronic blackboard client machine that is the participating apparatus. Such an image is displayed. Therefore, as shown in FIG. 15, the operation data includes SEQ (Sequence), the operation name of the operation data, the IP address of the electronic blackboard 2 that is the transmission source of the operation data, and the Port of the client machine (server machine). No., the IP address of the electronic blackboard 2 that is the transmission destination of the operation data, the port number of the client machine (server machine), the operation type of the operation data, the operation target of the operation data, and the data indicating the contents of the operation data are associated Shown. For example, in SEQ1, when a stroke is drawn on the client machine (Port No .: 50001) of the electronic blackboard (IP address: 192.0.0.1) as the hosting device, the electronic blackboard (IP address: 192.0. 0.1) indicates that operation data has been sent to the server machine (Port No .: 50000). In this case, the operation type is “STROKE”, the operation object is the page data ID “p005”, and the data indicating the contents of the operation data is data indicating the stroke. In SEQ2, the client computer of the electronic blackboard (IP address: 192.0.0.1) that is the participating device from the server device (Port No .: 50000) of the electronic blackboard (IP address: 192.0.0.1) that is the host device. (Port No .: 50001) indicates that operation data has been sent.
The operation composition processing unit 84 performs composition in the order in which the operation data is input to the operation composition processing unit 84. Therefore, if the communication network 9 is not congested, remote sharing is performed in the order of strokes by the users of the electronic blackboards 2. A stroke image (B) is displayed on the displays 3 of all the electronic blackboards 2 being processed.

  The page processing unit 85 has the same function as the page processing unit 37 in the image processing unit 30 of the client machine 20, and the server machine 90 also converts the page data shown in FIGS. 6 to 8 into a page data storage unit. Store in 850. Note that the page data storage unit 850 has the same contents as the page data storage unit 300 in the image processing unit 30, and therefore the description thereof is omitted.

<Process or Operation of Embodiment>
Subsequently, the processing or operation of the present embodiment will be described with reference to FIGS. 17 and 18. 17 and 18 are sequence diagrams showing the processing of each electronic blackboard.
In the embodiment shown in FIGS. 17 and 18, the electronic blackboard 2a serves as a hosting device (server machine and client machine) that hosts the remote sharing process, and the electronic blackboards 2b and 2c participate in the remote sharing process. A case where it plays a role as a device (client machine) will be described. Further, here, the displays 3a, 3b, 3c are connected to the electronic blackboards 2a, 2b, 2c, respectively, and further notebook PCs 6a, 6b, 6c are connected respectively. Moreover, in the electronic blackboards 2a, 2b, and 2c, electronic pens 4a, 4b, and 4c are used, respectively.

  First, a process for the electronic blackboards 2b and 2c to participate in the remote sharing process will be described. When the user turns on the power switch of the electronic blackboard 2a, the client machine 20 of the electronic blackboard 2a is activated. When the user performs an operation for starting the server machine 90 using an input device such as a touch panel, the server machine 90 is transferred from the remote start processing unit 61 of the client machine 20 to the remote connection request receiving unit 71 of the server machine 90 of the same electronic blackboard 2a. An instruction to start the process is output. Thereby, in the electronic blackboard 2a, not only the client machine 20 but also the server machine 90 can start various processes (step S21).

  Next, the UI image generation unit 33 in the client machine 20 of the electronic blackboard 2a generates connection information for establishing a connection with the electronic blackboard 2a, and the video superimposition unit 28 receives the display superimposition unit 36 from the UI image generation unit 33. The connection information obtained through is displayed on the display 3a (step S22). This connection information includes the host device's IP address and the passcode generated for the current remote sharing process. In this case, the passcode stored in the passcode management unit 810 is read by the remote connection processing unit 81 shown in FIG. 5 and transmitted in the order of the remote connection result transmission unit 72 and the remote start processing unit 61. The Further, the passcode is transmitted from the communication control unit 60 including the remote start processing unit 61 to the image processing unit 30 illustrated in FIG. 3 and finally input to the UI image generation unit 33. Thereby, the connection information includes the passcode. Then, the connection information is transmitted to the users of the electronic blackboards 2b and 2c by telephone or electronic mail by the user of the electronic blackboard 2a.

  If there is the connection destination management table 440, even if the connection information does not include the IP address of the hosting device, the participating device can make a participation request. An example of a screen for participation displayed by the UI image generation unit 33 of the participation apparatus is shown in FIG.

  Next, in the electronic blackboards 2b and 2c, when input of connection information is received by the operation of an input device such as a touch panel by each user, the remote participation processing unit 62 in the client machine 20 of each electronic blackboard 2a and 2b Based on the IP address, a participation request is made by transmitting a pass code to the remote connection request receiving unit 71 in the server 90 of the electronic blackboard 2a via the communication network 9 (steps S23 and S24). As a result, the remote connection request receiving unit 71 of the communication control unit 70 receives a participation request (including a passcode) from each of the electronic blackboards 2b and 2c, and outputs this passcode to the remote connection processing unit 81.

  Next, the remote connection processing unit 81 authenticates the passcode received from each client machine 20 of the electronic blackboards 2b and 2c using the passcode managed by the passcode management unit 810 (step S25). And the remote connection result transmission part 72 notifies an authentication result to the client machine 20 of each electronic blackboard 2b, 2c (S step S26, S27). If it is determined by the authentication in step S25 that the electronic blackboard is a legitimate electronic blackboard, communication of remote sharing processing is established between the electronic blackboard 2a that is the hosting device and the electronic blackboards 2b and 2c that are participating devices. The remote participation processing unit 62 in the client machine 20 of each electronic blackboard 2b, 2c enables the start of remote sharing processing with each other electronic blackboard (steps S28, S29).

(Output image display)
Next, image data processing in remote sharing processing will be described with reference to FIG.
First, the electronic blackboard 2b displays the output image (C) on the display 3b (step S30). Specifically, the image acquisition unit 31 of the electronic blackboard 2b receives the data of the output image (C) displayed on the notebook PC 6b from the notebook PC 6b via the video acquisition unit 21, and displays the display superimposing unit 36 and the video. By transmitting to the display 3b via the superimposing unit 28, the display 3b displays the output image (C).

  Next, the image processing unit 30 including the image acquisition unit 31 of the electronic blackboard 2 b transmits the output image (C) data to the remote image transmission unit 63, so that the communication control unit 60 including the remote image transmission unit 63 is performed. The data of the output image (C) is transmitted to the communication control unit 70 of the electronic blackboard 2a that is the hosting device via the communication network 9 (step S31). Accordingly, the remote image receiving unit 73 of the electronic blackboard 2a receives the output image (C) data and outputs it to the remote image processing unit 82, so that the remote image processing unit 82 outputs the output image to the image data management unit 830. The data of (C) is stored.

  Next, the electronic blackboard 2a, which is the hosting device, displays the output image (C) on the display 3a (step S32). Specifically, the remote image processing unit 82 of the electronic blackboard 2 a outputs the output image (C) data received from the remote image receiving unit 73 to the remote image transmitting unit 74. The remote image transmission unit 74 outputs the data of the output image (C) to the remote image reception unit 64 in the client machine 20 of the electronic blackboard 2a which is the same host device. The remote image receiving unit 64 outputs the data of the output image (C) to the display superimposing unit 36. The display superimposing unit 36 outputs the output image (C) data to the video superimposing unit 28. The video superimposing unit 28 outputs the data of the output image (C) to the display 3a. Thereby, the display 3a displays the output image (C).

  Next, the communication control unit 70 including the remote image transmission unit 74 in the server machine 90 of the electronic blackboard 2a as the hosting device via the communication network 9 is the electronic blackboard 2b that is the transmission source of the data of the output image (C). The data of the output image (C) is transmitted to the communication control unit 60 of the electronic blackboard 2c other than (step S33). Thereby, the remote image receiving part 64 of the electronic blackboard 2c which is a participating apparatus receives the data of an output image (C).

  Next, the electronic blackboard 2c displays the output image (C) on the display 3c (step S34). Specifically, the remote image receiving unit 64 of the electronic blackboard 2c outputs the data of the output image (C) received in step S33 to the display superimposing unit 36 of the electronic blackboard 2c. The display superimposing unit 36 outputs the output image (C) data to the video superimposing unit 28. The video superimposing unit 28 outputs the data of the output image (C) to the display 3c. Thereby, the display 3c displays the output image (C).

  When not only the output image (C) data but also the UI image (A) and stroke image (B) data are input to the display superimposing unit 36, the display superimposing unit 36 superimposes them. An image (A, B, C) is generated, and the video superimposing unit 28 outputs the data of the superimposed image (A, B, C) to the display 3c. When video conferencing video (E) data is sent from the video conference terminal 7 to the video superimposing unit 28, the video superimposing unit 28 converts the video superimposing image (A, B, C). Video conferencing video (E) data is superimposed by picture-in-picture and output to the display 3c.

(Display of superimposed image)
Next, a process for displaying a superimposed image in the remote sharing process will be described with reference to FIG.

  First, in the electronic blackboard 2b, the user draws a stroke image (B) on the electronic blackboard 2b using the electronic pen 4b (step S41).

  Next, as shown in FIG. 16, the display superimposing unit 36 of the electronic blackboard 2b performs a stroke image (B) on the UI image (A), the output image (C), and the background image (D). And the image superimposing unit 28 displays the superimposed images (A, B, C, D) on the display 3b of the electronic blackboard 2b (step S42). Specifically, the stroke processing unit 32 of the electronic blackboard 2b receives the stroke image (B) data as the operation data from the coordinate detection unit 22 and the contact detection unit 24 via the event distribution unit 25, and displays the superimposed image. It transmits to the part 36. Accordingly, the display superimposing unit 36 can superimpose the stroke image (B) on the UI image (A) and the output image (C), and the video superimposing unit 28 is placed on the display 3b of the electronic blackboard 2b. The superimposed image (A, B, C) can be displayed.

    Next, the image processing unit 30 including the stroke processing unit 32 of the electronic blackboard 2b transmits the stroke image (B) data to the remote operation transmission unit 65, so that the remote operation transmission unit 65 of the electronic blackboard 2b communicates. The data of the stroke image (B) is transmitted via the network 9 to the communication control unit 70 of the electronic blackboard 2a that is the hosting device (step S43). Thereby, the remote operation receiving unit 75 of the electronic blackboard 2a receives the data of the stroke image (B) and outputs it to the remote operation processing unit 83, so that the remote operation processing unit 83 sends the stroke image to the operation composition processing unit 84. The data of (B) is output. In this way, the data of the stroke image (B) drawn on the electronic blackboard 2b is sequentially transmitted to the remote operation processing unit 83 of the electronic blackboard 2a that is the hosting device every time it is drawn. The data of the stroke image (B) is data shown for each stroke data ID shown in FIG. Therefore, for example, as described above, when the user draws the alphabet “T” with the electronic pen 4, the stroke image (B) indicated by each of the two stroke data IDs is sequentially transmitted. Is done.

  Next, the electronic blackboard 2a, which is the hosting device, displays on the display 3a a superimposed image (A, B, C) including the data of the stroke image (B) sent from the electronic blackboard 2b (Step S). S44). Specifically, the operation composition processing unit 84 of the electronic blackboard 2a synthesizes the data of the plurality of stroke images (B) sequentially sent via the remote operation processing unit 83 and stores them in the operation data storage unit 840. Store and return to the remote operation processing unit 83. Thereby, the interval operation processing unit 83 outputs the data of the combined stroke image (B) received from the operation combining processing unit 84 to the remote operation transmitting unit 76. The remote operation transmitting unit 76 outputs the combined stroke image (B) data to the remote operation receiving unit 66 in the client machine 20 of the electronic blackboard 2a which is the same hosting device. The remote operation receiving unit 66 outputs the combined stroke image (B) data to the display superimposing unit 36 in the image processing unit 30. Therefore, the display superimposing unit 36 superimposes the combined stroke image (B) on the UI image (A) and the output image (C). Finally, the video superimposing unit 28 displays the superimposed image (A, B, C) superimposed by the display superimposing unit 36 on the display 3a.

  Next, the communication control unit 70 including the remote operation transmission unit 76 in the server machine 90 of the electronic blackboard 2a as the hosting device via the communication network 9 is the electronic blackboard 2b that is the transmission source of the stroke image (B) data. The composite stroke image (B) data is transmitted to the communication control unit 60 of the electronic blackboard 2c other than (step S45). Thereby, the remote operation receiving part 66 of the electronic blackboard 2c which is a participating apparatus receives the data of the synthesized stroke image (B).

  Next, the electronic blackboard 2c displays the superimposed image (A, B, C) on the display 3c (step S34). Specifically, the remote operation receiving unit 66 of the electronic blackboard 2c outputs the combined stroke image (B) data received in step S45 to the image processing unit 30 of the electronic blackboard 2c. The display superimposing unit 36 of the image processing unit 30 superimposes each data of the UI image (A) and the output image (C) and the data of the combined stroke image (B), and superimposes the superimposed image ( A, B, C) data is output. The video superimposing unit 28 outputs the data of the superimposed image (A, B, C) to the display 3c. Thereby, the display 3c displays a superimposed image (A, B, C).

  In the above processing, the output image (C) is displayed on the display 3, but a background image (D) may be displayed instead of the output image (C). Alternatively, the exclusive relationship between the output image (C) and the background image (D) may be stopped, and both the output image (C) and the background image (D) may be displayed on the display 3 at the same time.

(End of participation)
Next, a process in which the participating device ends participation in the remote sharing process will be described with reference to FIG. In the embodiment shown in FIG. 18, processing for the electronic blackboard 2c to finish participation is shown.

  First, in the electronic blackboard 2c, when a participation end request is received by an operation of an input device such as a touch panel by the user, the remote participation processing unit 62 receives the remote connection request receiving unit 71 in the server machine 90 of the electronic blackboard 2a as the hosting device. In response to this, a request to end participation is made (step S47). Thereby, the remote connection request receiving unit 71 of the communication control unit 70 receives the participation end request from the electronic blackboard 2c, and outputs the participation end request together with the IP address of the electronic blackboard 2c to the remote connection processing unit 81. . Then, the remote connection processing unit 81 of the electronic blackboard 2a, based on the IP address sent from the remote connection request receiving unit 71, from the participation base management table 820, the IP of the electronic blackboard 2c that has requested participation. The address and the name of the base where the electronic blackboard 2c is installed are deleted, and the remote connection result transmission unit 72 outputs the IP address of the electronic blackboard 2c and a notification of the deletion.

  Next, the communication control unit 70 including the remote connection result transmission unit 72 instructs the communication control unit 60 in the client machine 20 of the electronic blackboard 2c to end participation via the communication network 9 (step S48). As a result, the remote participation processing unit 62 of the communication control unit 60 in the electronic blackboard 2c disconnects the communication of the remote sharing process, thereby completing the participation process, thereby completing the participation (step S49).

(Screen example for participation)
FIG. 19 is a diagram showing a display example of a list of bases displayed on the display 3 of the participating devices (electronic blackboards 2b and 2c). Each participating device has a connection destination management table. Hereinafter, display contents indicated by numbers with parentheses will be described. The UI image generation unit 33 of the participating device displays the screen of FIG. 19 according to the user's operation.

(1) The following text will be displayed as an explanation of the screen: “Participate in the remote whiteboard. After selecting the name of the whiteboard base, press the join button.”
(2) Selection from the contact list or direct input The user can select whether to select from the contact list or directly enter an IP address.

(3) One record is composed of “whiteboard base name”, “IP address or host name”, and “group name” selected from the contact list. In addition, labels of “whiteboard base name”, “IP address or host name”, and “group name” are displayed. In addition, an icon and the first character of the whiteboard base name are displayed in one record. These are described below.

  There are “group01” and “group02” as group names. This group is determined by the first character of the display name of the whiteboard base name, the IP address, etc., and is a name for grouping related bases.

  The user selects a whiteboard base name (host apparatus) to participate by pressing a record in the contact list. By selecting the whiteboard base name, the IP address (6) is displayed. Selecting the selected record again cancels the selection.

In addition, when there is no icon at the head of the record, or when an online icon or an ongoing icon described later is displayed in gray (with low luminance), the record is displayed in gray and cannot be selected. The icon legend is described below.
(4) Scroll bar (5) Specify group name The user can input or select an arbitrary group name to narrow down records to be displayed in the contact list.
(6) IP address Displays the IP address of the whiteboard base name selected in the contact list.
(7) Join This is a button for inputting that the user participates in the conference.
(8) Cancel A button for the user to close the screen without participating in the conference.
(9) Currently held icon This is an icon indicating that participation is possible for the base of this label. It is an icon displayed when the connection state in FIG.
(10) Online icon This is an icon indicating that the base of this label is connected to the network and there is a remote license. This icon is displayed when the connection state in FIG. 13B is “online”.
(11) Currently held icon (cannot participate)
This icon indicates that the site is holding a meeting, but the software version is different and it cannot participate. In the figure, the icon is displayed in gray. This icon is displayed when the connection state in FIG. 13B is “currently held (different versions)”.
(12) Compatibility mode holding icon This icon indicates that the compatibility mode is being held. This icon is displayed when the connection state in FIG. 13B is “currently held (compatible)”.
(13) A record of a base that cannot be selected, which is an example of a record of a base that cannot be selected, is displayed in gray. When the connection state in FIG. 13B is “offline”, the icon is not displayed in this way.
(14) Record of selected base The record of the selected base is displayed in a predetermined color (or high luminance).
(15) Icon Legend The “Online” label is displayed for the online icon, the “Ongoing” label is displayed for the ongoing icon, and the “Ongoing in compatible mode” label is displayed for the ongoing icon in the compatible mode. (16) First character label of the whiteboard base name This is a label for distinguishing the first character of the whiteboard base name.

[Send remote context]
The remote context is a type of event that is notified to the electronic blackboard 2 at another base before reflecting the result of the event on the image of the display 3 when a specific event is detected. As described above, marker events, gesture events, page creation / deletion events, file read events, other site participation events, other site exit events, and the like are notified to other sites by the remote context. The remote context is an example of operation content notification information in the scope of claims.

  FIG. 20 is an example of a flowchart showing a remote context transmission procedure. In FIG. 20, an operation base that acquires an event that is a remote context, a server machine 90 that communicates with the client machine 20, and a reception base that receives the remote context perform processing.

  First, the electronic blackboard 2 at the operation base acquires an event (S110). Since the events that are remote contexts are determined in advance, these events are detected by the event distribution unit 25, the operation processing unit 26, the gesture processing unit 27, the finger determination unit 29-1, and the like.

  The context processing unit 39 determines a predetermined remote context name corresponding to the event, and transmits it to the server together with operation base identification information (such as an IP address) (S111). The remote context name may be, for example, a remote context (event name).

  Thereafter, the operation base acquires operation contents corresponding to the input coordinates and event (S112). If it is a marker event, the coordinates of the finger are acquired, and if it is a gesture event, the type of gesture is acquired. In the page creation / deletion event and file read event, the page number and file name are included in the operation content.

  The remote operation transmission unit 65 transmits operation data necessary for remote sharing processing to the server machine 90 (S113).

  The remote context receiving unit 78 of the server machine 90 receives the remote context and operation base identification information transmitted in step S111 (S114).

  The remote context transmission unit 77 transmits the remote context and operation base identification information to all bases registered in the participating base management table other than the base of operation (S115).

  The remote context receiving unit 78 of the receiving base receives the identification information of the remote context and the operating base (S116).

  The context processing unit 39 of the base that receives the remote context displays the remote context (S117). As will be described in detail later, an operation source name and a site color associated with the operation source identifier are acquired with reference to the participating site management table of the own site, and displayed on the display 3 together with the operation icon. Moreover, since the remote context will end its role if the user is notified, it is preferable that the remote context be automatically deleted so as not to obstruct the visual or drawing of the stroke. The context processing unit 39, for example, displays for 3 seconds (which is an example of the first predetermined time in the claims) and then fades out for 1 second (an example of the second predetermined time in the claims) ( Eliminate the remote context with an effect that gradually fades in brightness and color (and eventually disappears).

  On the other hand, the remote operation receiving unit 66 of the server machine receives the operation data transmitted in step S113 (S118).

  The remote operation transmission unit 65 transmits the operation data to all the sites registered in the participating site management table other than the operation source site (S119).

  The remote operation receiving unit 66 at the receiving base receives the operation data (S120).

  The image processing unit 30 at the receiving base reflects the operation data on the display 3 (S121). For example, a stroke or a marker is displayed, and a page is created by enlarging or reducing according to the gesture.

  As described above, the bases that receive the remote context in step S117 can know what operation is performed by the remote source and what kind of operation is performed by the remote context before receiving the operation data. It is possible to avoid drawing or performing contradictory operations.

  FIG. 21 shows an example of the remote context displayed on the display 3 by the site that has received the remote context. Although a total of eight remote contexts are displayed in FIG. 21, this is an explanatory diagram, and it is rare that eight remote contexts are displayed at the same time.

  In FIG. 21, icons and speech balloons are displayed as a set for one remote context. The icon is predetermined according to the remote context (event content). The base name is displayed in the balloon.

(1) Pressing of the shared icon The shared icon is an icon that the user presses to host a remote conference or participate in the conference, and is displayed on the upper right of the display 3 in FIG. When the user presses the shared icon 301, the context processing unit 39 of the base that receives the remote context displays the remote context. Specifically, a balloon in which the same icon as the shared icon and the site name are described is displayed near the shared icon 301. Note that “display near” means that the icon that is the same as the shared icon and the balloon that is not superimposed on the shared icon 301 are displayed closest to each other.

(2) Pressing the USB Memory Icon The USB memory icon 302 is an icon that the user presses to read a file from the USB memory, and is displayed on the right side of the display 3 in FIG. When the user presses the USB memory icon 302, the context processing unit 39 at the site that receives the remote context displays the remote context. Specifically, a balloon in which the same icon as the USB memory icon and the site name are described is displayed near the USB memory icon 302.

(3) Detection of Gesture Event When a gesture event is detected, the context processing unit 39 at the base that receives the remote context displays the remote context. Since the gesture event occurs at an arbitrary place, the context processing unit 39 displays an icon indicating that the gesture has been detected and a base name near the coordinates at which the gesture event has been detected. There are various types of gestures, and there is, for example, enlargement as described in the third embodiment. The icon indicating that a gesture has been detected preferably has a shape corresponding to the gesture. In the figure, an icon indicating that an enlargement gesture has been detected is displayed.

(4) Detection of pen-down event When a pen-down event is detected, the context processing unit 39 of the base that receives the remote context displays the remote context. Since the pen-down event occurs at an arbitrary location, the context processing unit 39 displays an icon and a base name indicating that a pen-down (writing has started) is detected near the coordinates where the pen-down event is detected. Near the coordinates where the pen down event is detected is near the start of the stroke. In the figure, an icon and a site name are displayed near the stroke “4”.

(5) Detection of marker event When a marker event is detected, the context processing unit 39 of the base that receives the remote context displays the remote context. Since the marker event occurs at an arbitrary location, the context processing unit 39 displays an icon indicating that the marker event has been detected and a base name near the coordinates at which the marker event has been detected. In the figure, an icon and a site name are displayed near the start point of the marker on the arc.

  The marker operated in Tokyo 1 is, for example, to indicate an alert that the user wants to pay attention to this part to a remote location (Fukuoka 1 or Osaka 2) where the user shares an image on the electronic blackboard 2. It is marked with a finger. When a user in a remote area (Fukuoka 1 or Osaka 2) does not know which part of the display 3 of the electronic blackboard 2 should be noticed, Tokyo 1 is paying attention to a part of the display 3 that is not intended There is. At this time, by indicating with a marker the part of Tokyo 1 that you want to pay attention to in remote areas (Fukuoka 1 and Osaka 2), it will lead to the promotion of smooth communication. Here, a specific application example of the marker has been described, but it is of course possible to use other than the marker.

(6) Page switching event detection A page thumbnail 304 is displayed at the bottom of the display 3. The user can switch the page displayed on the display 3 by pressing the thumbnail 304. This operation is a page switching event. When a page switching event is detected, the context processing unit 39 of the base that receives the remote context displays the remote context. Near the thumbnail 304, the context processing unit 39 displays an icon indicating that a page switching event has been detected (an icon having the same shape as the thumbnail) and a site name.

(7) Press of New Page Creation Icon The new page creation icon 303 is an icon that the user presses to create a new page, and is displayed on the right side of the display 3 in FIG. When the user presses the new page creation icon 303, the context processing unit 39 at the site that receives the remote context displays the remote context. Specifically, a balloon describing the same icon and base name as the new page creation icon 303 is displayed near the pressed new page creation icon 303.

(8) Pressing the capture icon The capture icon 305 is an icon that is pressed to capture an image displayed on the page, and is displayed at the bottom of the display 3 in FIG. When the user presses the capture icon 305, the context processing unit 39 of the base that receives the remote context displays the remote context. Specifically, a balloon in which the same icon and the name of the base as the capture icon 305 are described is displayed near the capture icon 305.

  Note that (1) to (8) are merely examples, and any operation performed on the electronic blackboard 2 can be displayed as a remote context.

  Like a stroke or marker drawing, an icon or the like for notifying the operation will be displayed near the place where the user operation will be reflected in the image, so the user will hesitate to draw at least the same place, Duplication can be suppressed. In addition, since it is possible to grasp in advance what operation the other party has started, it is possible to suppress contradictory operations.

  As described in the first embodiment, the contact sensor 115 can detect the coordinates of the shielding object that touches the screen regardless of whether it is an electronic pen or a finger. As in the first embodiment, when the drawing by the finger is assigned to the marker event, the operability of the image processing system 1 is improved if the operation content can be changed by using the electronic pen or the finger.

  In order to specify that the shielding object is an electronic pen, a pen down event of the electronic pen may be detected. However, since a mechanism such as a pen-down event cannot be used to specify that the shielding object is a finger, some kind of finger discrimination processing is necessary. Further, if it takes a long time for the finger discrimination process, the coordinates until the finger is discriminated are not drawn, and the user may feel uncomfortable.

  Therefore, in this embodiment, an image processing system 1 that performs finger discrimination processing without using a pen-down event and makes it difficult for the user to feel uncomfortable will be described. In the present embodiment, the constituent elements denoted by the same reference numerals in the first embodiment perform the same functions, and therefore only the main constituent elements of the present embodiment will be mainly described.

  FIG. 22A shows a schematic configuration diagram of the contact sensor 115. When the contact sensor 115 is an infrared ray blocking method, the two light receiving devices 115f and 115g capture that the light from the peripheral light emitting units 115a to 115d is blocked by the electronic pen 4 or the finger. Each of the light receiving devices 115f and 115g has a light receiving element arranged one-dimensionally in the horizontal direction, and the principle of triangulation depends on which light receiving element detects the light blockage in each of the two light receiving devices 115f and 115g. To detect the coordinates of the electronic pen 4 or the finger. Although there are four light receiving devices 115e to 115h in the figure, it is sufficient that there are at least two light receiving devices. In addition, the peripheral light emitting units 115a to 115d may be retroreflecting plates. In this case, the light receiving devices 115e to 115h have a light emitting function.

  FIG. 22B shows a side view of the contact sensor 115. Since the peripheral light emitting units 115a to 115d form a detection area from the front of the display 3, the light receiving devices 115e to 115h photograph the electronic pen before the electronic pen 4 contacts the display 3 (contact detection image event). . Thereafter, when the electronic pen 4 contacts the display 3, a pen down event is detected. Accordingly, there is a time delay from when the contact sensor 115 detects the light shielding until the pen down event is detected. The detection area is an example of a display surface in the claims.

  On the other hand, when the user touches the display 3 with his / her finger, the pen-down event is not detected even if time passes. In this embodiment, this time is used to detect a marker event. Also, by queuing the coordinates before the time elapses, the coordinates are drawn as markers from the beginning of writing with a finger.

  FIG. 23 is an example of a flowchart illustrating a procedure in which the finger determination unit 29-1 determines that drawing is performed with a finger and a marker is drawn.

  The event distribution unit 25 acquires a contact detection image event that the coordinate detection unit 22 has detected the coordinates, and notifies the finger determination unit 29-1 of the contact detection image event (S210).

  The finger determination unit 29-1 inquires the coordinate detection unit 22 about the thickness of the captured image and determines whether the thickness can be regarded as zero (S211). The thickness of the captured image is the width of the light-shielded area, and is determined by the number of light receiving elements that no longer detect light among the light receiving elements of the light receiving devices 115e to 115h. Since there are two light receiving devices, it may be determined based on the average number of light receiving elements of the two light receiving devices. By determining whether or not the thickness can be regarded as zero, it is possible to eliminate the case where the light shielding is detected due to noise and to detect that the finger is shielded.

  When the thickness cannot be regarded as zero (No in S211), the finger determination unit 29-1 determines whether the pen is down (S212). That is, whether or not a pen-down event is detected is detected, and if the pen is down, it can be determined that the electronic pen 4 is touching the display 3 instead of the finger. The pen down event is an example of contact information in the claims.

  If the pen is not down (No in S212), the finger determination unit 29-1 determines whether or not a gesture is being performed (S213). A gesture is an operation corresponding to enlargement / reduction, page turning, and the like, and “gesturing” means a process in which a gesture is detected. While the gesture processing unit detects the gesture, it is determined that the gesture is being performed.

  On the other hand, if the thickness can be regarded as zero (Yes in S211), the pen is down (Yes in S212), or the gesture is in progress (Yes in S213), whether or not the finger determination unit 29-1 is drawing the marker? Is determined (S214). Although it can be determined that there is a high possibility that an object other than the finger is shielded from light according to the determination results in steps S211 to S213, if the user is drawing a marker until just before, the coordinates drawn so far are used as the coordinates of the marker. The marker processing unit 38 transmits to the server machine 90 (S215). Thereby, the coordinates of all the markers can be shared.

  Returning to step S213, if the gesture is not being performed (No in S213), the finger determination unit 29-1 determines whether or not the finger input start waiting timer 29-2 is started (S216).

  If the finger input start standby timer 29-2 has not been started (No in S216), the finger determination unit 29-1 starts the finger input start standby timer 29-2 (S217).

  When the finger input start standby timer 29-2 is started (Yes in S216), the finger determination unit 29-1 adds the acquired coordinates to the end of the queue (S218).

  Next, the finger determination unit 29-1 determines whether or not the finger input start standby timer 29-2 has timed out (S219). The time-out time is, for example, several hundred milliseconds since the finger enters the detection area of the coordinate detection unit 22 until it touches the display 3. If not timed out, the processing from step S210 is repeated.

  When time-out occurs (Yes in S219), the finger determination unit 29-1 calculates the distance between the head of the queue and the last point, and whether or not the finger movement distance is greater than a threshold (for example, 50 [px]). Is determined (S219-2). By discriminating the moving distance in addition to the elapsed time, unnecessary marker display due to an operation with a small speed change such as writing error can be suppressed.

  When the movement distance of the finger is larger than the threshold (Yes in S219-2), the finger determination unit 29-1 determines that the drawing is performed by the finger, and a marker event is generated (S220).

  Next, the marker processing unit 38 determines whether there is coordinate data in the queue (S221).

  When there is coordinate data in the queue (Yes in S221), the marker processing unit 38 draws a marker by connecting the coordinates of the queue with a line from the top (S222). The line color is the base color registered in the participating base management table. This makes it possible to distinguish which base has drawn the marker at each base. Also, the marker line is thicker than the stroke, for example, 20px thick so that it can be distinguished from the stroke of the electronic pen. However, the transparency may be about 40% so that the background image can be visually recognized. Moreover, after drawing, it may be automatically deleted so as not to hinder the user's operation. In this case, for example, it is preferable to erase with an effect of fading out for 1 second after displaying for 3 seconds.

  When there is no coordinate data in the queue (No in S221), the marker processing unit 38 draws a marker by connecting the last coordinate and the acquired coordinate with a line (S223).

  Next, the marker processing unit 38 stores the acquired coordinates as the last coordinate (S224).

  Therefore, according to the present embodiment, each base transmits a drawing with a finger as a marker, so that the drawing content can be displayed in a different manner by a simple operation of using the finger instead of the electronic pen 4. For example, if the marker is drawn thicker than the stroke, the agenda to be emphasized can be easily shown. Moreover, since it is only necessary to draw with a finger instead of the electronic pen 4, it can be realized by a natural operation for the user.

  In the first and second embodiments, the gesture has been described. Conventionally, the enlargement / reduction gesture is effective only at the base where the gesture operation is performed, and is not shared with other bases.

  However, when an area that cannot be displayed on the display 3 occurs because the site A has expanded the page data, the area where the stroke or marker is written may not be displayed at the site A because the area is not enlarged at the site B. In addition to the strokes and markers, when the screen is enlarged, the remote context of the first embodiment may not be displayed depending on the display position.

  Therefore, in the present embodiment, an image processing system 1 that shares an enlargement / reduction operation using a gesture with each site will be described.

  FIG. 24 is an example of a functional block diagram of the electronic blackboard of the present embodiment. In the functional block diagram shown in FIG. 3, the finger determination unit 29-1, the finger input start standby timer 29-2, the marker processing unit 38, and the context processing unit 39 are not included.

  The gesture processing unit 27 creates an affine transformation matrix corresponding to the content of the gesture for the event determined to be a gesture operation by the event distribution unit 25. The affine transformation matrix can represent point scaling (enlargement / reduction), rotation, inversion, and translation. In this embodiment, an affine transformation matrix that performs enlargement / reduction and parallel movement is mainly created. The affine transformation matrix is an example of the operation information in the claims.

  FIG. 25A shows an example of an affine transformation matrix. The affine transformation matrix can be represented by a 3 × 3 matrix, and the upper left 2 × 2 matrix can be used for enlargement / reduction, and the lower left 1 × 2 matrix can be used for translation. In the example of FIG. 25A, the coordinates (x, y) are tripled in the x direction (y direction remains the same), and 3 in the x direction and 4 in the y direction are translated. Therefore, when the operation source site transmits the affine transformation matrix to other sites, the other sites can be similarly enlarged, reduced, and translated.

  FIG. 25B shows an example of movement when the affine transformation matrix is applied to the point (2, 1). The point (2, 1) is displayed at the point (9, 5), which matches the calculation result of the affine transformation matrix.

  The affine transformation matrix created by the gesture processing unit 27 is output to the image processing unit 30, and an image reflecting the gesture is displayed on the display 3. The affine transformation matrix is stored in the page data storage unit 300. The remote operation transmission unit 65 at the operation base transmits the affine transformation matrix to the remote operation reception unit 75 of the communication control unit 70, and the remote operation transmission unit 76 via the remote operation processing unit 83 transmits the affine transformation matrix of the client machine 20 at the other base. The affine transformation matrix is transmitted to the remote operation receiving unit 75. The image processing unit 30 at another base displays an image on which the gesture is reflected on the display 3.

  FIG. 26 is an example of a flowchart illustrating a procedure for sharing a gesture at two bases. In the following description, the operation source is A site, and the base that receives the gesture is B site.

  First, the gesture processing unit 27 at the A site detects a gesture and creates an affine transformation matrix (S310).

  Next, the page processing unit 37 of the image processing unit 30 performs an affine transformation on the coordinates displayed on the display 3, thereby displaying an image on which the gesture is reflected on the display 3 (S320). In order to transmit the affine transformation matrix, the affine transformation matrix is stored in the page data storage unit 300 in association with the page.

  The A site transmits the affine transformation matrix to the B site (S330). Specifically, the remote operation transmission unit 65 of the client machine 20 at the A site outputs the affine transformation matrix to the server machine 90. The remote operation receiving unit 75 of the server machine receives the affine transformation matrix and outputs it to the remote operation processing unit 83. The remote operation processing unit 83 reads the IP address of the B site registered in the participating site management table, and outputs the site identification information and the affine transformation matrix of the A site to the remote operation transmitting unit 76. Thereby, the remote operation transmitting unit 76 transmits the base identification information and the affine transformation matrix of the A base to the remote operation receiving unit 66 of the B base.

  The remote operation receiving unit 66 at site B receives the affine transformation matrix (S340). The affine transformation matrix is output to the image processing unit 30 at the B base, and the image processing unit 30 stores the affine transformation matrix in the page data storage unit 300.

  The page processing unit 37 at the site B does not immediately perform the affine transformation matrix on the page, but determines whether an event is detected at the site A (S350). This is because, when a stroke or marker is drawn at the A site, if the enlargement / reduction or translation is performed using the affine transformation matrix at the B site, it is necessary to change the display position of the coordinates input at the A site. is there.

Therefore, the page processing unit 37 determines whether or not the remote operation receiving unit 66 has received the following event. These are events that mainly change the display content of the page.
・ Pen-down event ・ Marker event ・ During gesture ・ Remote PC operation event Note that the remote PC operation means that the PC 6 is operated based on the coordinates detected by the coordinate detection unit in a state where the image of the PC 6 is displayed on the display 3. Say. That is, the user can operate the PC 6 by operating the display 3 (starting up an application, opening / closing a window, selecting a file, etc.). The pen-down event includes stroke deletion, stroke scaling, and stroke movement.

  When these events are detected (Yes in S350), the page processing unit 37 waits for 1 second (S360). One second is an example, and any time can be used as long as it can be detected that the event at the base A is interrupted.

  When the event is not detected (No in S350), the page processing unit 37 applies the affine transformation matrix stored in the page data storage unit 300 to the page and displays an image on which the gesture is reflected on the display 3 (S370). ).

  As described above, the image processing system according to the present embodiment can also share gestures with other bases, and users at each base can visually recognize the same place on the page. Also, by waiting for the event at the location where the gesture was performed, it is possible to perform enlargement / reduction at an appropriate timing.

1 Image processing system 2 Electronic blackboard (an example of an image processing device)
3 Display (an example of display means)
4 Electronic pen 5 USB memory 6 Notebook PC (an example of information processing terminal)
7 Video conference terminal 8 PC
DESCRIPTION OF SYMBOLS 9 Communication network 20 Client machine 30 Image processing part 60, 70 Communication control part 80 Data management part 90 Server machine

Japanese Patent No. 5081088

Claims (7)

An image processing system in which a plurality of information processing apparatuses including display means for displaying an image are connected via a network,
The first information processing apparatus
Operation accepting means for accepting an operation;
Operation content notification information creating means for creating operation content notification information for notifying the operation accepted by the operation accepting means;
First display means control means for displaying on the display means of the first information processing apparatus an image reflecting the operation content of the operation;
Transmission means for transmitting the operation content notification information and the operation content to a second information processing apparatus,
The second information processing apparatus
Displaying the operation content notification information transmitted from the first information processing apparatus on the display means of the second information processing apparatus;
An image processing system that displays an image reflecting the operation content on the display unit of the second information processing device based on the operation content transmitted from the first information processing device. The first information processing apparatus further includes coordinate detection means for detecting coordinates indicated by the instruction means for the display means,
The operation accepting means accepts an operation based on the coordinates detected by the coordinate detecting means,
A time measuring means for starting time measurement when the coordinate detecting means detects the indicating means;
Coordinate storage means for storing the coordinates detected by the coordinate detection means in time series;
Contact information receiving means for receiving contact information indicating that the instruction means has contacted the display means from the instruction means;
If the time measuring unit measures a time equal to or greater than a predetermined value before the contact information receiving unit receives the contact information, the coordinate storage unit stores the first or second display unit control unit. The display means by connecting the coordinates in a predetermined manner,
The image processing system according to claim 1. The operation content notification information includes identification information of the first information processing apparatus and coordinates detected by the coordinate detection unit when the operation reception unit receives an operation,
Displaying the identification information and the operation content notification information around the coordinates included in the operation content notification information of the display unit of the second information processing apparatus;
The image processing system according to claim 2. Displaying the identification information and the operation content notification information at a place predetermined according to the operation content notification information of the display means of the second information processing apparatus;
The image processing system according to claim 3.   The identification information and the operation content notification information displayed on the display unit of the second information processing apparatus are displayed for a first predetermined time, and then gradually deleted over a second predetermined time. The image processing system according to claim 3 or 4. A state storage unit in which communication state information indicating a communication state with the information processing device is registered in association with the identification information of the information processing device;
With reference to the state storage means, identification information of each information processing apparatus, and information processing apparatus list display means for displaying a communication state in a list form;
The image processing system according to claim 1, comprising: Operation accepting means for accepting an operation on the information processing apparatus;
Operation content notification information creating means for creating operation content notification information for notifying the operation;
A transmission unit that transmits the operation content of the operation to another information processing device after the operation content notification information is transmitted to the other information processing device; and the operation content notification information and the operation content are reflected An information processing apparatus for displaying an image on a display unit of the other information processing apparatus,
Display means control means for displaying on the display means of the information processing apparatus an image reflecting the operation content;
An information processing apparatus comprising:

Images