JP2016076775A - Image processing apparatus, image processing system, image processing method, and system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow a plurality of types of split stroke information, after an indicator enters a display screen until the indicator separates therefrom, to be handled as one unit of stroke information.SOLUTION: A first image processing section 30 includes a stroke processing section 32 for generating split stroke information continuously by splitting the stroke information in units of a constant time, starting at a moment in time when an electronic pen 4 enters a display screen, a communication control section 60 for transmitting the split stroke information, generated by the stroke processing section 32, to a second image processing section 30 while adding the continuation information indicating that the stroke is continuing, and a communication control section 60 for transmitting the split stroke information, generated by the stroke processing section 32, to the second image processing section while adding the end information indicating that the stroke has ended, when the electronic pen 4 separates from the display screen.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus, an image processing system, an image processing method, and a program suitable for sharing written stroke information.

An information sharing system using an electronic blackboard that allows a user to draw stroke images such as letters, numbers, and figures is already known with a remote place via a communication network such as the Internet.
In an information sharing system using an electronic blackboard, two methods are known as a method for controlling timing when transmitting stroke information generated by a transmitting terminal to a receiving terminal.
The first method is a method for collectively transmitting stroke information generated between the time when the electronic pen (indicator) enters the display screen of the electronic blackboard and the time when the electronic pen (indicator) leaves, and the second method is for a certain period of time. This is a method of sequentially transmitting stroke information when written within.
As an example of such a second method, Patent Document 1 discloses that stroke information displayed on a transmission-side terminal is transferred to a reception-side terminal after a certain period of time for the purpose of sharing stroke information between two terminals. A system is disclosed.

However, in the first conventional method, when it takes a long time until the electronic pen enters the display screen and then leaves on the transmission side terminal, the stroke information is not drawn on the reception side terminal. There is a problem that the time may be extended for a long time, and the user on the receiving side cannot visually recognize the change of the position of the stroke image for a long time.
On the other hand, in the second conventional method and Patent Document 1, since the transmitting side terminal sequentially transmits stroke information within a certain time, one unit of stroke information within a certain time is received. There is a problem that the receiving terminal cannot recognize the stroke from when the electronic pen enters the display screen until it leaves.
As a result, in the conventional receiving side terminal, the function operation such as the undo function and the deletion function can be performed for one unit of stroke information within a certain period of time, but the electronic pen enters the display screen and then leaves. When one piece of stroke information up to this is divided into fixed time units, the function operations such as the undo function and the delete function cannot be performed.
The present invention has been made in view of the above, and its purpose is to divide a single piece of stroke information from when the indicator enters the display screen to when it leaves and divides it into a plurality of fixed time units. It is possible to reproduce and handle the stroke information as one stroke information.

  In order to solve the above problem, the invention according to claim 1 is a display unit, an indicator that performs a writing operation by moving along the display screen of the display unit, and the instruction that moves along the display screen. Coordinate detection means for detecting each writing coordinate of the body, stroke information generation means for generating, as stroke information, coordinate information about the stroke by the writing operation based on the coordinate information of the coordinates detected by the coordinate detection means, A division generation unit that continuously generates divided stroke information by dividing the stroke information in units of a fixed time starting from the time point when the indicator enters the display screen, and the divided stroke information generated by the division generation unit First stroke transmitting means for adding continuation information indicating that the stroke is continuing and transmitting the information to another image processing apparatus; When the writing indicator leaves the display screen, second information is sent to the other image processing apparatus by adding end information indicating the end of the stroke to the divided stroke information generated by the divided generation means. Stroke transmission means.

  According to this aspect, even when one stroke information from when the indicator enters the display screen to when it leaves the display screen is divided into fixed time units, the divided stroke information is reproduced as one stroke information. Can be handled.

1 is an example of an overall configuration diagram of an image processing system. It is an example of the hardware block diagram of an electronic blackboard. It is an example of the functional block diagram of an electronic blackboard. It is an example of a functional block diagram of a file processing unit. It is an example of the functional block diagram of a communication control part. It is a conceptual diagram which shows the structure of page data. It is a conceptual diagram which shows the structure of stroke arrangement | sequence data. It is a conceptual diagram which shows the structure of coordinate arrangement | sequence data. It is a conceptual diagram which shows the structure of media data. It is a conceptual diagram for demonstrating the synthetic | combination process of each image layer. It is a figure for demonstrating the sequence which transmits stroke information after a pen approaches or contacts on a transmission side electronic blackboard until it leaves | separates. 6 is a flowchart for transmitting divided stroke information from when the electronic pen enters the display 3 to when it leaves the display-side electronic blackboard. 6 is a flowchart for receiving, combining, and drawing divided stroke information in a receiving-side electronic blackboard. It is a figure for demonstrating the data structure of the stroke management table of a page data storage part. It is a conceptual diagram which shows the structure of coordinate arrangement | sequence data. It is a figure for demonstrating the data structure of the division | segmentation stroke information communicated between a transmission side electronic blackboard and a reception side electronic blackboard. It is a conceptual diagram which shows the structure of coordinate arrangement | sequence data.

An embodiment of the present invention will be described. The present invention provides an electronic blackboard information sharing system in which stroke information can be displayed on the receiving terminal side within a certain time, and on the receiving terminal side, operations such as undoing and deleting strokes can be performed in one unit until the electronic pen is detached. In realization, it has the following features. In short, in order to transmit the divided stroke information to the terminal at a fixed time unit starting from the time when the electronic pen enters the display screen, and to combine the plurality of divided stroke information received at the receiving terminal side as one unit stroke, It is characterized in that the end flag of the divided stroke information is set to true / false and transmitted.
The features of the present invention described above will be described in detail with reference to the following drawings.

<System overview>
FIG. 1 is an overall configuration diagram of an image processing system according to the present embodiment. In FIG. 1, only two electronic blackboards 2a and 2b and electronic pens 4a and 4b associated with the electronic blackboards 2a and 2b are shown to simplify the description. A pen or the like may be used. The electronic blackboards 2a and 2b are examples of the information processing apparatus in the claims.

As shown in FIG. 1, the image processing system 1 includes a plurality of electronic blackboards 2a and 2b, a plurality of electronic pens 4a and 4b, USB memories 5a and 5b, notebook PCs (Personal Computer) 6a and 6b, a television ( Video) having conference terminals 7a, 7b and PC8. The electronic blackboards 2a and 2b and the PC 8 are connected via a communication network 9 so that they can communicate with each other. Further, the plurality of electronic blackboards 2a and 2b are provided with displays 3a and 3b, respectively.
Further, on the electronic blackboard 2a, an image drawn by an event (touch of the pen tip of the electronic pen 4a or the pen tip of the electronic pen 4a on the display 3a) generated by the electronic pen 4a is displayed on the display 3a. Can do. Note that the image displayed on the display 3a can be changed based not only on the electronic pen 4a but also on an event (a gesture such as enlargement, reduction, or page turning) generated by the user's hand Ha or the like.
The electronic pen 4 (indicator) performs a writing operation by moving along the display screens of the displays 3a and 3b.

  The electronic blackboard 2a can be connected to a USB memory 5a. The electronic blackboard 2a reads an electronic file such as PDF from the USB memory 5a, and the electronic blackboard 2a records an electronic file in the USB memory 5a. Can do. The electronic blackboard 2a is connected to a display port, DVI (Digital Visual Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), and a cable 10a1 capable of communication according to a standard such as VGA (Video Graphics Array). A notebook PC 6a is connected. The electronic blackboard 2a generates an event by touching the display 3a, and transmits event information indicating the event to the notebook PC 6a in the same manner as an event of an input device such as a mouse or a keyboard. Similarly, a television (video) conference terminal 7a is connected to the electronic blackboard 2a via a cable 10a2 capable of communication according to the above-mentioned standards. Note that the notebook PC 6a and the video conference terminal 7a may communicate with the electronic blackboard 2a by wireless communication complying with various wireless communication protocols such as Bluetooth (registered trademark).

  On the other hand, in the other bases where the electronic blackboard 2b is installed, the electronic blackboard 2b with the display 3b, the electronic pen 4b, the USB memory 5b, the notebook PC 6b, the video conference terminal 7b, the cable 10b1, and the cable 10b2 in the same manner as described above. Is used. Furthermore, the image displayed on the display 3b can be changed based on an event caused by the user's hand Hb or the like.

  Thereby, the image drawn on the display 3a of the electronic blackboard 2a at one base is also displayed on the display 3b of the electronic blackboard 2b at another base, and conversely on the display 3b of the electronic blackboard 2b at another base. The image drawn on is displayed on the display 3a of the electronic blackboard 2a at one base. As described above, the image processing system 1 can perform remote sharing processing for sharing the same image at a remote location, and thus is very convenient when used for a conference or the like at a remote location.

In the following, when an arbitrary electronic blackboard among the plurality of electronic blackboards is indicated, it is indicated as “electronic blackboard 2”. When an arbitrary display among a plurality of displays is shown, “display 3” is shown. When an arbitrary electronic pen among a plurality of electronic pens is indicated, it is indicated as “electronic pen 4”.
When an arbitrary USB memory is indicated among the plurality of USB memories, it is indicated as “USB memory 5”. When an arbitrary notebook PC is indicated among the plurality of notebook PCs, it is indicated as “node PC 6”. When an arbitrary video conference terminal is indicated among the plurality of video conference terminals, it is indicated as “video conference terminal 7”. In addition, “hand H” is used when an arbitrary hand among the hands of a plurality of users is shown. When an arbitrary cable is indicated among the plurality of cables, it is indicated as “cable 10”.

  In the present embodiment, an electronic blackboard is described as an example of an image processing device. However, the present invention is not limited to this, and other examples of the image processing device include electronic signboards (digital signage), sports and weather forecasts. It may be a telestrator to be used, a remote image (video) diagnostic device, or the like. The notebook PC 6 will be described as an example of the information processing terminal, but is not limited thereto, and other examples of the information processing terminal include a desktop PC, a tablet PC, a smartphone, a PDA, a digital video camera, and a digital camera. A terminal capable of supplying image frames such as a game machine may be used. Further, the communication network includes the Internet, a LAN (Local Area Network), a mobile phone communication network, and the like. In this embodiment, a USB memory is described as an example of a recording medium. However, the present invention is not limited to this, and various recording media such as an SD card may be used as another example of the recording medium.

<Hardware configuration of electronic blackboard>
Next, the hardware configuration of the electronic blackboard according to the present embodiment will be described with reference to FIG. FIG. 2 is a hardware configuration diagram of the electronic blackboard.

  As shown in FIG. 2, the electronic blackboard 2 includes a CPU 101 that controls the operation of the electronic blackboard 2 as a whole, a ROM 102 that stores a program used to drive the CPU 101 such as an IPL, and a RAM 103 that is used as a work area of the CPU 101. , An SSD 104 for storing various data such as a program for the electronic blackboard 2, a network controller 105 for controlling communication with the communication network 9, and an external storage controller 106 for controlling communication with the USB memory 5.

  The electronic blackboard 2 also includes a capture device 111 that captures still images or moving images displayed on the display of the notebook PC 6, a GPU (Graphics Processing Unit) 112 that specializes in graphics, and an output image from the GPU on the display 3 or A display controller 113 that controls and manages screen display for output to the video conference terminal 7 is provided.

  The electronic blackboard 2 further includes a sensor controller 114 that controls processing of the contact sensor 115 and a contact sensor 115 that detects that the electronic pen 4 or the hand H of the user touches the display 3. The contact sensor 115 performs coordinate input and coordinate detection by an infrared ray blocking method. In this coordinate input and coordinate detection method, two light emitting / receiving devices (not shown) installed at both upper ends of the display 3 emit a plurality of infrared rays in parallel with the display 3, This is a method of receiving light that is reflected by the reflecting member provided on the light source and returns on the same optical path as the light path of the light emitted by the light receiving element. The contact sensor 115 outputs infrared IDs (Identification) emitted from the two light receiving and emitting devices blocked by the object to the sensor controller 114, and the sensor controller 114 specifies the coordinate position that is the contact position of the object. All IDs shown below are examples of identification information.

  In addition, the contact sensor 115 is not limited to the infrared blocking method, and a capacitive touch panel that identifies a contact position by detecting a change in capacitance, or a contact position determined by a voltage change between two opposing resistance films. Various detection means such as a resistance film type touch panel to be identified and an electromagnetic induction type touch panel to identify a contact position by detecting electromagnetic induction caused by contact of a contact object with the display unit may be used.

  The electronic blackboard 2 includes an electronic pen controller 116. This electronic pen controller 116 communicates with the electronic pen 4 to determine whether or not there is a pen tip touch or a pen butt touch on the display 3. It should be noted that the electronic pen controller 116 may determine whether or not the user grips the part of the electronic pen 4 as well as the other electronic pen part, as well as the pen tip and the bottom of the electronic pen 4.

  Further, the electronic blackboard 2 includes the CPU 101, ROM 102, RAM 103, SSD 104, network controller 105, external storage controller 106, capture device 111, GPU 112, sensor controller 114, and electronic pen controller 116 as shown in FIG. Are provided with a bus line 120 such as an address bus and a data bus.

  The program for the electronic blackboard 2 may be recorded and distributed on a computer-readable recording medium such as a CD-ROM.

<Functional configuration of electronic blackboard>
Next, the functional configuration of the electronic blackboard will be described with reference to FIGS. First, the overall functional configuration of the electronic blackboard 2 will be described with reference to FIG. FIG. 3 is a functional block diagram of the electronic blackboard.

The electronic blackboard 2 has each functional configuration shown in FIG. 3 by the hardware configuration and program shown in FIG. The electronic blackboard 2 can be a “host device” that starts the remote sharing process first, and can also be a “participating device” that participates in the remote sharing process that has already started. The electronic blackboard 2 includes a client machine 20 and a server machine 90 as main functions. That is, the client machine 20 and the server machine 90 are included in one casing of the electronic blackboard 2.
The client unit 20 and the server unit 90 are functions realized in one housing of the electronic blackboard 2. When the electronic blackboard 2 is a hosting device, the electronic blackboard 2 implements the client unit 20 and the server unit 90.
Further, when the electronic blackboard 2 is a participating device, in the electronic blackboard 2, the client unit 20 is realized, but the server unit 90 is not realized. That is, in FIG. 1, when the electronic blackboard 2a is the hosting device and the electronic blackboard 2b is the participating device, the client unit 20 of the electronic blackboard 2a receives the other via the server unit 90 realized in the same electronic blackboard 2a. Communicate with the client unit 20 of the electronic blackboard 2b. On the other hand, the client unit 20 of the electronic blackboard 2b communicates with the client units of the other electronic blackboard 2a via the server unit 90 realized in the other electronic blackboard 2a.

[Functional configuration of client machine 20]
Subsequently, the functional configuration of the client machine 20 will be described mainly using FIGS. 3 to 5. The client machine 20 includes a video acquisition unit 21, a coordinate detection unit 22, an automatic adjustment unit 23, a contact detection unit 24, an event distribution unit 25, an operation processing unit 26, a gesture processing unit 27, a video superimposition unit 28, and a finger determination unit 29. -1, a finger input start standby timer 29-2, an image processing unit 30, and a communication control unit 60.

  Among these, the video acquisition unit 21 acquires an output video of a video output device such as a notebook PC 6 connected to the cable 10. When the video acquisition unit 21 receives an image signal from a video output device (such as a notebook PC 6), the video acquisition unit 21 analyzes the image signal, and the resolution of an image frame that is a display image of the video output device formed by the image signal, Image information such as the update frequency of the image frame is derived and output to the image acquisition unit 31.

  The coordinate detection unit 22 detects the coordinate position of an event (such as an operation in which the user's hand H is touched on the display 3) generated by the user on the display 3. The coordinate detection unit 22 also detects the touched area. The coordinate detection unit 22 detects each writing coordinate of the electronic pen 4 that moves along the display screen of the display 3 and outputs coordinate information.

  The automatic adjustment unit 23 is activated when the electronic blackboard 2 is activated, and adjusts the image processing parameters of the sensor camera in the optical sensor method of the contact sensor 115 so that the contact sensor 115 can output an appropriate value to the coordinate detection unit 22. To do.

  The contact detection unit 24 detects an event (such as an operation in which the pen tip of the electronic pen 4 or the pen butt of the electronic pen 4 is pressed (touched) on the display 3).

  The event distribution unit 25 distributes the coordinate position of the event detected by the coordinate detection unit 22 and the detection result detected by the contact detection unit 24 to each event of stroke drawing, UI operation, and gesture operation.

FIG. 10 is a conceptual diagram for explaining the composition processing of each image layer.
Here, “stroke drawing” is performed when the user presses the electronic pen 4 on the display 3 when a stroke image (B) described later shown in FIG. 10 is displayed on the display 3. In this state, the electronic pen 4 is moved and finally the electronic pen 4 is released from the display 3. By this stroke drawing, for example, alphabets “S” and “T” are drawn on the display 3. The “stroke drawing” includes not only drawing an image but also an event of deleting an already drawn image or editing a drawn image.

  The “UI operation” is an event in which the user presses a predetermined position with the electronic pen 4 or the hand H when a UI image (A) (described later) shown in FIG. 10 is displayed on the display 3. . By this UI operation, for example, the color or width of a line drawn by the electronic pen 4 is set.

  The “gesture operation” is an event in which the user touches or moves the display 3 with the hand H when a stroke image (B) described later shown in FIG. 10 is displayed on the display 3. is there. By this gesture operation, for example, by moving the hand H while the user touches the display 3 with the hand H, the image can be enlarged (or reduced), the display area can be changed, or the page can be switched. it can.

  The operation processing unit 26 executes various operations according to the UI element in which the event is generated, from what is determined as the UI operation by the event distribution unit 25. Examples of the UI element include a button, a list, a check box, and a text box. The gesture processing unit 27 generates a gesture event by determining the content of the gesture from the history of coordinates and the like for the event determined to be a gesture operation by the event distribution unit 25.

  The finger determination unit 29-1 determines that the finger is touching the screen. Since the coordinate detection unit 22 detects coordinates using an infrared ray blocking method and triangulation, it is possible to detect contact with an indicator other than an electronic pen. For this reason, it is discriminate | determined by the finger discrimination | determination part 29-1 whether it is a contact of an electronic pen, or a contact by pointing devices other than an electronic pen. In addition, although a finger is described as an example of an instruction tool other than the electronic pen, infrared rays can be shielded without using a finger.

  The finger determination unit 29-1 acquires a touch detection image event and a pen down event from the event distribution unit 25 for finger determination. The contact detection image event is an event in which light shielding is detected in the detection area of the contact sensor 115 (an area parallel to the screen that is several cm or less from the screen). The pen down event is an event in which the touch detection unit 24 detects a touch of the pen tip of the electronic pen 4 or the pen butt of the electronic pen 4. The pen-up event is an event in which the contact detection unit 24 detects that the pen tip of the electronic pen 4 or the pen bottom of the electronic pen 4 has left the detection area. In addition, the finger determination unit 29-1 acquires an event during gesture from the gesture processing unit.

  The finger determination unit 29-1 measures the time by the finger input start waiting timer 29-2 according to the presence / absence of a touch detection image event, a pen-down event, a gesture event, and the size of the image of the image event ( Count). Also, when the finger input start standby timer 29-2 starts counting, the finger determination unit 29-1 accumulates (queues) the coordinate values in time series.

  When the finger input start standby timer 29-2 counts a time equal to or longer than a predetermined time, the finger determination unit 29-1 determines that the finger is being drawn, and transmits the coordinates queued to the marker processing unit 38. . The marker processing unit 38 draws a series of coordinates as a marker, and thereafter continues drawing the marker using the acquired coordinates as the last coordinate of the marker.

  Accordingly, when the finger determination unit 29-1 determines that it is a finger, the marker processing unit 38 draws with a marker, and when it determines that it is an electronic pen, the stroke processing unit 32 draws a stroke with the pen. Determining that the finger determination unit 29-1 is a finger may be referred to as detecting a marker event.

  The video superimposing unit 28 displays the image superimposed by the display superimposing unit 36 described later as a video on the video output device (display 3). In addition, the video superimposing unit 28 performs picture-in-picture of the video sent from another video output device (such as the video conference terminal 7) with respect to the video from the video output device (such as the notebook PC 6). Further, the video superimposing unit 28 performs switching for displaying the video that is picture-in-picture and displayed on a part of the display 3 on the entire display 3.

  The image processing unit 30 performs superimposition processing of each image layer. A specific example of superposition will be described with reference to FIG. 10 as appropriate. The image processing unit 30 includes an image acquisition unit 31, a stroke processing unit 32, a UI image generation unit 33, a background generation unit 34, a layout management unit 35, a marker processing unit 38, a context processing unit 39, a display superimposing unit 36, and a page processing unit. 37, a file processing unit 40, a page data storage unit 300, and a remote license management table 310.

  Among these, the image acquisition unit 31 acquires each frame as an image from the video acquired by the video acquisition unit 21. This image corresponds to an output image (C) from the video output device (notebook PC 6 or the like) shown in FIG.

  The stroke processing unit 32 draws an image, deletes the drawn image, or edits the drawn image based on the event related to the stroke drawing assigned by the event distribution unit 25. The image by the stroke drawing corresponds to the stroke image (B) shown in FIG. In addition, the results of drawing, deleting, and editing an image based on the stroke and the like are stored in the operation data storage unit 840 as operation data described later.

  The UI image generation unit 33 generates a UI (user interface) image set in advance on the electronic blackboard 2. This UI image corresponds to the UI image (A) shown in FIG.

  The background generation unit 34 receives media data from the page data read from the page data storage unit 300 by the page processing unit 37 from the page processing unit 37. The background generation unit 34 outputs the received media data to the display superimposing unit 36. The image based on the media data corresponds to the background image (D) shown in FIG. The pattern of the background image (D) is plain or grid display.

  The layout management unit 35 provides the layout information indicating the layout of each image output from the image acquisition unit 31, the stroke processing unit 32, and the UI image generation unit 33 (or the background generation unit 34) to the display superimposing unit 36. Is managing. Thereby, the layout management unit 35 causes the display superimposing unit 36 to display the output image (C) and the stroke image (B) in the UI image (A) and the background image (D). It is possible to instruct whether to hide.

  The context processing unit 39 transmits a marker event, a gesture event, a page creation / deletion event, and a file read event to the server machine 90 via the remote context transmission unit 68 of the communication control unit 60. In addition, the context processing unit 39 acquires a marker event, a gesture event, a page creation / deletion event, and a file read event from the remote context receiving unit 67 of the communication control unit 60. Also, the other site participation event and the other site leaving event are acquired from the participating site management table 610 of the communication control unit 60. The context processing unit 39 generates a base image according to the acquired event and base and displays it on the display 3. Note that a page creation / deletion event occurs when a user performs a page creation or deletion operation on a UI image. The file read event occurs when the file input unit 42a reads a file. The other site participation event occurs when a new site is registered in the participating site management table 610. The other site exit event occurs when a site is deleted from the participating site management table 610.

  The display superimposing unit 36 is based on the layout information output from the layout management unit 35, and includes an image acquisition unit 31, a stroke processing unit 32, a UI image generation unit 33 (background generation unit 34), a marker processing unit 38, and a background generation unit. The layout of each image output from 34 is performed.

The page processing unit 37 stores the data of the stroke image (B) and the data of the output image (C) in one page data and stores them in the page data storage unit 300.
The data of the stroke image (B) forms part of page data as stroke arrangement data (each stroke data) indicated by the stroke arrangement data ID shown in FIG. The data of the output image (C) forms part of the page data as the media data indicated by the media data ID shown in FIG. Then, when this media data is read from the page data storage unit 300, it is handled as data of the background image (D).

  Further, the page processing unit 37 transmits the media data of the once stored page data to the display superimposing unit 36 via the background generating unit 34 so that the video superimposing unit 28 displays the background image (D). 3 can be displayed again. The page processing unit 37 can return the stroke arrangement data (each stroke data) in the page data to the stroke processing unit 32 so that the stroke can be re-edited. Further, the page processing unit 37 can delete or duplicate page data.

  That is, the data of the output image (C) displayed on the display 3 when the page processing unit 37 stores the page data in the page data storage unit 300 is temporarily stored in the page data storage unit 300 and then When read from the page data storage unit 300, it is read as media data indicating the background image (D). Then, the page processing unit 37 outputs stroke arrangement data indicating the stroke image (B) among the page data read from the page data storage unit 300 to the stroke processing unit 32. Further, the page processing unit 37 outputs media data indicating the background image (D) among the page data read from the page data storage unit 300 to the background generation unit 34.

  The display superimposing unit 36 includes an output image (C) from the image acquisition unit 31, a stroke image (B) from the stroke processing unit 32, a UI image (A) from the UI image generation unit 33, and a background generation unit 34. The background image (D) is superimposed according to the layout designated by the layout management unit 35. As a result, as shown in FIG. 10, each of the UI image (A), the stroke image (B), the output image (C), and the background image (D) is displayed in the order in which the user can see even if the images overlap. It has Leia composition.

  Further, the display superimposing unit 36 can also superimpose the image (A) and the image (B) exclusively by switching the image (C) and the image (D) shown in FIG. For example, when the cable 10 between the electronic blackboard 2 and the video output device (notebook PC 6 or the like) is disconnected while the image (A), the image (B), and the image (C) are initially displayed. According to the designation of the layout management unit 35, the drawing (C) can be removed from the superimposition target and the image (D) can be displayed. The display superimposing unit 36 also performs display enlargement, display reduction, and display area movement processing.

The page data storage unit 300 stores page data as shown in FIG. FIG. 6 is a conceptual diagram showing the configuration of page data. The page data is data for one page displayed on the display 3 (stroke arrangement data (each stroke data) and media data). Since there are many types of parameters included in the page data, the contents of the page data will be described here separately in FIGS.
The page data storage unit 300 includes a divided stroke management table 300a for managing / saving divided stroke information and a stroke management table 300b for managing / saving stroke information.

  As shown in FIG. 6, the page data includes a page data ID for identifying an arbitrary page, a disclosure time indicating the start time of display of this page, and rewriting of the contents of the page by a stroke, a gesture, or the like. Are associated with the end time indicating the time when the operation is stopped, the stroke arrangement data ID for identifying the stroke arrangement data generated by the stroke of the electronic pen 4 or the user's hand H, and the media data ID for identifying the media data. Is remembered. The stroke arrangement data is data for displaying a stroke image (B) shown in FIG. The media data is data for displaying a background image (D) shown in 16 described later on the display 3.

  With such page data, for example, when the user draws the alphabet “S” with the electronic pen 4, the stroke is written in one stroke, and therefore, one stroke data ID is displayed and one letter of the alphabet [S] is indicated. However, when the user draws the alphabet “T” with the electronic pen 4, it is written with two strokes, so that one stroke of the alphabet “T” is indicated by two stroke data IDs.

  The stroke arrangement data shows detailed information as shown in FIG. FIG. 7 is a conceptual diagram showing a configuration of stroke arrangement data. As shown in FIG. 7, one stroke arrangement data is represented by a plurality of stroke data. One stroke data includes a stroke data ID for identifying the stroke data, a start time indicating the start time of writing one stroke, an end time indicating the end time of writing one stroke, the color of the stroke, A stroke width and coordinate array data ID for identifying an array of stroke passing points are shown.

Further, the coordinate array data shows detailed information as shown in FIG. FIG. 8 is a conceptual diagram showing the configuration of the coordinate array data. As shown in FIG. 8, the coordinate array data includes one point (X coordinate value, Y coordinate value) on the display 3, and a time difference (ms from the disclosure time of the stroke when the one point is passed) ) And writing pressure information of the electronic pen 4 at this one point.
That is, the collection of one point shown in FIG. 8 is shown by one coordinate array data shown in FIG. For example, when the user draws the alphabet “S” with the electronic pen 4, it is drawn with a single stroke, but passes through a plurality of passing points before the drawing of “S”. Point information is shown.

  Further, the media data in the page data shown in FIG. 6 shows detailed information as shown in FIG. FIG. 9 is a conceptual diagram showing the configuration of media data. As shown in FIG. 9, the media data is stored in the page data storage unit 300 from the page processing unit 37, the media data ID in the page data shown in FIG. The recording time, the position of the image displayed on the display 3 by the page data (X coordinate value, Y coordinate value), the size of the image (width, height), and data indicating the contents of the media data are shown in association with each other. Has been. Among these, the position of the image displayed on the display 3 by the page data is displayed by the page data when the coordinates of the upper left corner of the display 3 are (X coordinate value, Y coordinate value) = (0, 0). The position of the upper left corner of the image to be displayed is shown.

  Returning to FIG. 3, the remote license management table 310 manages license data necessary for executing the remote sharing process. In the remote license management table 310, the product ID of the electronic blackboard 2, the license ID used for authentication, and the license expiration date are managed in association with each other.

(Functional configuration of the file processing unit 40)
Next, the functional configuration of the file processing unit 40 shown in FIG. 3 will be described with reference to FIG. FIG. 4 is a functional block diagram of the file processing unit. Hereinafter, the electronic blackboard 2 that starts the remote sharing process first will be referred to as “host device”, and the electronic blackboard 2 that will participate in the remote sharing processing that has already been started will be referred to as “participating device”.

The file processing unit 40 includes a recovery processing unit 41, a file input unit 42a, a file output unit 42b, a file conversion unit 43, a file transmission unit 44, an address book input unit 45, a backup processing unit 46, a backup output unit 47, and a setting management unit. 48, a setting file input unit 49a, and a setting file output unit 49b. Furthermore, the file processing unit 40 includes an address book management table 410, a backup data storage unit 420, a setting file storage unit 430, and a connection destination management table 440.
Among these, the recovery processing unit 41 detects the abnormal end after the electronic blackboard 2 ends abnormally, and recovers the unsaved page data. For example, in the case of a normal end, page data is recorded as a PDF file in the USB memory 5 via the file processing unit 40, but in the case of an abnormal end such as when the power is down, the page data is stored in the page data storage unit 300. Is still recorded. Therefore, when the power is turned on again, the recovery processing unit 41 recovers by reading the page data from the page data storage unit 300.

The file input unit 42a reads a PDF file from the USB memory 5 and stores each page in the page data storage unit 300 as page data.
The file conversion unit 43 converts the page data stored in the page data storage unit 300 into a PDF format file.
The file output unit 42 b records the PDF file output by the file conversion unit 43 in the USB memory 5.

The file transmission unit 44 transmits the PDF file generated by the file conversion unit 43 by attaching it to an electronic mail. The transmission destination of the file is displayed by displaying the contents of the address book management table 410 on the display 3 by the display superimposing unit 36, and the file transmitting unit 44 accepts the selection of the destination by the operation of the input device such as a touch panel by the user. It is determined.
The address book input unit 45 reads a list file of e-mail addresses from the USB memory 5 and manages it in the address book management table 410.
The backup processing unit 46 performs backup by storing the file output by the file output unit 42 b and the file transmitted by the file transmission unit 44 in the backup data storage unit 420. If the user does not set backup, backup processing is not performed.
The backup output unit 47 stores the backed up file in the USB memory 5. At the time of storage, a password is input for security by the operation of an input device such as a touch panel by the user.

The setting management unit 48 stores and reads various setting information of the electronic blackboard 2 in the setting file storage unit 430 for management. Examples of the various setting information include network settings, date and time settings, region and language settings, mail server settings, address book settings, connection destination list settings, and backup settings. The network setting is, for example, an IP address setting of the electronic blackboard 2, a net mask setting, a default gateway setting, a DNS (Domain Name System) setting, or the like.
The setting file output unit 49b records various setting information of the electronic blackboard 2 in the USB memory 5 as a setting file. Note that the settings file cannot be viewed by the user due to security.
The setting file input unit 49a reads a setting file stored in the USB memory 5 and reflects various setting information on various settings of the electronic blackboard.
The address book input unit 50 reads a list file of connection destination IP addresses for remote sharing processing from the USB memory 5 and manages it in the connection destination management table 440.

(Functional configuration of communication control unit 60)
Next, the functional configuration of the communication control unit 60 will be described with reference to FIG. FIG. 5 is a functional block diagram of the communication control unit 60. The communication control unit 60 controls communication with other electronic blackboards 2 and communication with the later-described communication control unit 70 in the server machine 90 via the communication network 9. Therefore, the communication control unit 60 includes a remote start processing unit 61, a remote participation processing unit 62, a remote image transmission unit 63, a remote image reception unit 64, a remote operation transmission unit 65, a remote operation reception unit 66, a remote context transmission unit 68, A remote context receiving unit 67 and a participating site management table 610 are provided.
Among these, the remote start processing unit 61 makes a request for starting a new remote sharing process to the server machine 90 of the same electronic blackboard 2, and receives the request result from the server machine 90. In this case, the remote start processing unit 61 refers to the remote license management table 310 and makes a request to start remote sharing processing when license information (product ID, license ID, and expiration date) is managed. be able to. However, if the license information is not managed, a request to start remote sharing processing cannot be made.

The participation base management table 610 is a table for managing an electronic blackboard as a participating device currently participating in the remote sharing process when the electronic blackboard is a sponsor device.
The remote participation processing unit 62 transmits the remote connection processing to the remote sharing processing to the remote connection request receiving unit 71 in the server 90 of the electronic blackboard 2 as the hosting device that has already started the remote sharing processing via the communication network 9. Request participation. Also in this case, the remote participation processing unit 62 refers to the remote license management table 310. In addition, when the remote participation processing unit 62 participates in the remote sharing process that has already been started, the IP address of the electronic blackboard 2 of the participation destination is acquired with reference to the connection destination management table 440. Note that the remote participation processing unit 62 may not input the connection destination management table, and may input the IP address of the electronic blackboard 2 as a participation destination by an operation of an input device such as a touch panel by the user.

The remote image transmission unit 63 transmits the output image (C) sent from the video acquisition unit 21 via the image acquisition unit 31 to the server machine 90.
The remote image receiving unit 64 receives image data from a video output device connected to another electronic blackboard 2 from the server machine 90 and outputs it to the display superimposing unit 36, thereby enabling remote sharing processing.
The remote operation transmission unit 65 transmits various operation data necessary for remote sharing processing to the server machine 90. Examples of the operation data include adding a stroke, deleting a stroke, editing a stroke (enlarging, reducing, moving), storing page data, creating page data, duplicating page data, deleting page data, and displaying. Data related to the switching of the current page. In addition, the remote operation receiving unit 66 receives the operation data input from the other electronic blackboard 2 from the server machine 90 and outputs it to the image processing unit 30 to perform remote sharing processing.
The remote context transmission unit 68 acquires a marker event, a gesture event, a page creation / deletion event, and a file read event from the context processing unit 39 and transmits them to the server machine 90 as a remote context.
The remote context receiving unit 67 acquires the event information generated on the other electronic blackboard from the server machine 90, and the context processing unit 39 of the image processing unit 30 displays the base image.

[Functional configuration of server machine]
Next, the functional configuration of the server machine 90 will be described with reference to FIG. The server machine 90 is provided in each electronic blackboard 2, and any electronic blackboard 2 can serve as a server machine. Therefore, the server machine 90 includes a communication control unit 70 and a data management unit 80.

(Functional configuration of communication control unit 70)
Next, the functional configuration of the communication control unit 70 will be described with reference to FIG.
The communication control unit 70 controls communication with the communication control unit 60 in the client machine 20 in the same electronic blackboard 2 and the communication control unit 70 in the client machine 20 in another electronic blackboard 2 via the communication network 9. . The data management unit 80 manages operation data, image data, and the like.
More specifically, the communication controller 70 includes a remote connection request receiver 71, a remote connection result transmitter 72, a remote image receiver 73, a remote image transmitter 74, a remote operation receiver 75, a remote operation transmitter 76, The remote context receiving unit 78 and the remote context transmitting unit 77 are provided.
Among these, the remote connection request receiving unit 71 receives a remote sharing processing start request from the remote start processing unit 61 or receives a participation request for remote sharing processing from the remote participation processing unit 62. The remote connection result transmission unit 72 transmits the result of the remote sharing processing start request to the remote start processing unit 61, or transmits the result of the participation request for the remote sharing processing to the remote participation processing unit 62.

The remote image receiving unit 73 receives the image data (data of the output image (C)) from the remote image transmitting unit 63 and transmits it to the remote image processing unit 82 described later. The remote image transmission unit 74 receives image data from the remote image processing unit 82 and transmits this image data to the remote image reception unit 64.
The remote operation receiving unit 75 receives operation data (data such as a stroke image (B)) from the remote operation transmitting unit 65 and transmits it to the remote operation processing unit 83 described later. The remote operation transmission unit 76 receives operation data from the remote operation processing unit 83 and transmits this operation data to the remote operation reception unit 66.
The remote context receiver 78 receives the remote context from the client machine and outputs it to the remote operation processor 83. The remote operation processing unit 83 adds identification information (for example, an IP address) of the local site and outputs it to the remote context transmission unit 77. The remote context transmission unit 77 transmits the remote context and identification information of its own base to each client machine.

(Functional configuration of the data management unit)
Next, the functional configuration of the data management unit 80 will be described with reference to FIG. The data management unit 80 includes a remote connection processing unit 81, a remote image processing unit 82, a remote operation processing unit 83, an operation composition processing unit 84, and a page processing unit 85. Further, the server 90 includes a passcode management unit 810, a participating site management table 820, an image data management unit 830, an operation data storage unit 840, and a page data storage unit 850.
Among these, the remote connection processing unit 81 starts the remote sharing process and ends the remote sharing process. The remote connection processing unit 81 also receives the license information received by the remote connection request receiving unit 71 from the remote start processing unit 61 together with the remote sharing processing start request or the remote sharing processing unit 62 from the remote sharing processing participation request. Based on the received license information, it is confirmed whether there is a license and whether it is within the license period. Further, the remote connection processing unit 81 confirms whether or not the participation request from the other electronic blackboard 2 as the client machine exceeds the predetermined number of possible participations.
Further, the remote connection processing unit 81 has the same passcode that is sent when another electronic blackboard 2 has requested to participate in remote sharing processing as the passcode managed by the passcode management unit 810. It is determined whether or not there is the same, and if they are the same, participation in the remote sharing process is permitted. This passcode is issued by the remote connection processing unit 81 when a new remote sharing process is started, and the user of the electronic blackboard 2 as a participating apparatus who intends to participate in the remote sharing process can It is transmitted from the user of the electronic blackboard 2 by telephone or e-mail. As a result, the user of the electronic blackboard 2 as a participating device who intends to participate in the remote sharing process inputs the pass code to the electronic blackboard 2 and requests participation, thereby permitting participation. Note that the user's usability is given priority over security, and the confirmation of the passcode may be omitted only by confirming the license status.

  In addition, the remote connection processing unit 81 uses the participation site information included in the participation request sent from the remote participation processing unit 62 of the electronic blackboard 2 as the participation device via the communication network 9 to the participation site of the server 90. Store in the management table 820. Then, the remote connection processing unit 81 reads out the remote site information stored in the participating site management table 820 and transmits it to the remote connection result transmission unit 72. The remote connection result transmission unit 72 transmits the remote base information to the remote start processing unit 61 in the client unit 20 of the same sponsor device. The remote start processing unit 61 stores the remote site information in 610 in the participating site management table. As a result, the host device manages remote site information in both the client unit 20 and the server unit 90.

The remote image processing unit 82 receives image data from a video output device (node PC 6 or the like) connected to a client device of each electronic blackboard 2 (including a client device of its own electronic blackboard that is the hosting device) during remote sharing processing. (Output image (C)) is received and stored in the image data management unit 830, and the display order of the image data to be remotely shared in the order of time of arrival at the server device 90 of the electronic blackboard 2 that is the host device Determine. In addition, the remote image processing unit 82 refers to the participation base management table 820, and adds to all the client machines 20 of the electronic blackboard 2 participating in the remote sharing process (including the client machine of its own electronic blackboard that is the host device). The image data is transmitted in the determined order via the communication control unit 70 (remote image transmission unit 74).
The remote operation processing unit 83 performs various types of operation data (stroke images (B, B, B) drawn on the client machine of each electronic blackboard 2 (including the client machine of its own electronic blackboard which is the hosting apparatus) during the remote sharing process. ) And the like, and the display order of the images to be remotely shared is determined in the order of time of arrival at the server 90 of the electronic blackboard 2 that is the host device. The various operation data is the same as the various operation data described above. Further, the remote operation processing unit 83 refers to the participating site management table 820 and operates the operation data on all the client machines 20 of the electronic blackboard 2 (including the client machine of its own electronic blackboard which is the hosting apparatus) during the remote sharing process. Send.

The operation synthesis processing unit 84 synthesizes the operation data of each electronic blackboard 2 output from the remote operation processing unit 83, stores the operation data as a result of this synthesis in the operation data storage unit 840 and the remote operation processing unit 83. Return to. The operation data is transmitted from the remote operation transmission unit 76 to each of the electronic blackboard client machine that is the hosting apparatus and the electronic blackboard client machine that is the participating apparatus. Such an image is displayed.
The operation composition processing unit 84 performs composition in the order in which the operation data is input to the operation composition processing unit 84. Therefore, if the communication network 9 is not congested, remote sharing is performed in the order of strokes by the users of the electronic blackboards 2. A stroke image (B) is displayed on the displays 3 of all the electronic blackboards 2 being processed.
The page processing unit 85 has the same function as the page processing unit 37 in the image processing unit 30 of the client machine 20, and the server machine 90 also converts the page data shown in FIGS. 6 to 8 into a page data storage unit. Store in 850. Note that the page data storage unit 850 has the same contents as the page data storage unit 300 in the image processing unit 30, and therefore the description thereof is omitted.

<Process or Operation of Embodiment>
Next, processing and operations in the present embodiment will be described with reference to FIGS.
FIG. 11 is a diagram for explaining a sequence for transmitting stroke information from when the electronic pen enters (approaches or contacts) to the transmitter-side electronic blackboard 2a until it leaves. In FIG. 11, step S1xx represents processing at the transmission-side electronic blackboard 2a, and step S2xx represents processing at the reception-side electronic blackboard 2b.
In the transmission-side electronic blackboard 2a, the event sorting unit 25 detects a pen-down event when the electronic pen 4 enters the display 3 and approaches or comes into contact (S101).
The stroke processing unit 32 of the image processing unit 30 provided on the transmission-side electronic blackboard 2a continuously divides the divided stroke information by dividing the stroke information by a certain time unit starting from the time when the electronic pen 4 enters the display screen. Generate. The transmission-side electronic blackboard 2a transmits the divided stroke information to the reception-side electronic blackboard 2b when a predetermined time has elapsed after the electronic pen 4 contacts the display 3 (S103).
Here, the stroke processing unit 32 adds continuation information indicating that the stroke is continuing to the generated divided stroke information, and transmits the continuation information to the image processing unit 30 of the reception-side electronic blackboard 2b.
As a result, from the transmission-side electronic blackboard 2a to the reception-side electronic blackboard 2b, as the divided stroke information, stroke data ID, start time, color, width, coordinate arrangement, and termination flag as continuation information indicating that the stroke is continuing (: False) is transmitted (S109).
On the other hand, the reception-side electronic blackboard 2b receives the divided stroke information transmitted from the transmission-side electronic blackboard 2a (S201).

Here, a process until the stroke image drawn on the transmission side electronic blackboard 2a is displayed on the display 3 of the reception side electronic blackboard 2b will be described.
First, in the transmission-side electronic blackboard 2a, the user draws a stroke image (B) on the electronic blackboard 2a using the electronic pen 4a.

  Next, as shown in FIG. 10, the display superimposing unit 36 of the electronic blackboard 2a performs a stroke image (B) on the UI image (A), the output image (C), and the background image (D). And the video superimposing unit 28 causes the superimposed images (A, B, C, D) to be displayed on the display 3a of the electronic blackboard 2a. Specifically, the stroke processing unit 32 of the electronic blackboard 2a receives the data of the stroke image (B) as the operation data from the coordinate detection unit 21 and the contact detection unit 24 via the event distribution unit 25, and performs display superimposition. It transmits to the part 36. Accordingly, the display superimposing unit 36 can superimpose the stroke image (B) on the UI image (A) and the output image (C), and the video superimposing unit 28 is placed on the display 3a of the electronic blackboard 2a. Superimposed images (A, B, C) can be displayed.

  Next, the image processing unit 30 including the stroke processing unit 32 of the electronic blackboard 2a transmits the stroke image (B) data to the remote operation transmission unit 65, so that the remote operation transmission unit 65 of the electronic blackboard 2a is connected to the communication network. 2, the data of the stroke image (B) is transmitted to the communication control unit 70 of the electronic blackboard 2b which is the host apparatus. Accordingly, the remote operation receiving unit 75 of the electronic blackboard 2b receives the stroke image (B) data and outputs it to the remote operation processing unit 83, so that the remote operation processing unit 83 sends the stroke image to the operation composition processing unit 84. The data of (B) is output. In this way, the data of the stroke image (B) drawn on the electronic blackboard 2a is sequentially transmitted to the remote operation processing unit 83 of the electronic blackboard 2b that is the hosting device every time it is drawn. The data of the stroke image (B) is data shown for each stroke data ID shown in FIG.

  Next, the electronic blackboard 2b, which is the hosting device, displays on the display 3b a superimposed image (A, B, C) including the data of the stroke image (B) sent from the electronic blackboard 2a. Specifically, the operation composition processing unit 84 of the electronic blackboard 2b synthesizes the data of the plurality of stroke images (B) sequentially sent via the remote operation processing unit 83 and stores them in the operation data storage unit 840. Store and return to the remote operation processing unit 83. As a result, the remote operation processing unit 83 outputs the combined stroke image (B) data received from the operation synthesis processing unit 84 to the remote operation transmission unit 76. The remote operation transmission unit 76 outputs the combined stroke image (B) data to the remote operation reception unit 66 in the client unit 20 of the electronic blackboard 2b, which is the same host device. The remote operation receiving unit 66 outputs the combined stroke image (B) data to the display superimposing unit 36 in the image processing unit 30. Therefore, the display superimposing unit 36 superimposes the combined stroke image (B) on the UI image (A) and the output image (C). Finally, the video superimposing unit 28 displays the superimposed image (A, B, C) superimposed by the display superimposing unit 36 on the display 3b.

Further, returning to FIG. 11, the transmission-side electronic blackboard 2a transmits the divided stroke information to the reception-side electronic blackboard 2b when a predetermined time has elapsed since the previous transmission of the divided stroke information (S103 ′).
As a result, the coordinate array and the termination flag (: false) are transmitted as the end information indicating the end of the stroke as the divided stroke information from the transmission-side electronic blackboard 2a to the reception-side electronic blackboard 2b (S109 ′).
On the other hand, the reception-side electronic blackboard 2b receives the divided stroke information transmitted from the transmission-side electronic blackboard 2a (S201 ′).
In the transmission-side electronic blackboard 2a, the event distribution unit 25 detects a pen-up event when the electronic pen 4 is detached from the display 3 (S102).
When the electronic pen 4 is detached from the display 3, the transmission-side electronic blackboard 2a transmits the divided stroke information to the reception-side electronic blackboard 2b (S109 "). As a result, the transmission-side electronic blackboard 2a transmits to the reception-side electronic blackboard 2b. In addition, the end flag (: true) is transmitted as the coordinate detection end information indicating that the end time, the coordinate arrangement, and the coordinate detection are ended as the divided stroke information.
On the other hand, the reception-side electronic blackboard 2b receives the divided stroke information transmitted from the transmission-side electronic blackboard 2a (S201 ′).
Therefore, for example, as described above, when the user draws the alphabet “T” with the electronic pen 4, the stroke image (B) indicated by each of the two stroke data IDs is sequentially transmitted. Is done. At this time, each stroke image is divided and transmitted sequentially every time a fixed time has elapsed.

FIG. 12 is a flowchart for transmitting the divided stroke information from when the electronic pen enters the display 3 to when it leaves the display side electronic blackboard 2a.
When the electronic pen 4 approaches or comes into contact with the display 3, a pen down event is detected by the event distribution unit 25, and the pen down event is output to the stroke processing unit 32 (S101). At this time, timing by a timer is started.
Next, when the electronic pen 4 is detached from the display 3, a pen-up event is detected by the event distribution unit 25, and the pen-up event is output to the stroke processing unit 32. It is determined whether or not it has been done (S102).
When the pen up event is not input (S102, NO), the stroke processing unit 32 proceeds to step S103. On the other hand, when a pen-up event is input (S102, YES), the process proceeds to step S104.
The stroke processing unit 32 determines whether or not a certain time has elapsed since the contact detection in step S101 (step S103). The stroke processing unit 32 proceeds to step S120 when a predetermined time has elapsed (S103, YES). On the other hand, when the predetermined time has not elapsed (S103, NO), the process returns to step S102.

When a pen-up event is input, the stroke processing unit 32 sets the end flag of the divided stroke information to true (S104).
Next, the stroke processing unit 32 adds an end time to the divided stroke information (S105), and proceeds to step S106.

When the fixed time has elapsed (S103, YES), the stroke processing unit 32 sets the end flag of the divided stroke information to false (S120), and proceeds to step S106.
The stroke processing unit 32 determines whether or not the divided stroke information is transmitted for the first time after the electronic pen enters the display 3 and approaches or comes into contact (S106). The stroke processing unit 32 proceeds to step S107 when the divided stroke information is transmitted for the first time (S106, YES). On the other hand, if it is not the divided stroke information transmitted for the first time (S106, NO), the process proceeds to step S108.
Next, the stroke processing unit 32 adds the stroke data ID, start time, color, and width to the divided stroke information (S107).
Next, the stroke processing unit 32 adds untransmitted coordinate array data to the divided stroke information (S108).
Next, the stroke processing unit 32 transmits the divided stroke information (S109).

Next, the stroke processing unit 32 determines whether or not the divided stroke information of the end flag (: true) has been transmitted (S110). The stroke processing unit 32 ends the process when the divided stroke information of the end flag (: true) is transmitted (S10, YES). On the other hand, when the division stroke information of the end flag (: true) is not transmitted (S110, NO), the process returns to step S102.
When the divided stroke information is the last piece of the stroke information, the end flag becomes true in step S104 described above. Therefore, when the end flag is true, all the divided strokes in one stroke information are obtained. Since this indicates that the transmission of information has been completed, the reception-side electronic blackboard 2b prompts the division stroke information to be combined.
In addition, as described above, when the divided stroke information that is the first fragment of one stroke information is transmitted, the color / width information necessary for drawing the stroke and the stroke data ID necessary for storing the stroke are provided. To do.
Further, the steps between steps S102 to S110 are looped until the electronic pen 4 is detached from the display 3. Once the divided stroke information is transmitted, the divided stroke information is reset (data cleared) while returning from the loop (steps S102 to S110).
In this way, the stroke information is divided into fixed time units starting from the time point when the electronic pen 4 enters the display screen, and divided stroke information is continuously generated, and the stroke is continued in the generated divided stroke information. Continuation information indicating the presence is transmitted to the second image processing unit 30, and when the electronic pen 4 leaves the display screen, end information indicating the end of the stroke is generated in the generated divided stroke information. Is added and transmitted to the second image processing unit 30, so even if one stroke information from when the electronic pen 4 enters the display screen to when it leaves the screen is divided into fixed time units, it is divided into a plurality of pieces. It is possible to reproduce and handle the stroke information as one stroke information.

FIG. 13 is a flowchart for receiving, combining, and drawing divided stroke information in the reception-side electronic blackboard 2b.
In addition, reception of the division | segmentation stroke information shown to step S201 respond | corresponds to step S201 of the sequence diagram shown in FIG. That is, the flowchart shown in FIG. 13 is processed every time the divided stroke information is received.
The communication control unit 60 receives the divided stroke information from the transmission-side electronic blackboard 2a via the communication network 9 (S201).
That is, the communication control unit 60 outputs the divided stroke information received from the transmission-side electronic whiteboard 2a to the remote operation receiving unit 66, and the remote operation receiving unit 66 outputs the divided stroke information to the image processing unit 30. The processing unit 30 outputs the divided stroke information to the stroke processing unit 32, the stroke processing unit 32 outputs the divided stroke information to the page processing unit 37, and the page processing unit 37 further outputs the divided stroke information to the page data storage unit 300. Remember.
The page processing unit 37 combines the divided stroke information received so far with the divided stroke information stored in the divided stroke table 300a of the page data storage unit 300 (S202).
The page processing unit 37 outputs the combined divided stroke information to the display superimposing unit 36, generates an image in which the images of the respective layers (A) to (D) are superimposed in the display superimposing unit 36, and outputs the images to the video superimposing unit 28. Output. Further, the video superimposing unit 28 generates a video signal in which the video signal is superimposed on the image of each layer and draws it on the screen of the display 3 (S203).
Thus, when continuation information is added to the newly received divided stroke information, the new divided stroke information is combined with the divided stroke information being stored in the divided stroke management table 300a, and the divided stroke management table 300a. By storing in, the divided stroke information can be handled as a set of information.

Next, the page processing unit 37 determines whether the received end flag of the divided stroke information is true (S204). If the end flag of the received divided stroke information is true (S204, YES), the process proceeds to step S206. On the other hand, if the termination flag of the received divided stroke information is not true but false (S204, NO), the process proceeds to step S205.
If the end flag of the received divided stroke information is false (S204, NO), the received divided stroke information is temporarily stored in the divided stroke management table 300a of the page data storage unit 300 (S205).

On the other hand, if the end flag of the received divided stroke information is true (S204, YES), the combined stroke information is stored in the stroke management table 300b of the page data storage unit 300 (S206).
Next, the page processing unit 37 deletes all the divided stroke information stored in the divided stroke management table 300a of the page data storage unit 300 (S207).
If the termination flag of the divided stroke information received at this time is true according to the determination processing in step S204 described above, the combined stroke information is approached by the electronic pen 4 entering the display 4 on the transmission-side electronic blackboard 2a. It can be determined that all pieces of divided stroke information from contact to separation are covered.
Therefore, the stroke information is transferred to the stroke management table 300b that manages all the stroke information drawn so far in steps S206 and S207. By saving in the stroke management table 300b, operations such as deletion similar to other stroke information can be performed.
As a result, the combined stroke information from the start of reception of the divided stroke information to the end information can be deleted as a unit. Further, the display position can be moved to another position with the combined stroke information as a unit.
In this way, the division stroke information is received via the communication network, and the division stroke information from the reception start to the end information is stored by combining and storing the division stroke information from the reception start to the end information of the division stroke information. It can be handled as a single piece of information.
If end information is added to the received divided stroke information, the new divided stroke information is combined with the divided stroke information stored in the divided stroke management table 300a, and the combined divided stroke information is combined. Is stored in the stroke management table 300b as stroke information, so that the divided stroke information from the reception start to the end information can be handled as a set of information.
Further, after the stroke information is stored in the stroke management table 300b, all the divided stroke information stored in the divided stroke management table 300a is deleted, whereby all the divided stroke information can be deleted from the divided stroke management table 300a.

Returning to FIG. 7, the stroke management table will be described.
Same as CBS normal stroke management. Strokes are arranged as an array, and each stroke has coordinate data as an array.

FIG. 14 is a diagram for explaining the data configuration of the stroke management table 300b of the page data storage unit 300.
In the stroke management table 300b, as stroke information, one stroke data includes a stroke data ID for identifying the stroke data, a start time indicating the start time of writing one stroke, and a write end time of one stroke. The coordinate arrangement data for identifying the end time shown, the stroke color, the stroke width, and the arrangement of the stroke passing points is shown, and corresponds to one stroke before the completion of the combination.
Further, the coordinate array data shows detailed information as shown in FIG. FIG. 15 is a conceptual diagram showing a configuration of coordinate array data. As shown in FIG. 15, the coordinate array data includes one point (X coordinate value, Y coordinate value) on the display 3, and a time difference (ms from the disclosure time of the stroke when passing through this one point. ) And writing pressure information of the electronic pen 4 at this one point.

FIG. 16 is a diagram for explaining the data structure of the divided stroke information communicated between the transmission-side electronic blackboard and the reception-side electronic blackboard.
In the divided stroke management table 300a, as divided stroke information, one divided stroke data includes a stroke data ID for identifying the divided stroke data, a start time indicating the start time of writing one stroke, and one stroke writing. End time indicating end time, stroke color, stroke width, coordinate array data for identifying the array of stroke passing points, and end flag indicating whether this divided stroke information is the end of the stroke (: False).
Further, the coordinate array data shows detailed information as shown in FIG. FIG. 17 is a conceptual diagram showing the configuration of coordinate array data. As shown in FIG. 17, the coordinate array data includes one point on the display 3 (X coordinate value, Y coordinate value), and a time difference (ms from the disclosure time of the stroke when the one point is passed. ) And writing pressure information of the electronic pen 4 at this one point.
Note that information such as the start time and color in the divided stroke information may be transmitted once for each stroke, and therefore information such as the start time and color is not transmitted in transmission of all the divided stroke information.

Stroke information is sent to the terminal at a fixed time from the timing when the electronic pen approaches or contacts, and the end stroke information is terminated at the receiving terminal side in order to be combined in units from the time the electronic pen approaches or contacts to the terminal. Since the flag is transmitted with the flag set to true, the stroke information can be displayed on the receiving terminal side within a certain time in the electronic blackboard information sharing system, and the stroke is displayed on the receiving terminal side until the electronic pen enters the display screen and then leaves. Since it can be handled by one unit of stroke, operations such as undo function and delete function can be performed.
A toolbar for the user to select an operation function is displayed on the display screen of the electronic blackboard. The undo function is a function for returning a stroke displayed on the display 3 by one stroke as a unit to each stroke unit. The deletion function is a function for selecting and deleting a stroke displayed on the display 3 in units of one stroke as one unit.
As a usage method, for example, when a map is displayed on the display 3 and a route from the point A to the point Z is written, a plurality of routes with different passing points are drawn as stroke images, respectively. The undo function and the delete function can be used with one stroke as one unit.

<Configuration, operation and effect of exemplary embodiment of the present invention>
<First aspect>
The electronic blackboard 2 (image processing apparatus) according to this aspect includes a display 3 (display means), an electronic pen 4 (indicator) that performs a writing operation by moving along the display screen of the display 3, and a display screen. Based on the coordinate information of the coordinates detected by the coordinate detection unit 22 (coordinate detection means) and the coordinates detected by the coordinate detection unit 22, the coordinate information about the stroke by the writing operation is stroke information. A stroke processing unit 32 (stroke information generating means) that generates the stroke as a starting point, and a stroke processing unit that continuously generates divided stroke information by dividing the stroke information in a fixed time unit starting from the time when the electronic pen 4 enters the display screen. 32 (division generation means) and the stroke is continued in the division stroke information generated by the stroke processing unit 32. A communication control unit 60 (first stroke transmission unit) that adds continuation information to be transmitted and transmits it to the electronic blackboard 2 (another image processing apparatus), and a stroke processing unit 32 when the electronic pen 4 leaves the display screen. A communication control unit 60 (second stroke transmitting means) that adds the end information indicating that the stroke has been completed to the divided stroke information generated by the above and transmits it to the electronic blackboard 2 (another image processing apparatus); It is characterized by having.
According to this aspect, the stroke information is divided into fixed time units starting from the time when the electronic pen 4 enters the display screen, and the divided stroke information is continuously generated, and the stroke continues in the generated divided stroke information. When the electronic pen 4 is detached from the display screen when the continuation information indicating that it is in the middle is added and transmitted to the electronic blackboard 2 (another image processing apparatus), the stroke ends in the generated divided stroke information. Since the end information indicating this is added and transmitted to the electronic blackboard 2 (another image processing apparatus), the stroke information from when the electronic pen 4 enters the display screen to when it leaves the display is divided into fixed time units. Even in this case, the stroke information divided into a plurality of pieces can be reproduced and handled as stroke information.

<Second aspect>
The image processing system of this aspect is an image processing system 1 including the electronic blackboard 2 (image processing apparatus) described in the first aspect and the second electronic blackboard 2 (other image processing apparatus), The electronic blackboard 2 (another image processing apparatus) 2 receives a divided stroke information from the electronic blackboard 2 (image processing apparatus) via a communication network, and starts receiving divided stroke information. And a stroke processing unit 32 (combined storage unit) for combining and storing the divided stroke information from the end information to the end information in the divided stroke management table 300a (first storage unit).
According to this aspect, the divided stroke information from the reception start to the end information is received by receiving the divided stroke information via the communication network and combining and storing the divided stroke information from the reception start to the end information of the divided stroke information. Information can be handled as a group of information.

<Third aspect>
When the continuation information is added to the divided stroke information newly received by the communication control unit 60 (reception unit), the stroke processing unit 32 (combination storage unit) of this aspect has the divided stroke management table 300a (first The new stroke information is combined with the divided stroke information being stored in the storage means) and stored in the divided stroke management table 300a (first storage means).
According to this aspect, when continuation information is added to the newly received divided stroke information, the divided stroke management is performed by combining the new divided stroke information with the divided stroke information being stored in the divided stroke management table 300a. By storing in the table 300a, the divided stroke information can be handled as a set of information.

<4th aspect>
When the end information is added to the divided stroke information received by the communication control unit 60 (reception unit), the stroke processing unit 32 (combination storage unit) of this aspect has a divided stroke management table 300a (first storage). And the new divided stroke information is combined with the divided stroke information that is being stored, and the combined divided stroke information is stored in the stroke management table 300b (second storage means) as stroke information. To do.
According to this aspect, when end information is added to the received divided stroke information, the new divided stroke information is combined with the divided stroke information being stored in the divided stroke management table 300a, and the combined pieces are combined. Since the divided stroke information is stored in the stroke management table 300b as stroke information, the divided stroke information from the reception start to the end information can be handled as a set of information.

<5th aspect>
The stroke processing unit 32 (combination storage unit) of this aspect stores the stroke information in the stroke management table 300b (second storage unit), and then stores the division information being stored in the divided stroke management table 300a (first storage unit). All the stroke information is deleted.
According to this aspect, after the stroke information is stored in the stroke management table 300b, all the divided stroke information is deleted from the divided stroke management table 300a by deleting all the divided stroke information stored in the divided stroke management table 300a. be able to.

<Sixth aspect>
The image processing method of this aspect is an image processing method by the electronic blackboard 2 described in the first aspect, and the first image processing unit 30 is a unit of a fixed time starting from the time when the electronic pen 4 enters the display screen. In step S120, S107, S108, the stroke information is divided and the divided stroke information is continuously generated, and continuation information indicating that the stroke is continuing is added to the generated divided stroke information. The first stroke transmission step (S120, S109) to be transmitted to the electronic blackboard 2 (another image processing apparatus), and when the electronic pen 4 leaves the display screen, the stroke ends with the generated divided stroke information. A second stroke transmission step (S104, S109) for adding the end information indicating that it has been performed and transmitting it to the electronic blackboard 2 (another image processing apparatus) , Characterized in that the run.
According to this aspect, the stroke information is divided into fixed time units starting from the time when the electronic pen 4 enters the display screen, and the divided stroke information is continuously generated, and the stroke continues in the generated divided stroke information. When the electronic pen 4 leaves the display screen, the continuation information indicating that it is in the middle is added and transmitted to the second image processing unit 30. When the electronic pen 4 leaves the display screen, the generated divided stroke information indicates that the stroke has ended. Since the end information is added and transmitted to the electronic blackboard 2 (another image processing apparatus), even when the stroke information from when the electronic pen 4 enters the display screen to when it is released is divided into fixed time units, The stroke information divided into a plurality of pieces can be reproduced and handled as one piece of stroke information.

<Seventh aspect>
The electronic blackboard 2 (another image processing apparatus) according to this aspect receives the division stroke information from the electronic blackboard 2 (image processing apparatus) via the communication network (S201), and ends from the reception start of the division stroke information. A combined storage step (S202) is performed in which the divided stroke information up to the information is combined and stored in the divided stroke management table 300a (first storage means).
According to this aspect, the divided stroke information from the reception start to the end information is received by receiving the divided stroke information via the communication network and combining and storing the divided stroke information from the reception start to the end information of the divided stroke information. Information can be handled as a group of information.

<Eighth aspect>
A program according to this aspect causes a processor to execute each step described in the sixth aspect.
According to this aspect, each step can be executed by the processor.

<Ninth aspect>
A program according to this aspect causes a processor to execute each step described in the seventh aspect.
According to this aspect, each step can be executed by the processor.

DESCRIPTION OF SYMBOLS 1 ... Image processing system, 2 ... Electronic blackboard, 3 ... Display, 4 ... Electronic pen, 5 ... USB memory, 6 ... Notebook PC, 7 ... Video conference terminal, 8 ... PC, 9 ... Communication network, 10a1 ... Cable, 20 ... Client machine, 90 ... Server machine, 20 ... Client part, 90 ... Server part, 20 ... Client machine, 21 ... Video acquisition part, 22 ... Coordinate detection part, 23 ... Automatic adjustment part, 24 ... Contact detection part, 25 ... Event distribution unit, 26 ... operation processing unit, 27 ... gesture processing unit 28 ... video superimposition unit, 29 ... finger discrimination unit, 29-1 ... finger discrimination unit, 29-2 ... finger input start standby timer, 30 ... image processing , 31 ... Image acquisition unit, 38 ... Marker processing unit, 2 ... Stroke processing unit, 36 ... Display superimposition unit, 33 ... UI image generation unit, 34 ... Background generation unit, 35 ... Layout management unit, 39 ... Conte , Page processing unit, 40 ... file processing unit, 41 ... recovery processing unit, 43 ... file conversion unit, 44 ... file transmission unit, 45 ... address book input unit, 46 ... backup processing unit, 47 ... backup Output unit 48 ... Setting management unit 60 ... Communication control unit 61 ... Remote start processing unit 62 ... Remote participation processing unit 63 ... Remote image transmission unit 64 ... Remote image reception unit 65 ... Remote operation transmission unit 66 ... Remote operation receiving unit, 68 ... Remote context transmitting unit, 67 ... Remote context receiving unit, 71 ... Remote connection request receiving unit, 80 ... Data management unit, 72 ... Remote connection result transmitting unit, 81 ... Remote connection processing unit, 82 ... Remote image processing unit, 83 ... Remote operation processing unit, 84 ... Operation composition processing unit, 101 ... CPU, 102 ... ROM, 103 ... RAM, 104 ... SSD, 105 ... Network 106: External storage controller, 111 ... Capture device, 113 ... Display controller, 115 ... Contact sensor, 114 ... Sensor controller, 116 ... Electronic pen controller, 112 ... GPU, 120 ... Bus line, 300 ... Page data storage unit , 300a ... divided stroke management table, 300b ... stroke management table, 310 ... remote license management table, 410 ... address book management table, 420 ... backup data storage unit, 430 ... setting file storage unit, 440 ... connection destination management table, 610 ... Participating site management table, 810 ... passcode management unit, 830 ... image data management unit, 840 ... operation data storage unit

JP 2009-218868 A

Claims (9)

Display means;
An indicator for performing a writing operation by moving along the display screen of the display means;
Coordinate detection means for detecting each handwritten coordinate of the indicator that moves along the display screen;
Stroke information generating means for generating, as stroke information, coordinate information about a stroke by the writing operation based on coordinate information of coordinates detected by the coordinate detecting means;
A division generation unit that continuously generates divided stroke information by dividing the stroke information in units of a fixed time starting from a time when the indicator enters the display screen;
First stroke transmission means for adding continuation information indicating that the stroke is continuing to the divided stroke information generated by the division generation means and transmitting the information to another image processing apparatus;
When the indicator leaves the display screen, end information indicating the end of the stroke is added to the divided stroke information generated by the divided generation unit and transmitted to the other image processing apparatus. Stroke transmission means of
An image processing apparatus comprising: An image processing system comprising the image processing apparatus according to claim 1 and another image processing apparatus,
The other image processing apparatus includes:
Receiving means for receiving the divided stroke information from the image processing apparatus via a communication network;
A combined storage unit that combines the divided stroke information from the start of reception of the divided stroke information to the end information and stores the combined stroke information in a first storage unit;
An image processing system comprising: The binding storage means includes
In the case where continuation information is added to the divided stroke information newly received by the receiving means, the new divided stroke information is combined with the divided stroke information stored in the first storage means to The image processing system according to claim 2, wherein the image processing system is stored in one storage unit. The binding storage means includes
When end information is added to the divided stroke information received by the receiving means, the new divided stroke information is combined with the divided stroke information stored in the first storage means, and 4. The image processing system according to claim 3, wherein a group of divided stroke information is stored in the second storage means as stroke information. The binding storage means includes
5. The image processing system according to claim 4, wherein after the stroke information is stored in the second storage unit, all of the divided stroke information stored in the first storage unit is deleted. An image processing method by the image processing apparatus according to claim 1,
A division generation step of continuously generating divided stroke information by dividing the stroke information in units of a fixed time starting from a time point when the indicator enters the display screen;
A first stroke transmission step of adding continuation information indicating that the stroke is continuing to the divided stroke information generated by the division generation step and transmitting the information to another image processing apparatus;
When the indicator leaves the display screen, end information indicating the end of the stroke is added to the divided stroke information generated by the divided generation step and transmitted to the other image processing apparatus. An image processing method comprising: performing a stroke transmission step of: The other image processing apparatus includes:
Receiving the divided stroke information from the image processing device via a communication network;
The image processing according to claim 6, further comprising: a combination storing step of combining the divided stroke information from the reception start of the divided stroke information to the end information and storing the combined stroke information in a first storage unit. Method.   A program causing a processor to execute each step according to claim 6.   A program for causing a processor to execute each step according to claim 7.

Images