JP2016143183A - Information processing apparatus, image processing system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus configured to transmit a conversion candidate of handwritten information to another device.SOLUTION: An information processing apparatus (electronic blackboard) 2 for displaying information on a display unit 3 includes: pointed position detection means 25 (event processing section) that detects a position of information pointed on a display surface; visual information generation means 32 (stroke processing section) which generates visual information from the pointed position detected by the pointed position detection means; a character recognition section 321 which recognizes a symbol formed by one or more pieces of visual information generated by the visual information generation means, and determines a recognition candidate of the symbol; data conversion means 322 (serialization/deserialization section) that converts the symbol and the recognition candidate to data to be transmitted; and a communication control section 60 which transmits the symbol and the recognition candidate converted to the data by the data conversion means, to another information processing apparatus.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an information processing apparatus, an image processing system, and a program.

  2. Description of the Related Art Conventionally, an electronic blackboard on which a background image is displayed on a large display and a user can hand-write letters, numbers, figures, etc. is used on the background image in a meeting at a company, an educational institution, an administrative organization, or the like.

  In such an electronic blackboard, the large display serves as both a display device and an input device, and the user can input coordinates to the large display using a dedicated pen or a finger. The electronic blackboard superimposes characters, numbers, figures, etc. drawn by the user by connecting the coordinates in time series and displays them on the display device.

  The electronic blackboard can store the characters displayed in this manner as an image, but can recognize characters and numbers to convert them into text data. If converted into text data, the storage capacity can be reduced as compared with the case where the data is stored as an image, and the reuse of characters becomes easier.

  However, it is known that the recognition rate of handwritten characters is not necessarily 100%. For this reason, a technique for dealing with erroneous recognition of characters has been devised (for example, see Patent Document 1). Patent Document 1 discloses a character processing device including a candidate buffer that holds a recognition supplement for each character of a document so that the character can be reconverted even after the character is recognized.

  However, although the character processing device disclosed in Patent Document 1 stores conversion candidates that can be specified by a pointer in the candidate buffer, there is a problem that it is difficult to transfer the candidate buffer to another device. That is, since it is difficult to transfer the character-recognized character and the conversion candidate to another device, the other device cannot convert the character using the conversion candidate.

  In view of the above-described problems, an object of the present invention is to provide an information processing apparatus capable of transmitting handwritten information conversion candidates to another apparatus.

  In view of the above problems, the present invention is an information processing apparatus that displays information on a display device, and includes an indication position detection unit that detects an indication position on an information display surface, and the indication position detected by the indication position detection unit Visual information generating means for generating visual information, symbol recognition means for recognizing a symbol formed by one or more visual information generated by the visual information generating means, and determining a recognition candidate for the symbol; Data conversion means for converting the recognition candidate into data that can be transmitted, and transmission means for transmitting the symbol converted into data that can be transmitted by the data conversion means and the recognition candidate to another information processing apparatus.

  An information processing apparatus capable of transmitting handwritten information conversion candidates to another apparatus can be provided.

It is an example of the whole block diagram of the image processing system of this embodiment. It is an example of the hardware block diagram of an electronic blackboard. It is an example of the functional block diagram of an electronic blackboard. It is an example of a functional block diagram of a file processing unit. It is a figure which shows an example of a function structure of a communication control part and a server machine. It is an example of the conceptual diagram which shows stroke arrangement | sequence data. It is an example of the conceptual diagram which shows coordinate arrangement | sequence data. It is an example of the block diagram of each image layer. It is an example of the sequence diagram which showed the process of each electronic blackboard. It is an example of the sequence diagram which showed the process of each electronic blackboard. It is an example of the flowchart figure which shows the procedure in which the character recognition part 321 recognizes a symbol from the coordinate of the visual information handwritten. It is an example of the symbol drawn on the display and its recognition result. It is an example of the flowchart figure which shows the procedure in which the stroke process part 32 preserve | saves the text containing a recognition process candidate. It is an example of the figure for demonstrating serialization and deserialization.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

<Terminology>
Terms used in the following embodiments will be described.
Object: Visual information displayed on a display with one or more coordinates connected. The objects of this embodiment include text, graphics, strokes, and background images.
Text: Symbols such as characters, numerical values, special symbols (%, &, =, to, etc.), which are visual information that can be handled by codes. The text is converted into a code by symbol recognition (sometimes referred to as character recognition). In the present embodiment, the handwritten state is also included in the text.
A figure: a figure having a predetermined shape such as a circle or a rectangle. It may be converted into a code representing a figure determined by shape recognition, or may be recognized as a diagram surrounded by straight lines or curves.
Stroke: Visual information that cannot be distinguished from either text or graphics is a stroke.
-Page data: An object for one page displayed on the display.

<System overview>
FIG. 1 is an overall configuration diagram of an image processing system according to the present embodiment. In FIG. 1, only two electronic blackboards 2a and 2b and associated electronic pens 4a and 4b are shown to simplify the description. A pen or the like may be used. The electronic blackboards 2a and 2b are installed at different bases.

  As shown in FIG. 1, the image processing system 1 includes a plurality of electronic blackboards 2a and 2b, a plurality of electronic pens 4a and 4b, USB memories 5a and 5b, notebook PCs (Personal Computers) 6a and 6b, a video conference. Terminals 7a and 7b (also referred to as video conference terminals) and a PC 8 are provided. The electronic blackboards 2a and 2b and the PC 8 are connected via a communication network 9 so that they can communicate with each other. The PC 8 is a PC for sharing the screen with the electronic blackboard 2. Further, the plurality of electronic blackboards 2a and 2b are respectively provided with displays 3a and 3b (the display 3 is an example of a display device, and information is displayed on the display surface of the display). The electronic blackboard 2a displays an image drawn by an event generated by the electronic pen 4a (touching the pen tip of the electronic pen 4a or the pen bottom of the electronic pen 4a on the display 3a) on the display 3a. be able to. Note that the electronic blackboard 2a changes the image displayed on the display 3a based on not only the electronic pen 4a but also an event (a gesture such as enlargement, reduction, and page turning) generated by the user's hand Ha or the like. You can also

  Further, a USB memory 5a can be connected to the electronic blackboard 2a, and the electronic blackboard 2a reads an electronic file such as PDF (Portable Document Format) from the USB memory 5a or records an electronic file in the USB memory 5a. be able to. Further, the electronic blackboard 2a is connected via a cable 10a1 capable of communication according to standards such as DisplayPort, DVI (Digital Visual Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), and VGA (Video Graphics Array). A notebook PC 6a is connected. Then, the electronic blackboard 2a detects the contact of the electronic pen 4a or the hand Ha with the display 3a and generates an event. Event information indicating the event is sent to the notebook PC 6a in the same manner as an event of an input device such as a mouse or a keyboard. Send.

  Similarly, a video conference terminal 7a is connected to the electronic blackboard 2a via a cable 10a2 capable of communication according to the above standard. Note that the notebook PC 6a and the video conference terminal 7a may communicate with the electronic blackboard 2a by wireless communication complying with various wireless communication protocols such as Bluetooth (registered trademark) and wireless LAN.

  On the other hand, in the other bases where the electronic blackboard 2b is installed, the electronic blackboard 2b with the display 3b, the electronic pen 4b, the USB memory 5b, the notebook PC 6b, the video conference terminal 7b, the cable 10b1, and the cable 10b2 in the same manner as described above. Is used. Furthermore, the electronic blackboard 2b can change the image displayed on the display 3b based on an event caused by the user's hand Hb or the like.

  Thereby, the image drawn on the display 3a of the electronic blackboard 2a at one base is also displayed on the display 3b of the electronic blackboard 2b at the other base, and conversely on the display 3b of the electronic blackboard 2b at the other base. The image drawn on is displayed on the display 3a of the electronic blackboard 2a at one base. As described above, the image processing system 1 can perform remote sharing processing for sharing the same image at a remote location, and thus is very convenient when used for a conference or the like at a remote location.

  In the following, when an arbitrary electronic blackboard among the plurality of electronic blackboards is indicated, it is indicated as “electronic blackboard 2”. When an arbitrary display among a plurality of displays is shown, “display 3” is shown. When an arbitrary electronic pen among a plurality of electronic pens is indicated, it is indicated as “electronic pen 4”. When an arbitrary USB memory is indicated among the plurality of USB memories, it is indicated as “USB memory 5”. When an arbitrary notebook PC among the plurality of notebook PCs is indicated, it is indicated as “notebook PC 6”. When an arbitrary video conference terminal is indicated among the plurality of video conference terminals, it is indicated as “video conference terminal 7”. In addition, “hand H” is used when an arbitrary hand among the hands of a plurality of users is shown. When an arbitrary cable is indicated among the plurality of cables, it is indicated as “cable 10”.

  In the present embodiment, an electronic blackboard is described as an example of an image processing device. However, the present invention is not limited to this, and other examples of the image processing device include electronic signboards (digital signage), sports and weather forecasts. It may be a telestrator to be used, a remote image (video) diagnostic device, or the like. The notebook PC 6 will be described as an example of the information processing terminal, but is not limited thereto. Other examples of the information processing terminal include a desktop PC, a tablet PC, a wearable PC, a smartphone, and a PDA (Personal Digital Assistant). It may be a terminal capable of supplying image frames such as a digital video camera, a digital camera, and a game machine.

  Further, the communication network 9 includes the Internet, a LAN (Local Area Network), a mobile phone communication network, and the like. In this embodiment, a USB memory is described as an example of a recording medium. However, the present invention is not limited to this, and various recording media such as an SD card may be used as another example of the recording medium.

<Hardware configuration of electronic blackboard>
Next, the hardware configuration of the electronic blackboard according to the present embodiment will be described with reference to FIG. FIG. 2 is an example of a hardware configuration diagram of the electronic blackboard.

  As shown in FIG. 2, the electronic blackboard 2 includes a CPU 101 that controls the operation of the electronic blackboard 2 as a whole, a ROM 102 that stores programs used to drive the CPU 101 such as an IPL (Initial Program Loader), and a work area of the CPU 101. Control of communication with the RAM 103, the SSD 104 for storing the program 1041 for providing the functions of the electronic blackboard 2 and various data, the network controller 105 for controlling communication with the communication network 9, and the USB memory 5 The external storage controller 106 is provided.

  The electronic blackboard 2 displays a capture device 111 that displays video information as a still image or a moving image on the display of the notebook PC 6, a GPU (Graphics Processing Unit) 112 that specializes in graphics, and an output image from the GPU 112. 3 and a display controller 113 for controlling and managing screen display for output to the video conference terminal 7. Note that the capture device 111 can acquire video information of a still image or a moving image displayed on the display of the notebook PC 6 by the notebook PC 6 and output it to the display controller 113.

  The electronic blackboard 2 further includes a sensor controller 114 that controls processing of the contact sensor 115 and a contact sensor 115 that detects that the electronic pen 4 or the hand H of the user touches the display 3. The contact sensor 115 performs coordinate input and coordinate detection by an infrared ray blocking method. In this method of inputting coordinates and detecting coordinates, two light emitting / receiving devices installed at both upper ends of the display 3 radiate a plurality of infrared rays in parallel to the display 3, and reflections provided around the display 3. In this method, light reflected by the member on the same optical path is received. The contact sensor 115 captures an infrared light blocking position blocked by an object (the electronic pen 4 or the hand H) and outputs the image to the sensor controller 114. The sensor controller 114 specifies a coordinate position (an example of an instruction position) that is a contact position of an object based on the blocking position.

  In addition, the contact sensor 115 is not limited to the infrared ray blocking method, but a capacitance type touch panel that identifies a contact position by detecting a change in capacitance, or a contact position determined by a voltage change of two opposing resistance films. Various detection means such as a resistance film type touch panel to be identified and an electromagnetic induction type touch panel to identify a contact position by detecting electromagnetic induction caused by contact of a contact object with the display unit may be used.

  The electronic blackboard 2 includes an electronic pen controller 116. This electronic pen controller 116 communicates with the electronic pen 4 to determine whether or not the pen tip or pen butt touches the display 3. It should be noted that the electronic pen controller 116 may determine whether there is a touch on a part gripped by the user of the electronic pen 4 and other parts of the electronic pen 4 as well as the pen tip and the bottom of the electronic pen 4. .

  Further, the electronic blackboard 2 electrically connects the CPU 101, ROM 102, RAM 103, SSD 104, network controller 105, external storage controller 106, capture device 111, GPU 112, and sensor controller 114 as shown in FIG. A bus line 120 such as an address bus or a data bus is provided.

  The program 1041 executed by the CPU 101 of the electronic blackboard 2 may be recorded and distributed on a computer-readable recording medium such as a CD-ROM, or may be distributed from an information processing apparatus that distributes the program via the communication network 9. May be distributed.

<Functional configuration of electronic blackboard>
Next, the functional configuration of the electronic blackboard 2 will be described with reference to FIGS. First, the overall functional configuration of the electronic blackboard 2 will be described with reference to FIG. FIG. 3 is an example of a functional block diagram of the electronic blackboard.

  The electronic blackboard 2 implements each function shown in FIG. 3 by the CPU 101 shown in FIG. 2 executing the program 1041 and controlling the hardware shown in FIG. The electronic blackboard 2 includes a client machine 20 and a server machine 90. That is, the client machine 20 and the server machine 90 are included in one housing of the electronic blackboard 2. Hereinafter, it demonstrates in order.

<< Functional Configuration of Client Machine 20 >>
The functional configuration of the client machine 20 will be described mainly with reference to FIGS. The client machine 20 includes a video acquisition unit 21, a coordinate detection unit 22, an automatic adjustment unit 23, a contact detection unit 24, an event processing unit 25, an operation processing unit 26, a gesture processing unit 27, a video synthesis unit 28, an image processing unit 30, The communication control unit 60 is included.

  The video acquisition unit 21 acquires an output video of a video output device such as a notebook PC 6 connected to the cable 10. When receiving the image signal from the video output device (notebook PC 6), the video acquisition unit 21 analyzes the image signal, and the resolution of the image frame that is a display image of the video output device formed by the image signal, Image information such as an image frame update frequency is derived and output to the image acquisition unit 31.

  The coordinate detection unit 22 detects the coordinate position of the event generated by the user on the display 3. The coordinate detection unit 22 also detects the area touched by the user.

  The automatic adjustment unit 23 automatically adjusts the coordinate detection unit 22 when the electronic blackboard 2 is activated (when the power is turned on or when returning from the energy saving mode).

  The contact detection unit 24 detects an event (the pen tip of the electronic pen 4 or the pen tip of the electronic pen 4 is touched on the display 3) generated by the user.

  The event processing unit 25 distributes stroke drawing (object drawing), UI operation, and gesture operation based on the coordinate position of the event detected by the coordinate detection unit 22 and the detection result detected by the contact detection unit 24. Here, stroke drawing is drawing by the user with the electronic pen 4 or the hand H. The UI operation is an operation of a UI image (a UI image (A) described later) by the user, for example, an operation for setting a color, a width, and the like of a line drawn by the electronic pen 4. The gesture operation is a gesture operation by the user. For example, when the user moves the hand H while the hand H is in contact with the display 3, the image is enlarged (or reduced), the display area is changed, or This is an operation for switching pages.

  The operation processing unit 26 executes various operations in accordance with the UI element in which the event is generated, from what is determined as the UI operation by the event processing unit 25. Examples of the UI element include a button, a list, a check box, and a text box.

  The gesture processing unit 27 performs an operation corresponding to what is determined as a gesture operation by the event processing unit 25.

  The video synthesizing unit 28 displays the image synthesized by the display synthesizing unit 36 described later as a video on the display 3 or the like. In addition, the video composition unit 28 performs picture-in-picture processing on the video sent from another video output device (such as the video conference terminal 7) in response to the video from the video output device (such as the notebook PC 6). Displayed on the display 3. Further, the video composition unit 28 performs switching for displaying the video image-in-picture and displayed on a part of the display 3 on the entire display 3.

  Next, the image processing unit 30 will be described. The image processing unit 30 performs synthesis processing of each image layer as shown in FIG. FIG. 8 is an example of a configuration diagram of each image layer. The image processing unit 30 includes an image acquisition unit 31, a stroke processing unit 32, a UI image generation unit 33, a background generation unit 34, a layout management unit 35, a display composition unit 36, a page processing unit 37, a file processing unit 40, and page data storage. Section 300 and a remote license management table 310.

  Among these, the image acquisition unit 31 acquires each frame as an image from the video acquired by the video acquisition unit 21. This image corresponds to an output image (C) from the video output device (notebook PC 6) shown in FIG.

  The stroke processing unit 32 draws an image based on a stroke or the like by the event processing unit 25, such as a touch of the pen tip of the electronic pen 4, a touch of the pen butt, or a touch by the user's hand H, or a drawn image. Delete or edit the rendered image. The image drawn by this stroke corresponds to the stroke image (B) shown in FIG. In addition, each result of drawing, deleting, and editing an image based on the stroke and the like is stored in an operation data storage unit 840 described later as operation data described later.

  The stroke processing unit 32 includes a character recognition unit 321 and a serialization / deserialization unit 322. The character recognizing unit 321 analyzes the coordinate data acquired by the event processing unit 25 and discriminates it into text (a symbol such as a character, a numerical value, or a special symbol), a figure such as a circle or a rectangle, or a stroke. When it is determined as text, handwritten visual information is recognized as a symbol. There are various methods for symbol recognition, such as a method using an extended cell feature, a method using a weighted direction histogram as a feature amount, a method using a contour direction contribution feature, and any method can be used in this embodiment. Good. However, in any method, a standard pattern of the feature amount of each symbol is prepared, and a standard pattern symbol closest to the feature amount pattern of the handwritten text is recognized as text.

  At the time of recognition, the character recognition unit 321 stores at least one recognition processing candidate (an example of a recognition candidate) in association with each symbol. In the recognition of characters (symbols), a text having the highest probability of matching with handwritten text is identified as a recognition result based on the pattern similarity. Then, text other than the text with the highest probability of matching becomes a recognition processing candidate (for the convenience of processing, the text with the highest probability of matching is included in the recognition processing candidate of this embodiment). The recognition process candidates are extracted in a range of a predetermined minimum number and maximum number in descending order of the matching probability. The symbols and recognition process candidates are output to the page processing unit 37.

  The serialization / deserialization unit 322 serializes the text and the recognition process candidate, or deserializes the serialized text and the recognition process candidate. Details will be described later.

  The UI image generation unit 33 generates a UI (user interface) image set in advance on the electronic blackboard 2. This UI image corresponds to the UI image (A) shown in FIG.

  The background generation unit 34 generates a background image displayed on the display 3. This background image corresponds to the background image (D) shown in FIG. The pattern of the background image is plain or grid display.

  The layout management unit 35 manages layout information indicating the layout of each image output from the image acquisition unit 31, the stroke processing unit 32, the UI image generation unit 33, and the background generation unit 34. Output. Thereby, the layout management unit 35 causes the display composition unit 36 to display the output image (C) and the stroke image (B) in which position in the UI image (A) and the background image (D) or It is possible to instruct whether to hide.

  Based on the layout information output from the layout management unit 35, the display synthesis unit 36 performs the layout of each image output from the image acquisition unit 31, the stroke processing unit 32, the UI image generation unit 33, and the background generation unit 34. .

  The page processing unit 37 records text, graphics, strokes, and a background image (D) including recognition processing candidates in the page data storage unit 300 as page data for one page. Of these, text, graphics, and strokes including recognition process candidates are stroke images (B). The page processing unit 37 redisplays the recorded page and re-edits the stroke. Further, the page processing unit 37 deletes or duplicates the page.

  The display composition unit 36 outputs an output image (C) from the image acquisition unit 31, a stroke image (B) from the stroke processing unit 32, a UI image (A) from the UI image generation unit 33, and a background generation unit 34. The background image (D) is synthesized according to the layout designated by the layout management unit 35. Accordingly, as shown in FIG. 8, each of the UI image (A), the stroke image (B), the output image (C), and the background image (D) is displayed in the order in which the user can see each image even if they overlap. It has Leia composition.

  The display composition unit 36 can also perform composition by switching between the output image (C) and the background image (D) shown in FIG. For example, when the cable 10 connecting the electronic blackboard 2 and the video output device (notebook PC 6 or the like) is removed, the output image (C) can be removed from the synthesis target by the designation of the layout management unit 35. The display composition unit 36 also performs enlargement of the synthesized image, reduction of display, and movement processing of the display area.

  The page data storage unit 300 stores page data as shown in Table 1. Table 1 is an example of a conceptual diagram showing page data. The page data is data for one page (text, figure, stroke, and background image) displayed on the display 3. The contents of the page data will be described below.

ページデータは、表１に示されているように、任意の１ページを識別するためのページデータＩＤ、このページの表示を開始した時刻を示す開示時刻、ストロークやジェスチャ等によるページの内容の書き換えが行われなくなった時刻を示す終了時刻、電子ペン４やユーザの手Ｈによるストロークによって生じたストローク配列データを識別するためのストローク配列データＩＤ、及び、メディアデータを識別するためのメディアデータＩＤが関連付けて記憶されている。これらのＩＤを含む全てのＩＤは、識別情報の一例である。

As shown in Table 1, the page data is a page data ID for identifying an arbitrary page, a disclosure time indicating the start time of display of this page, and rewriting of the contents of the page by a stroke, a gesture, etc. The end time indicating the time when the operation is stopped, the stroke arrangement data ID for identifying the stroke arrangement data generated by the stroke of the electronic pen 4 or the user's hand H, and the media data ID for identifying the media data It is stored in association. All IDs including these IDs are examples of identification information.

  Here, the media data is image data displayed on the display 3 when the page data is stored. For example, when the background image (D) and the stroke image (B) are displayed on the display 3, the media data includes both the data of the background image (D) and the data of the stroke image (B). means. When the background image (D) is not displayed on the display 3 and only the stroke image (B) is displayed, the media data means the data of the stroke image (B).

  With such page data, for example, when the user draws the alphabet “S” with the electronic pen 4, the stroke is written with one stroke, and therefore, one letter (S with one stroke data ID described below) is used for one letter [S]. Is shown. However, when the electronic pen 4 is used to draw the alphabet “T”, it is written with two strokes, so that one stroke of the alphabet “T” is indicated by two stroke data IDs.

  FIG. 6 is an example of a conceptual diagram showing stroke arrangement data. The stroke arrangement data shows detailed information as shown in FIG. One stroke arrangement data is represented by a plurality of stroke data. One stroke data means one stroke from pen down to pen up. One stroke data includes a stroke data ID for identifying the stroke data, a start time indicating the start time of writing one stroke, an end time indicating the end time of writing one stroke, the color of the stroke, The stroke width, the coordinate array data ID for identifying the array of the passing points of the stroke, and the symbol ID of the symbol determined from the coordinates are shown. The symbol ID is identification information for uniquely identifying a symbol within one page data. For example, when one symbol is handwritten by one stroke like “S”, one symbol ID is associated with one stroke data ID. When one symbol is handwritten with two strokes as in “T”, one symbol ID is associated with two stroke data IDs.

  The coordinate array data specified by the coordinate array data ID indicates detailed information as shown in FIG. FIG. 7 is an example of a conceptual diagram showing coordinate array data. As shown in FIG. 7, the coordinate array data includes one point (X coordinate value, Y coordinate value) on the display 3, and a differential time (from the disclosure time of drawing a stroke when the one point is passed) ms) and information on the writing pressure of the electronic pen 4 at this one point. That is, a collection of one point shown in FIG. 8 represents one coordinate array data specified by the stroke data ID shown in FIG. For example, when the alphabet “S” is drawn with the electronic pen 4, it is written with a single stroke, but passes through a plurality of passing points before the drawing of “S”, so the coordinate array data is information on these passing points. Is shown.

表２は記号データを概念的に示す例である。記号データは、記号を特定するための記号ＩＤ、手書きされた１つ以上のストロークから変換されたテキスト、及び、テキスト以外の１つ以上の認識処理候補を有する。ページデータに記号データが紐付けられているため、ユーザがページをディスプレイ３に表示すると認識処理候補からテキストを選択することができる。

Table 2 is an example conceptually showing the symbol data. The symbol data includes a symbol ID for identifying the symbol, text converted from one or more handwritten strokes, and one or more recognition processing candidates other than the text. Since the symbol data is associated with the page data, when the user displays the page on the display 3, the user can select the text from the recognition process candidates.

次に、表３を用いてメディアデータについて説明する。表３は、メディアデータを示す概念的な表の一例である。メディアデータは、各メディアデータを識別するためのメディアデータＩＤ、メディアデータの種類を示すデータ種類、メディアデータが記録された時刻を示す記録時刻、メディアデータの左上コーナーの座標のＸ座標値、Ｙ座標値、メディアデータの幅、高さ、及び、メディアデータのファイルを示す画像データを有している。

Next, media data will be described using Table 3. Table 3 is an example of a conceptual table showing media data. The media data includes a media data ID for identifying each media data, a data type indicating the type of the media data, a recording time indicating the time when the media data was recorded, an X coordinate value of the coordinates of the upper left corner of the media data, Y It has image data indicating coordinate values, media data width and height, and media data files.

  Returning to FIG. The remote license management table 310 manages license data necessary for executing remote sharing processing. The remote sharing process is a process in which the electronic blackboard 2a at one site shares an image displayed on the displays 3a and 3b with the electronic blackboard 2b at the other site.

表４は、遠隔ライセンス管理テーブルの概念図の一例を示す。遠隔ライセンス管理テーブル３１０では、電子黒板２のプロダクトＩＤ、認証に用いられるライセンスＩＤ、及びライセンスの有効期限が関連付けて管理されている。

Table 4 shows an example of a conceptual diagram of the remote license management table. In the remote license management table 310, the product ID of the electronic blackboard 2, the license ID used for authentication, and the license expiration date are managed in association with each other.

  Next, the functional configuration of the file processing unit 40 shown in FIG. 3 will be described with reference to FIG. FIG. 4 is an example of a functional block diagram of the file processing unit 40. In the following, the electronic blackboard 2 that starts the remote sharing process first is indicated as “host device”, and the electronic blackboard 2 that participates in the remote sharing process that has already been started later is indicated as “participating device”.

  The file processing unit 40 includes a recovery processing unit 41, a file input unit 42a, a file output unit 42b, a file conversion unit 43, a file transmission unit 44, an address book input unit 45, a backup processing unit 46, a backup output unit 47, and a setting management unit. 48, a setting file input unit 49a, and a setting file output unit 49b. Further, the file processing unit 40 includes an address book management table 410, a backup data storage unit 420, a setting file storage unit 430, and a connection destination management table 440.

  The recovery processing unit 41 detects the abnormal end after the electronic blackboard 2 is ended due to the abnormal operation, and recovers the unsaved page data.

  The file input unit 42a reads a PDF file from the USB memory 5 and takes each page as page data.

  The file conversion unit 43 converts the page data stored in the page data storage unit 300 into a PDF format file.

  The file output unit 42 b records the PDF file output by the file conversion unit 42 in the USB memory 5.

  The file transmission unit 44 transmits the PDF file generated by the file conversion unit 43 by attaching it to an electronic mail. The transmission destination of this file is determined by displaying the contents of the address book management table 410 on the display 3 by the display composition unit 36 and the file transmission unit 44 accepting selection of a destination from the user. Further, the file transmission unit 44 can accept an input of a mail address as a destination by the user.

表５はアドレス帳管理テーブル４１０の一例を示す。アドレス帳管理テーブル４１０には、宛先の名前及び宛先の電子メールのメールアドレスが関連付けて管理されている。

Table 5 shows an example of the address book management table 410. In the address book management table 410, the name of the destination and the mail address of the destination e-mail are managed in association with each other.

  Returning to FIG. 4, the address book input unit 45 reads the e-mail address list file from the USB memory 5 and manages it in the address book management table 410.

  The backup processing unit 46 performs backup by storing the file output by the file output unit 42 b and the file transmitted by the file transmission unit 44 in the backup data storage unit 420. If the user does not set backup, backup processing is not performed. The backup data is stored in PDF format as shown in Table 6. Table 6 shows an example of backup data.

バックアップ出力部４７は、バックアップされたファイルをＵＳＢメモリ５に記憶する。この記憶には、パスワードが必要である。

The backup output unit 47 stores the backed up file in the USB memory 5. This memory requires a password.

  The setting management unit 48 manages various setting information of the electronic blackboard 2. Examples of the various setting information include network settings, date and time settings, region and language settings, mail server settings, address book settings, connection destination list settings, and backup settings. The network setting is, for example, an IP address setting of the electronic blackboard 2, a net mask setting, a default gateway setting, a DNS (Domain Name System) setting, or the like.

  The setting file output unit 49b records various setting information of the electronic blackboard 2 in the USB memory 5 as a setting file. Note that the configuration file is encrypted for security reasons, and the user cannot see the contents.

  The setting file input unit 49a reads a setting file stored in the USB memory 5 and reflects various setting information on various settings of the electronic blackboard.

  The address book input unit 50 reads a list file of connection destination IP addresses for remote sharing processing from the USB memory 5 and manages it in the connection destination management table 440 shown in Table 7.

表７は接続先管理テーブル４４０を概念的に示す。接続先管理テーブル４４０は、電子黒板２が遠隔共有処理に参加しようとする参加装置である場合、この参加装置のユーザが主催装置としての役割を果たす電子黒板のＩＰアドレスを入力する手間を削減するために、該ＩＰアドレスを予め管理しておくためのテーブルである。この接続先管理テーブル４４０では、参加可能な主催装置としての電子黒板２が設置されている拠点の名称、及び、主催装置としての電子黒板２のＩＰアドレスが関連付けて管理されている。なお、接続先管理テーブル４４０は無くてもよい。但し、この場合には、参加装置のユーザは、電話や電子メール等によって、主催装置のユーザから主催装置としての電子黒板２のＩＰアドレスを知る必要がある。

Table 7 conceptually shows the connection destination management table 440. When the electronic blackboard 2 is a participating device that intends to participate in the remote sharing process, the connection destination management table 440 reduces the effort for the user of the participating device to input the IP address of the electronic blackboard that plays the role of the host device. Therefore, this is a table for managing the IP address in advance. In the connection destination management table 440, the name of the base where the electronic blackboard 2 as a hosting apparatus that can participate is installed and the IP address of the electronic blackboard 2 as the hosting apparatus are managed in association with each other. Note that the connection destination management table 440 may be omitted. However, in this case, the user of the participating device needs to know the IP address of the electronic blackboard 2 as the hosting device from the user of the hosting device by telephone or electronic mail.

<< Functional Configuration of Communication Control Unit 60 >>
Next, the functional configuration of the communication control unit 60 will be described with reference to FIG. FIG. 5 is a diagram illustrating an example of functional configurations of the communication control unit 60 and the server machine 90. The communication control unit 60 controls communication performed with another electronic blackboard 2 via the communication network 9 and communication performed with the communication control unit 70 described later in the server machine 90. Therefore, the communication control unit 60 includes a remote start processing unit 61, a remote participation processing unit 62, a remote image transmission unit 63, a remote image reception unit 64, a remote operation transmission unit 65, a remote operation reception unit 66, and a participation base management table. 610.

  Among these, the remote start processing unit 61 makes a request for starting a new remote sharing process to the server machine 90 of the electronic blackboard 2 in the same housing, and receives the request result from the server machine 90. In this case, the remote start processing unit 61 refers to the remote license management table 310 and makes a request to start remote sharing processing when license information (product ID, license ID, and expiration date) is managed. be able to. If the license information is not managed, a request to start remote sharing processing cannot be made.

  Further, the remote start processing unit 61 refers to the participating site management table 610. The participation base management table 610 is a table for managing an electronic blackboard as a participating device currently participating in the remote sharing process when the electronic blackboard 2 is a sponsor device. In this participating site management table 610, as shown in Table 8, the name of the site where the participating electronic blackboard 2 is installed and the IP address of the electronic blackboard 2 are managed in association with each other. Table 8 shows an example of the participating base management table.

遠隔参加処理部６２は、通信ネットワーク９を介して、既に遠隔共有処理を開始している主催装置としての電子黒板２のサーバ機９０における遠隔接続要求受信部７１に対して、遠隔共有処理への参加要求を行う。この場合も、遠隔参加処理部６２は、遠隔ライセンス管理テーブル３１０を参照する。また、遠隔参加処理部６２が、既に開始されている遠隔共有処理に参加する場合には、接続先管理テーブル４４０を参照して、参加先の電子黒板２のＩＰアドレスを取得する。なお、遠隔参加処理部６２によって接続先管理テーブルが参照されず、ユーザによって参加先の電子黒板２のＩＰアドレスが入力されてもよい。

The remote participation processing unit 62 transmits the remote connection processing to the remote sharing processing to the remote connection request receiving unit 71 in the server 90 of the electronic blackboard 2 as the hosting device that has already started the remote sharing processing via the communication network 9. Request participation. Also in this case, the remote participation processing unit 62 refers to the remote license management table 310. In addition, when the remote participation processing unit 62 participates in the remote sharing process that has already been started, the IP address of the electronic blackboard 2 of the participation destination is acquired with reference to the connection destination management table 440. The remote participation processing unit 62 may not refer to the connection destination management table, and the user may input the IP address of the electronic blackboard 2 as the participation destination.

  The remote image transmission unit 63 transmits the output image (C) sent from the video acquisition unit 21 via the image acquisition unit 31 to the server machine 90.

  The remote image receiving unit 64 receives image data from a video output device connected to another electronic blackboard 2 from the server 90 and outputs it to the display composition unit 36, thereby enabling remote sharing processing.

  The remote operation transmission unit 65 transmits various operation data necessary for remote sharing processing to the server machine 90. Examples of the operation data include adding a stroke, deleting a stroke, editing a stroke (enlarging, reducing, moving), storing page data, creating page data, duplicating page data, deleting page data, and displaying. Data related to the switching of the current page. In addition, the remote operation receiving unit 66 receives the operation data input from the other electronic blackboard 2 from the server machine 90 and outputs it to the image processing unit 30 to perform remote sharing processing.

<Functional configuration of server machine>
Next, the functional configuration of the server machine 90 will be described with reference to FIG. The server machine 90 is provided in each electronic blackboard 2, and any electronic blackboard 2 can serve as a server machine. Therefore, the server machine 90 has a communication control unit 70 and a data management unit 80.

<< Functional Configuration of Communication Control Unit 70 >>
Next, the functional configuration of the communication control unit 70 will be described. The communication control unit 70 communicates with the communication control unit 60 in the client machine 20 in the same electronic blackboard 2 and communicates with the communication control unit 60 in the client machine 20 in another electronic blackboard 2 via the communication network 9. To control. The data management unit 80 manages operation data, image data, and the like.

  More specifically, the communication control unit 70 includes a remote connection request receiving unit 71, a remote connection result transmitting unit 72, a remote image receiving unit 73, a remote image transmitting unit 74, a remote operation receiving unit 75, and a remote operation transmitting unit. 76.

  Among these, the remote connection request receiving unit 71 receives a remote sharing processing start request from the remote start processing unit 61 or receives a participation request for remote sharing processing from the remote participation processing unit 62 via the communication network 9. .

  The remote connection result transmission unit 72 transmits the result of the remote sharing processing start request to the remote start processing unit 61 or transmits the result of the participation request for the remote sharing processing to the remote participation processing unit 62 via the communication network 9. .

  The remote image receiving unit 73 receives the image data (data of the output image (C)) from the remote image transmitting unit 63 and transmits it to the remote image processing unit 82 described later. The remote image transmission unit 74 receives image data from the remote image processing unit 82 and transmits this image data to the remote image reception unit 64.

  The remote operation reception unit 75 receives various operation data from the remote operation transmission unit 65 and transmits it to the remote operation processing unit 83 described later. The remote operation transmission unit 76 receives operation data from the remote operation processing unit 83 and transmits the operation data to the remote operation reception unit 66.

<< Functional Configuration of Data Management Unit 80 >>
Next, the functional configuration of the data management unit 80 will be described. The data management unit 80 includes a remote connection processing unit 81, a remote image processing unit 82, a remote operation processing unit 83, an operation composition processing unit 84, and a page processing unit 85. Further, the data management unit 80 includes a passcode management unit 810, a participating site management table 820, an image data management unit 830, an operation data storage unit 840, and a page data storage unit 850.

  Among these, the remote connection processing unit 81 starts the remote sharing process and ends the remote sharing process. The remote connection processing unit 81 also receives the license information received by the remote connection request receiving unit 71 from the remote start processing unit 61 together with the remote sharing processing start request or the remote sharing processing unit 62 from the remote sharing processing participation request. Based on the received license information, it is confirmed whether there is a license and whether it is within the license period. Further, the remote connection processing unit 81 confirms whether or not the participation request from the other electronic blackboard 2 as the client machine exceeds the predetermined number of possible participations.

  Further, the remote connection processing unit 81 has the same passcode that is sent when another electronic blackboard 2 has requested to participate in remote sharing processing as the passcode managed by the passcode management unit 810. It is determined whether or not there is the same, and if they are the same, participation in the remote sharing process is permitted. This passcode is issued by the remote connection processing unit 81 when a new remote sharing process is started, and the user of the electronic blackboard 2 as a participating device who intends to participate in the remote sharing process can obtain the electronic code as the hosting device. It is transmitted from the user of the blackboard 2 by telephone or e-mail. As a result, the user of the electronic blackboard 2 as a participating device who intends to participate in the remote sharing process inputs the pass code to the electronic blackboard 2 and requests participation, thereby permitting participation. Note that the user's usability is given priority over security, and the confirmation of the passcode may be omitted only by confirming the license status.

  In addition, the remote connection processing unit 81 uses the participation site information included in the participation request sent from the remote participation processing unit 62 of the electronic blackboard 2 as the participation device via the communication network 9 to the participation site of the server 90. Store in the management table 820. Thereby, in the electronic blackboard 2 as the hosting device, the participation base management table having the same contents is managed by both the client machine 20 and the server machine 90. Further, the remote connection processing unit 81 reads out the remote site information stored in the participating site management table 820 and sends it to the remote start processing unit 61 of the electronic blackboard 2 as the server machine via the communication network 9.

  The remote image processing unit 82 receives image data from a video output device (such as a notebook PC 6) connected to a client device of each electronic blackboard 2 (including a client device of its own electronic blackboard that is the hosting device) during remote sharing processing. (Output image (C)) is received, and the display order of the images to be remotely shared is determined in the order of time of arrival at the server device 90 of the electronic blackboard 2 that is the host device. In addition, the remote image processing unit 82 refers to the participation base management table 820, and adds to all the client machines 20 of the electronic blackboard 2 participating in the remote sharing process (including the client machine of its own electronic blackboard that is the host device). The image data is transmitted in the determined order via the communication control unit 70 (remote image transmission unit 74).

  The remote operation processing unit 83 performs various types of operation data (stroke images (B, B, B) drawn on the client machine of each electronic blackboard 2 (including the client machine of its own electronic blackboard which is the hosting apparatus) during the remote sharing process. ) And the like, and the display order of the images to be remotely shared is determined in the order of time of arrival at the server 90 of the electronic blackboard 2 that is the host device. The various operation data is the same as the various operation data described above. Further, the remote operation processing unit 83 refers to the participating site management table 820 and operates the operation data on all the client machines 20 of the electronic blackboard 2 (including the client machine of its own electronic blackboard which is the hosting apparatus) during the remote sharing process. Send.

  The operation synthesis processing unit 84 synthesizes the operation data of each electronic blackboard 2 output from the remote operation processing unit 83, stores the operation data as a result of this synthesis in the operation data storage unit 840 and the remote operation processing unit 83. Return to. The operation data is transmitted from the remote operation transmission unit 76 to each of the electronic blackboard client machine that is the hosting apparatus and the electronic blackboard client machine that is the participating apparatus. Such an image is displayed.

表９は、操作データ記憶部８４０に記憶されている操作データの概念図の一例である。操作データは、SEQ(Sequence)、操作データの操作名、操作データの送信元である電子黒板２のＩＰアドレス（クライアント機又はサーバ機のPort No.を含む）、操作データの送信先である電子黒板２のＩＰアドレス（クライアント機又はサーバ機のPort No.を含む）、操作データの操作種類、操作データの操作対象、及び、操作データの内容を示すデータが関連付けられて示されている。

Table 9 is an example of a conceptual diagram of operation data stored in the operation data storage unit 840. The operation data includes SEQ (Sequence), the operation name of the operation data, the IP address of the electronic blackboard 2 that is the transmission source of the operation data (including the port number of the client machine or the server machine), and the electronic that is the transmission destination of the operation data. The IP address of the blackboard 2 (including the port number of the client machine or the server machine), the operation type of the operation data, the operation target of the operation data, and data indicating the contents of the operation data are shown in association with each other.

  For example, in SEQ1, when a stroke is drawn on the client machine (Port No .: 50001) of the electronic blackboard (IP address: 192.0.0.1) as the hosting device, the electronic blackboard (IP address: 192.0. 0.1) indicates that operation data has been sent to the server machine (Port No .: 50000). In this case, the operation type is “STROKE”, the operation object is the page data ID “p005”, and the data indicating the contents of the operation data is data indicating the stroke. In SEQ2, the client computer of the electronic blackboard (IP address: 192.0.0.1) that is the participating device from the server device (Port No .: 50000) of the electronic blackboard (IP address: 192.0.0.1) that is the host device. (Port No .: 50001) indicates that operation data has been sent.

  The operation composition processing unit 84 performs composition in the order in which the operation data is input to the operation composition processing unit 84. Therefore, if the communication network 9 is not congested, remote sharing is performed in the order of strokes by the users of the electronic blackboards 2. A stroke image (B) is displayed on the displays 3 of all the electronic blackboards 2 being processed.

  The page processing unit 85 has the same function as the page processing unit 37 in the image processing unit 30 of the client machine 20, and the server machine 90 also converts the page data shown in Tables 1 to 3 and FIGS. The data is stored in the page data storage unit 850. Note that the page data storage unit 850 has the same contents as the page data storage unit 300 in the image processing unit 30, and therefore the description thereof is omitted.

<Processing or operation of electronic blackboard 2>
Subsequently, processing or operation of the present embodiment will be described with reference to FIGS. 9 and 10. 9 and 10 are examples of sequence diagrams showing the processing of each electronic blackboard.

  In the embodiment shown in FIGS. 9 and 10, the electronic blackboard 2a serves as a hosting device (server machine and client machine) that hosts the remote sharing process, and the electronic blackboards 2b and 2c participate in the remote sharing process. A case where it plays a role as a device (client machine) will be described. First, a process for the electronic blackboards 2b and 2c to participate in the remote sharing process will be described.

S21: By receiving a user instruction for enabling the server function from the remote start processing unit 61 in the client machine 20 of the same electronic blackboard 2a, the remote connection request receiving unit 71 in the server machine 90 of the electronic blackboard 2a that is the sponsoring device, Enable server capability flags.
S22: The UI image generation unit 33 in the client machine 20 of the electronic blackboard 2a generates connection information for establishing a connection with the electronic blackboard 2a, and the video composition unit 28 transmits the display composition unit 36 from the UI image generation unit 33. The connection information obtained through the display is displayed on the display 3a. This connection information includes the host device's IP address and the passcode generated for the current remote sharing process. Then, the connection information is transmitted to the users of the electronic blackboards 2b and 2c by telephone or electronic mail by the user of the electronic blackboard 2a. If there is the connection destination management table 440, even if the connection information does not include the IP address of the hosting device, the participating device can make a participation request.
S23, S24: Next, the electronic blackboards 2b, 2c accept input of connection information from each user, and the remote participation processing unit 62 in the client machine 20 of each of the electronic blackboards 2a, 2b performs the connection based on the IP address of the connection information. Then, a pass code is transmitted to the remote connection request receiving unit 71 in the server machine 90 of the electronic blackboard 2a via the communication network 9 to make a participation request.
S25: The remote connection processing unit 81 in the server machine 90 authenticates the passcode received from each client machine 20 of the electronic blackboards 2b and 2c using the passcode managed by the passcode management unit 810.
S26, S27: Then, the remote connection result transmitting unit 72 notifies the authentication result to the client machine 20 of each electronic blackboard 2b, 2c.
S28, S29: If it is determined by the authentication in step S25 that the electronic blackboard is a legitimate electronic blackboard, communication of remote sharing processing between the electronic blackboard 2a that is the sponsoring device and the electronic blackboards 2b and 2c that are the participating devices is performed. Once established, the remote participation processing unit 62 in the client machine 20 of each of the electronic blackboards 2b and 2c validates a sharing flag indicating whether or not there is a remote sharing process with another electronic blackboard.

Next, output image (C) processing in remote sharing processing will be described.
S30: In the embodiment shown in FIG. 9, the image acquisition unit 31 in the client machine 20 to which the user's notebook PC 6b is connected generates image data that is a display screen of the notebook PC 6b using the video provided by the notebook PC 6b. The video composition unit 28 displays an image (output image (C)) on the display 3b of the electronic blackboard 2b via the display composition unit 36.
S31: Then, the remote image transmission unit 63 in the client machine 20 of the electronic blackboard 2b transmits the image data to the remote image reception unit 73 in the server machine 90 of the electronic blackboard 2a that is the hosting device.
S32: Next, the video composition unit 28 in the client machine 20 of the electronic blackboard 2a performs the remote image processing unit 82 and the remote image transmission unit 74 in the server machine 90, and the remote image reception unit 64 and the display composition unit 36 in the client machine 20. The image data (output image (C)) sent via is displayed on the display 3a.
S33: Then, the remote image transmission unit 74 in the server machine 90 of the electronic blackboard 2a serving as the hosting device sends an image to the remote image reception unit 64 in the client machine 20 of the electronic blackboard 2c other than the electronic blackboard 2b that provided the image data. Send data.
S34: The video composition unit 28 in the client machine 20 of the electronic blackboard 2c displays the image data (output image (C)) sent via the remote image reception unit 64 and the display composition unit 36 on the display 3c ( Step S34).

Next, the stroke image (B) process in the remote sharing process will be described.
S41: The user is drawing a stroke image (B) on the electronic blackboard 2b using the electronic pen 4b.
S42: Next, as shown in FIG. 8, the display composition unit 36 in the client machine 20 of the electronic blackboard 2b performs the UI image (A), the output image (C), and the background image (D). The stroke image (B) is synthesized, and the video composition unit 28 displays the synthesized image (A, B, C, D) on the display 3b of the electronic blackboard 2b.
S43: The remote operation transmission unit 65 in the client machine 20 of the electronic blackboard 2b transmits the data of the stroke image (B) to the remote operation reception unit 75 in the server machine 90 of the electronic blackboard 2a that is the host device.
S44: Next, the operation composition processing unit 84 in the server machine 90 of the electronic blackboard 2a sends the operation data (here, the stroke image (B)) sent via the remote operation receiving unit 75 and the remote operation processing unit 83. Are described as only the data of the data) and returned to the remote operation processing unit 83. Thereby, the data of the stroke image (B) is transmitted to the client machine 20 of the electronic blackboard 2a via the remote operation transmission unit 76 in the server machine 90 of the electronic blackboard 2a and the remote operation reception unit 66 in the client machine 20 of the electronic blackboard 2a. Is transmitted to the display composition unit 36. Therefore, the display combining unit 36 combines the stroke image (B) with the UI image (A), the output image (C), and the background image (D). Then, the video composition unit 28 displays the composite image (A, B, C, D) synthesized by the display composition unit 36 on the display 3a.
S45: Further, the remote operation transmission unit 76 in the server 90 of the electronic blackboard 2a uses the operation data synthesized by the operation synthesis processing unit 84 (here, only the data of the stroke image (B) is described) as the stroke image. The data of (B) is transmitted to the remote operation receiving unit 66 in the client machine 20 of the electronic blackboard 2c other than the electronic blackboard 2b that provided the data.
S46: Next, the display composition unit 36 in the client machine 20 of the electronic blackboard 2c generates a composite image (A, B, C, D) in the same manner as in step S43, and the video composition unit 28 displays the electronic blackboard 2c. The composite image (A, B, C, D) is displayed on the display 3c.

Next, a process for terminating the remote sharing process communication will be described. In the embodiment shown in FIG. 10, processing for the electronic blackboard 2c to finish participation is shown.
S47: The remote participation processing unit 62 in the client machine 20 of the electronic blackboard 2c makes a participation termination request to the remote connection request receiving unit 71 in the server machine 90 of the electronic blackboard 2a as the hosting device.
S48: Then, the remote connection processing unit 81 in the server 90 of the electronic blackboard 2 uses the participation base management table 820 and the name of the base where the electronic blackboard 2 that requested the end of participation is installed and the electronic blackboard 2 The IP address is deleted, and the remote connection result transmission unit 72 instructs the remote participation processing unit 62 in the client machine 20 of the electronic blackboard 2c to end the participation.
S49: As a result, the remote participation processing unit 62 in the client machine 20 of the electronic blackboard 2c disconnects the communication, thereby performing the participation termination process.

  In the embodiment shown in FIGS. 9 and 10, the client machine provides the image frame, but the server machine may transmit the image frame. In this embodiment, the client machine that provides the image frame transmits the stroke image (B). However, another client machine or server machine may transmit the stroke image (B). Furthermore, in this embodiment, a client machine that does not provide an image frame transmits a position instruction event, but a server machine that does not provide an image frame may transmit position instruction event information.

<< Symbol recognition process >>
Next, the symbol recognition process will be described with reference to FIGS. 11 and 12. FIG. 11 is an example of a flowchart showing a procedure for recognizing a symbol from the coordinates of visual information handwritten by the character recognition unit 321, and FIG. 12 shows an example of a symbol drawn on the display 3 and its recognition result.

  First, the stroke processing unit 32 acquires the coordinates of the visual information received as an event by the event processing unit 25 (S10).

  Next, the character recognition unit 321 analyzes the coordinates of the visual information (S20).

  The character recognition unit 321 determines which of text, a figure (circle or square), or a stroke is drawn based on the analysis result of the coordinates of the visual information (S30). For example, if a symbol is recognized and a suitable symbol can be identified with a certain degree of accuracy, it is determined to be a text. If a graphic is recognized and matched with a graphic with a certain degree of accuracy, it is determined to be a graphic. If neither symbol nor figure is recognized, it is determined as a stroke.

  If it is not text (No in S30), the stroke processing unit 32 creates a stroke or a graphic (S40). That is, it is drawn and sent to the display composition unit 36.

  Next, the stroke processing unit 32 stores the stroke and the figure on a memory such as the RAM 103 and sends them to the display composition unit 36 (S43). A stroke and a figure are displayed on the stroke image (B) of FIG.

  Next, the serialization / deserialization unit 322 serializes strokes and graphics (S45).

  If it is text (Yes in S30), the character recognition unit 321 creates text and recognition process candidates in descending order of accuracy (S50). Here, symbol recognition processing may be performed again. This text is also displayed in the stroke image (B) of FIG.

  Next, the character recognizing unit 321 saves the text and the recognition processing candidate in a memory such as the RAM 103 and sends it to the display combining unit 36 (S60).

  Since the display synthesis unit 36 displays text, strokes, and graphics on the display 3, when the user draws the text, the text with the highest accuracy is displayed.

  For example, as shown in FIG. 12A, the user has handwritten the character “A”, but the character recognition unit 321 recognizes that the character is “Previous” as shown in FIG. . That is, it is erroneously converted. In this embodiment, since the recognition process candidate is generated in addition to “Before”, the user can later select a correct recognition result.

  Next, the serialization / deserialization unit 322 serializes the text and the recognition process candidate (S70). That is, the text and the recognition process candidate are converted into a set of data. The serialization will be described with reference to FIG.

  Next, the serialization / deserialization unit 322 sends the serialized stroke and figure, text, and recognition process candidate to the communication control unit 60 (S80). The transmission procedure may be the same as steps S41 to S45 in FIG. When the electronic blackboard 2 that has recognized the symbol is a client machine, the serialized symbol data is transmitted by the remote operation transmission unit 65 of the communication control unit 60 to the remote operation reception unit 75 of another electronic blackboard 2 via the network 9. When the electronic blackboard 2 that has recognized the symbol serves as both the server machine and the client machine, the remote operation receiving unit 75 receives the serialized symbol data via the remote operation transmitting unit 65.

  Thereby, the serialized symbol data can be transmitted to another electronic blackboard 2. Other electronic blackboards 2 can display text without performing recognition processing, display recognition processing candidates as described below, and select other characters from the recognition processing candidates.

<< Symbol display by other electronic blackboard 2 >>
Next, a procedure when another electronic blackboard 2 displays text will be described with reference to FIG. FIG. 13 is an example of a flowchart illustrating a procedure in which the stroke processing unit 32 saves text including recognition process candidates.

  First, the remote operation receiving unit 75 of the communication control unit 70 of another electronic blackboard 2 receives the serialized data (S10). The remote operation processing unit 83 acquires the serialized data via the remote operation receiving unit 75 and transmits it to the client machine 20 of the electronic blackboard 2 other than the transmission source via the remote operation transmitting unit 76. Thereby, the stroke processing unit 32 of the image processing unit 30 acquires the serialized data.

  Next, the serialization / deserialization unit 322 deserializes the serialized data (S20).

  The stroke processing unit 32 determines whether the deserialized data is text (S30). As described with reference to FIG. 11, data to be serialized includes a stroke and a graphic or text (recognition processing candidate). The determination of whether or not the text is made is performed with reference to identification information for identifying the object.

  In the case other than the text (No in S30), the stroke processing unit 32 creates a stroke or a graphic (S40). The stroke processing unit 32 draws the coordinate data transmitted as the operation data (STROKE) to create a stroke, and creates graphic image data indicated by the graphic code.

  In the case of text (Yes in S30), the stroke processing unit 32 creates a text and a recognition process candidate (S50). Since the symbol has already been recognized, text image data is created using the deserialized text and the character code of the recognition process candidate.

  Next, the character recognition unit 321 stores the stroke, figure, text, and recognition process candidate in the RAM 103 (S60). By deserialization, the text and the recognition process candidate can be restored as if the other electronic blackboard 2 recognized the symbol.

  Thereafter, the display composition unit 36 causes the display 3 to display text and recognition process candidates, strokes or figures. Assume that another electronic blackboard 2 displays text as shown in FIG. For example, the user touches the text to be converted (front) with the electronic pen 4a or the like for a predetermined time. Thereby, recognition process candidates are displayed as shown in FIG. When the user selects the correct text (a) from the recognition process candidates, the selected text is displayed on the display 3 as shown in FIG.

  Therefore, according to the electronic blackboard 2 of the present embodiment, not only the text but also the recognition processing candidates can be shared with other electronic blackboards 2, and the user of the other electronic blackboard 2 can select the correct text from the recognition processing candidates.

<< Serialization, Deserialization >>
FIG. 14 is an example of a diagram for explaining serialization and deserialization. Serialization refers to converting the internal state of an object into a byte stream. Deserialization is to reproduce the same object from the byte stream again.

  In the electronic blackboard 2, these are data handled by the program, and it is difficult to store and transmit the data as it is. Serialization is a process of converting data handled by a program into a format that can be saved as a file. Therefore, transmission becomes possible.

  In addition, the serialized data transmitted is converted into data handled by the original program by deserialization.

  The serialized text and recognition process candidate shown in FIG. 14 are transmitted to the electronic blackboard 2. By deserializing the electronic blackboard 2 as the transmission destination, it is possible to restore the state immediately after the electronic blackboard 2 as the transmission source recognizes the symbol as if the own device recognized the symbol.

<Preferred application example>
The best mode for carrying out the present invention has been described above with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the scope of the present invention. And substitutions can be added.

  For example, the display of the electronic blackboard 2 is not limited to a flat panel type, and may be formed by an image projected by a projector. In this case, the coordinates of the hand H and the electronic pen 4 may be acquired by a camera or the like, or may be acquired by a time difference between arrival times of ultrasonic waves emitted from the electronic pen at different measurement points.

  1 and the like, the video conference terminal 7 is illustrated as a device different from the electronic blackboard 2, but the function of the video conference terminal 7 may be incorporated in the electronic blackboard 2. That is, the electronic blackboard 2 has a camera function, and displays an image captured by the camera on the display 3 or transmits it to another electronic blackboard 2.

  In the present embodiment, the recognition and reconversion of symbols by the electronic blackboard 2 have been described. However, the recognition and reconversion of symbols may be performed by an information processing apparatus that does not have the function as the electronic blackboard 2.

1 image processing system 2 electronic blackboard 3 display 4 electronic pen 6 notebook PC
7 Video conference terminal 20 Client machine 25 Event processing unit 32 Stroke processing unit 36 Display composition unit 37 Page processing unit 90 Server machine 321 Character recognition unit 322 Serialization / deserialization unit

Japanese Patent No. 3534469

Claims (6)

An information processing device for displaying information on a display device,
Indicated position detecting means for detecting an indicated position with respect to a display surface of information;
Visual information generating means for generating visual information based on the indicated position detected by the indicated position detecting means;
Symbol recognition means for recognizing a symbol formed by one or more pieces of visual information generated by the visual information generation means and determining a recognition candidate for the symbol;
Data conversion means for converting the symbol and the recognition candidate into transmittable data;
An information processing apparatus comprising: a transmission unit configured to transmit the symbol converted to data that can be transmitted by the data conversion unit and the recognition candidate to another information processing apparatus.   The information processing apparatus according to claim 1, wherein the data conversion unit serializes not only the symbol but also the recognition candidate to convert the symbol and the recognition candidate in a format handled by a program into transmittable data. Receiving means for receiving the symbol and the recognition candidate converted into transmittable data;
The information processing apparatus according to claim 1, wherein the data conversion unit converts the symbol and the recognition candidate converted into transmittable data received by the receiving unit into data of a format handled by a program.   The information according to claim 3, wherein the data conversion unit converts the symbol and the recognition candidate into data of a format handled by a program by deserializing the symbol and the recognition candidate converted into transmittable data. Processing equipment. There is an image processing system in which two or more information processing devices that display information on a display device are connected to be able to communicate with each other,
The first information processing apparatus includes designated position detection means for detecting a designated position on the information display surface;
Visual information generating means for generating visual information based on the indicated position detected by the indicated position detecting means;
Symbol recognition means for recognizing a symbol formed by one or more pieces of visual information generated by the visual information generation means and determining a recognition candidate for the symbol;
First data conversion means for converting the symbol and the recognition candidate into transmittable data;
Transmission means for transmitting the symbol converted to data that can be transmitted by the data conversion means and the recognition candidate to another information processing apparatus,
The second information processing apparatus
Receiving means for receiving the symbol and the recognition candidate converted into transmittable data;
An image processing system comprising: second data conversion means for converting the symbol and the recognition candidate converted into transmittable data received by the receiving means into data in a format handled by a program. An information processing device that displays information on a display device,
Indicated position detecting means for detecting an indicated position with respect to a display surface of information;
Visual information generating means for generating visual information from the indicated position detected by the indicated position detecting means;
A symbol recognizing unit for recognizing a symbol formed by one or more pieces of visual information generated by the visual information generating unit and determining a recognition candidate for the symbol;
Data conversion means for converting the symbol and the recognition candidate into transmittable data; and
The program for functioning as a transmission means which transmits the said symbol converted into the data which can be transmitted by the said data conversion means, and the said recognition candidate to another information processing apparatus.

Images