JP5440926B2 - Information processing system and program thereof - Google Patents

Description

  The present invention relates to an information processing system capable of inputting information using a coded pattern indicating position coordinates, and more particularly to information processing after converting the position coordinates of a coded pattern into position coordinates on a display screen of a computer device. .

  In recent years, electronic pens that digitize written information have been developed, and “Anoto pen” developed by Swedish company Anoto is known as a representative example. Anotopen is used together with special paper on which a dot pattern indicating a position coordinate patterned by a predetermined algorithm is printed. Anotopen has a small camera for imaging the dot pattern printed on the dedicated paper at the pen tip, a processor for calculating the position coordinates on the dedicated paper from the captured dot pattern, and the calculated position coordinates etc. It is equipped with a data communication unit that transmits to. When the user writes characters on the special paper with an anotopen or puts a check mark on the image designed on the special paper, the small camera captures the dot pattern printed on the special paper as the pen moves. The entry information such as characters and images written by the user is recognized from the continuous position coordinates calculated by the processor. Then, this entry information is transmitted from the Anotopen by the data communication unit to a nearby terminal device such as a personal computer or a mobile phone (for example, see Patent Document 1).

  In addition, using such an electronic pen, a computer, a projector, and a screen, when the screen is filled in or scanned with the electronic pen, the computer performs a predetermined process accordingly and projects the computer display screen onto the screen with the projector. The system which performs is proposed (for example, patent documents 2-5). The electronic pen reads the dot pattern formed on the screen, and the computer analyzes the position coordinates based on the information transmitted from the electronic pen and performs processing according to the position coordinates. Even if an electronic pen without ink is used, if you trace the screen with an electronic pen, you can project an image with lines drawn from the projector, control the image, and operate software. It is supposed to be possible.

  Also, the computer screen is projected on the whiteboard by the projector, and the coordinates entered with the markers on the whiteboard are read with the coordinate reader using ultrasonic or infrared rays and sent to the computer, so that the user clicks on the computer screen. In a system capable of calibrating, it is also proposed to calibrate the coordinates of the whiteboard and the coordinates of the personal computer screen by first projecting a calibration point by a projector and instructing the point with a marker (patent) Reference 6).

  In addition, it has been proposed that a transparent sheet on which a dot pattern is printed is pasted on a computer display and handwritten on the transparent sheet with an electronic pen so that the dot pattern can be read with the electronic pen and handwritten on the computer. (For example, Patent Document 7). Furthermore, in order to calibrate the position coordinates in the computer display and the position coordinates in the dot pattern of the transparent sheet attached to the display, it is also proposed to read the dot pattern near the four corners of the display with an electronic pen. (Patent Document 8).

Japanese translation of PCT publication No. 2003-511761 JP 2008-152755 A JP 2002-149331 A JP 2008-155421 A Special table 2003-508831 gazette JP 2007-233999 A JP 2003-256122 A JP 2008-165384 A

  In such a system, even if the position coordinates in the dot pattern obtained from the processing of the electronic pen can be converted into the position coordinates of the display by calibration, the information from the electronic pen is selected depending on the running application. Must be processed.

  Accordingly, the present invention provides an information processing system for converting position coordinates of a screen or a transparent sheet on which a coding pattern indicating position coordinates is formed into position coordinates on a display screen of a display unit of a computer device. It is an object to select and process information obtained by reading a digitized pattern.

An information processing system according to the present invention receives a screen on which a coding pattern indicating position coordinates is formed, a reading unit that reads the coding pattern, and information on the coding pattern transmitted from the reading unit, An information processing system comprising a computer device that performs processing according to the information, and a projector that receives an image signal from the computer device and projects an image on the screen,
The computer apparatus displays on the display means a receiving means for receiving information on the coding pattern transmitted from the reading means , and a movable window having an enlarge button, a reduce button, and a writing area. converts the position coordinates of the coded pattern contained in the received information to the position coordinates on the display screen of the display unit, the coordinate conversion position coordinates if the coordinates transformed coordinates is in the writing area Based on the drawing button, the stroke is drawn in the writing area and saved in a file. When the coordinate position is recognized as indicating the enlargement button or the reduction button, the writing area is determined according to the designated button. And zooming in and out the drawn stroke with the display magnification of the writing area and Characterized in that it comprises processing means for displaying in accordance with the location, is stored in the display magnification and the receiving writing to the writing area when the coordinate transformed same file by drawing a stroke on the basis of the position coordinates in the position And

According to this information processing system, when the coded pattern formed on the screen is read by the reading means, the reading means transmits information on the coded pattern to the computer device. Computer system receives the information about the coded pattern from the reading unit, performs a process corresponding to the information, and transmits the image signal to the projector for projecting an image onto a screen. In such an information processing system, the processing means displays a movable window having an enlargement button, a reduction button, and a writing area on the display means, and the position coordinates of the coding pattern included in the information received by the receiving means. After converting to the position coordinates on the display screen of the display means, if the position coordinates that have been converted are within the writing area, a stroke is drawn in the writing area based on the position coordinates that have been converted to a file. When the saved position coordinate is recognized as indicating the enlargement button or the reduction button, the writing area is enlarged or reduced according to the designated button, and the drawn stroke is written. Display according to the display magnification and position of the area, and write at the display magnification and position. When you draw a stroke based on when receiving a write to see area coordinates transformed coordinates Suruga stored in the same file, the coordinate conversion position coordinates are not in the write region, as a stroke in the write-in area It can be prevented from taking in.

Alternatively, the information processing system according to the present invention receives a transparent sheet on which a coding pattern indicating position coordinates is formed, a reading unit that reads the coding pattern, and information related to the coding pattern transmitted from the reading unit. An information processing system including a computer device that performs processing according to the information,
The transparent sheet is affixed to a display screen of a display unit of the computer device, and the computer device includes a reception unit that receives information related to a coding pattern transmitted from the reading unit, an enlargement button, a reduction button, and displayed on the display means a moveable window having a writing area, and converts the position coordinates of the coded pattern contained in the received information by the receiving means to the position coordinates on the display screen of the display unit, which is the coordinate transformation coordinates are stored in a file to draw the stroke on the basis of the coordinate conversion position coordinates on the writing area in some cases in the write area, the coordinate conversion position coordinates of the enlargement button or the reduction button If it is recognized that the instruction is to be given, the writing area is enlarged according to the designated button. Is reduced, and the drawn stroke is displayed in accordance with the display magnification and position of the writing area, and when the writing to the writing area at the display magnification and position is received, the stroke is converted based on the coordinate position converted. And processing means for drawing and saving the same file .

According to this information processing system, when the encoded pattern formed on the transparent sheet is read by the reading unit, the reading unit transmits information on the encoded pattern to the computer device. The computer apparatus receives information on the coding pattern from the reading unit, performs processing according to the information, and transmits an image signal to a projector that projects an image on a transparent sheet. In such an information processing system, the processing means displays a movable window having an enlargement button, a reduction button, and a writing area on the display means, and the position coordinates of the coding pattern included in the information received by the receiving means. After converting to the position coordinates on the display screen of the display means, if the position coordinates that have been converted are within the writing area, a stroke is drawn in the writing area based on the position coordinates that have been converted to a file. When the saved position coordinate is recognized as indicating the enlargement button or the reduction button, the writing area is enlarged or reduced according to the designated button, and the drawn stroke is written. Display according to the display magnification and position of the area, and write at the display magnification and position. When you draw a stroke based on when receiving a write to see area coordinates transformed coordinates Suruga stored in the same file, the coordinate conversion position coordinates are not in the write region, as a stroke in the write-in area It can be prevented from taking in.

  In the information processing system, when the processing means receives the information by the receiving means and determines that the coordinate conversion function for the coordinate conversion is not defined, the processing means obtains the coordinate conversion function. It may be configured to start the calibration process. Thus, even when the coded pattern is read by the reading means, when the information received by the computer device cannot be properly recognized, the coordinates for converting the position coordinates of the coded pattern into the position coordinates on the display screen of the display means Since the calibration process for obtaining the conversion function is started, the information received by the computer apparatus can be appropriately recognized.

A program according to the present invention causes a computer to function as a computer device in the information processing system. By installing this program in a computer device, the above computer device can be realized and an information processing system can be configured.

  According to the present invention, highly accurate calibration processing is performed between the position coordinates of a screen or a transparent sheet on which a coding pattern indicating position coordinates is formed and the position coordinates on the display screen of the display unit of the computer device. be able to.

It is a system configuration figure of the information processing system in a 1st embodiment. It is explanatory drawing which shows the relationship between the arrangement | positioning of the dot in a dot pattern, and the value converted. (A) shows a dot pattern typically, and (b) is a figure showing an example of information corresponding to it. It is the schematic which shows the structure of an electronic pen. It is a functional block diagram of a computer apparatus. It is a flowchart of the calibration starting process in 1st Embodiment. It is a flowchart of the calibration process in 1st Embodiment. It is a figure which shows the image of the mark for a calibration projected on the screen in 1st Embodiment. It is a figure which shows the position coordinate of the display reference point on the display screen of a computer apparatus. It is a figure shown about the same recognition range as the position coordinate of the recognition reference point on a screen. It is a figure which shows the image projected on the screen. It is a figure which shows the image which the display screen which the handwriting input application started starts projecting on the screen. It is a flowchart of the process which writes and inputs a handwritten character etc. with an electronic pen in the writing area of a handwriting input application. It is a figure which shows a mode that it writes in the writing area projected on the screen shown in FIG. 12 with an electronic pen. It is a figure which shows a mode that the window of a handwriting input application is moved from the position shown in FIG. 14 by operation of an electronic pen. It is a figure which shows a mode that it writes in the writing area projected on the screen shown in FIG. 15 with an electronic pen. It is a figure which shows a mode that a writing area is expanded by tapping the enlargement button on the screen shown in FIG. 16 with an electronic pen. It is a figure which shows a mode that the stroke is further written in the enlarged writing area projected on the screen shown in FIG. 17 with an electronic pen. It is a figure which shows a mode that the area | region displayed in a writing area | region is changed and the stroke is continuously written in the window projected on the screen shown in FIG. It is a figure which shows a mode that the writing area | region shown in FIG. 19 was displayed entirely with the electronic pen. It is a system configuration figure of the information processing system in a 2nd embodiment.

  Hereinafter, a preferred information processing system of the present invention will be described in detail with reference to the drawings.

<First Embodiment>
[System configuration of information processing system]
FIG. 1 is a system configuration diagram of an information processing system 10 in the first embodiment. As shown in FIG. 1, the information processing system 10 in the first embodiment includes an electronic pen 1 (reading unit) used by a user, a computer device 2 that receives and processes entry information from the electronic pen 1, and The projector 3 is configured to project the display content of the display (display means) 26 of the computer device 2 onto the screen 4.

[screen]
First, the screen 4 will be described. The screen 4 has a sheet 402 having a dot pattern (coded pattern), which will be described later, formed on the entire surface by printing on the front surface of a magnet board (white board) 401 that is hard enough to apply sufficient pressure to the electronic pen 1 from above. They are pressed with a magnet, or pasted with glue, adhesive or adhesive. A wall surface of a room may be used instead of the magnet plate 401. Further, instead of the paper 402, a plastic sheet on which a dot pattern is printed may be used, or the dot pattern may be directly formed on a magnet plate (white board) 401. When the user writes a character or the like on the screen 4 with the electronic pen 1, the electronic pen 1 reads the dot pattern printed on the paper 402 locally and continuously along the movement locus of the pen tip portion. The coordinates of the local position are calculated, and the electronic data (entry information) is transmitted to the computer device 2. On the other hand, the projector 3 receives the image signal from the computer apparatus 2 and projects the same image as the image displayed on the display unit 26 onto the screen 4 in the area 403.

  When the calibration process is executed in the computer apparatus 2, a coordinate input area 403 in which position coordinates corresponding to the display screen of the computer apparatus 2 can be accurately input on the screen 4 by the electronic pen 1 is defined. Definition information is stored in the storage means 25 of the computer apparatus 2.

[Dot pattern]
Next, an Anoto dot pattern (coded pattern) printed on the sheet 402 of the screen 4 will be described. FIG. 2 is a diagram illustrating the relationship between the dots of the dot pattern printed on the paper 402 and the values to which the dots are converted. As shown in FIG. 2, each dot of the dot pattern is associated with a predetermined value depending on its position. In other words, each dot is associated with a value of 0 to 3 depending on which direction the top, bottom, left, or right is shifted from the reference position of the grid (intersection of the vertical line and horizontal line). The value of each dot can be further converted into a first bit value for the X coordinate and a second bit value for the Y coordinate. The position coordinates on the screen 4 are determined by the combination of information thus associated. This dot pattern is printed with an ink containing carbon that absorbs infrared rays.

  FIG. 3A shows an arrangement of dot patterns at a certain position. As shown in FIG. 3A, 6 × 6 dots are arranged within a range of about 2 mm in length and width so that a unique pattern can be obtained regardless of where 6 × 6 dots are taken on the screen 4. Yes. The dot pattern formed by these 36 dots holds relative position coordinates on the paper 402 and a dot pattern address (coded pattern address). FIG. 3B is a diagram in which each dot shown in FIG. 3A is converted into a value associated with the regularity shown in FIG. 2 according to the shift direction from the reference position of the lattice. This conversion is performed by the electronic pen 1 that captures an image of a dot pattern.

[Electronic pen]
Next, the electronic pen 1 will be described. FIG. 4 is a schematic diagram showing the structure of the electronic pen 1. As shown in FIG. 4, the electronic pen 1 includes a pen unit 104, an LED 105, a CMOS camera 106, a pressure sensor 107, a processor 108 including a CPU, a memory 109 such as a ROM and a RAM, A communication unit 111 including a real time clock 110, an antenna, and the like, and a battery 112 are provided. The tip of the pen unit 104 is a pen point unit 103, and the user brings the pen point unit 103 of the electronic pen 1 into contact with the paper 402 of the screen 4 to enter strokes of characters or the like, or tap ( Lightly hitting the screen 4 with the pen tip 103. Here, the first contact of the pen tip 103 of the electronic pen 1 with the paper 402 or the like is called pen-down, and the removal of the pen tip 103 from the contacted (contacted) state is called pen-up. . A trajectory written between the pen-down and pen-up of the electronic pen 1 is one stroke, and characters, figures, etc. are composed of one or a plurality of strokes. Since the electronic pen 1 is used for tapping or writing characters on the screen 4 projected by the projector 3, the pen unit 104 does not have to be filled with ink.

  The battery 112 is for supplying electric power to each component in the electronic pen 1. For example, the battery 112 is configured to turn on / off the electronic pen 1 itself by attaching and detaching a cap (not shown) of the electronic pen 1. May be. The real time clock 110 transmits time information indicating the current time (time stamp) and supplies the time information to the processor 108. The pressure sensor 107 detects the pressure applied through the pen unit 104 from the pen tip unit 103 when the user writes or taps a character or mark on the screen 4 with the electronic pen 1, that is, the writing pressure, and the value is detected by the processor. To 108.

  The processor 108 switches on / off the switches of the LED 105 and the CMOS camera 106 based on the writing pressure data supplied from the pressure sensor 107. That is, when a user writes a character or the like on the screen 4 with the electronic pen 1, the pen pressure is applied to the pen tip portion 103, and when the pressure sensor 107 detects a writing pressure higher than a predetermined value, the processor 108 It is determined that entry has started, and the LED 105 and the CMOS camera 106 are operated. Further, when the user finishes entering one stroke and moves the electronic pen 1 away from the screen 4, the pressure sensor 107 detects pen-up because no writing pressure exceeding a predetermined value is detected.

  The LED 105 and the CMOS camera 106 are attached near the pen tip portion 103 of the electronic pen 1, and an opening 102 is formed in a portion of the housing 101 that faces the LED 105 and the CMOS camera 106. The LED 105 illuminates infrared rays toward the vicinity of the pen tip portion 103 on the screen 4. The region is slightly shifted from the position where the pen tip portion 103 contacts the screen 4. The CMOS camera 106 captures a dot pattern in the area illuminated by the LED 105 and supplies image data of the dot pattern to the processor 108. Here, since carbon absorbs infrared rays, the infrared rays irradiated by the LED 105 are absorbed by the carbon contained in the dots. Therefore, the dot portion has a small amount of infrared reflection, and the portion other than the dot has a large amount of infrared reflection. Therefore, by photographing with the CMOS camera 106, by setting a threshold value based on the difference in the amount of reflected infrared light, it is possible to distinguish the dot region containing carbon from the other regions. Note that the shooting area by the CMOS camera 106 is a range including a size of about 2 mm × about 2 mm as shown in FIG. 3A, and the shooting by the CMOS camera 106 is performed at regular intervals of about 50 to 100 times per second. Is called.

  The processor 108 performs X and Y coordinates (simply “position coordinates”, “position coordinates” on the screen 4 of the stroke (handwriting) to be entered by the user from the dot pattern of the image data supplied by the CMOS camera 106 during the user entry. (Also called “coordinate data”) and dot pattern addresses (coded pattern addresses) unique to the screen 4 are continuously calculated. That is, the processor 108 converts the image data of the dot pattern as shown in FIG. 3 (a) supplied by the CMOS camera 106 into the data array shown in FIG. 3 (b). The data is converted into a Y coordinate bit value, and X, Y coordinate data and a dot pattern address are calculated from the data array by a predetermined calculation method. Then, the processor 108 associates the current time (time stamp: entered time information) transmitted from the real-time clock 110, the writing pressure data, the dot pattern address, and the X and Y coordinate data. Since the 6 × 6 dot pattern on the screen 4 does not overlap in the screen 4, when the user enters characters or the like with the electronic pen 1, it indicates which position on the screen 4 the entered position corresponds to. It can be specified by coordinate calculation by the processor 108.

  The memory 109 stores property information such as a pen ID such as “pen01” for identifying the electronic pen 1, a pen manufacturer number, and a pen software version. Then, the communication unit 111 associates a pen ID (electronic pen identification information), time information (time stamp), writing pressure data, dot pattern address, and X, Y coordinate data, and enters computer information as entry information. 2 to send. Transmission to the computer apparatus 2 by the communication unit 111 is performed immediately and sequentially by wireless transmission such as Bluetooth (registered trademark).

[Computer device]
Next, the computer apparatus 2 will be described. The computer device 2 includes, as hardware, an antenna device capable of data communication with the electronic pen 1, a processor such as a CPU, a memory such as ROM and RAM, a display, a personal computer including a mouse and a keyboard, and the like. . FIG. 5 is a functional block diagram of the computer apparatus 2. Functionally, the computer apparatus 2 includes an input unit 21 such as a mouse and a keyboard, a reception unit 22, a processing unit 24, a storage unit 25, a display unit 26, and a transmission unit 27. Then, the computer device 2 performs a predetermined process based on the entry information received from the electronic pen 1.

  The receiving unit 22 includes an antenna, a receiving circuit, and the like, and receives entry information from the electronic pen 1 and transmits it to the processing unit 24. The transmission unit 27 transmits an image signal used for displaying an image on the display unit 26 to the projector 3 according to an instruction from the processing unit 24. A data transmission method to the projector 3 is a wireless method even if a wired method is used. There may be. The display means 26 is constituted by a display or the like, and displays the contents instructed by the processing means 24.

  The storage means 25 is configured by a memory such as a hard disk, ROM, or RAM. The storage means 25 stores a program for calibrating the position coordinates in the dot pattern (coded pattern) on the screen 4 and the position coordinates on the display screen in the display means 26 of the computer device 2. . With respect to this program, the storage means 25 stores position coordinates such as display reference points and recognition reference points displayed during the calibration process, images of calibration marks, and the like. Further, the storage means 25 also stores a program for processing entry information generated by handwriting on the screen 4 with the electronic pen 1. In the storage unit 25, various applications such as a handwriting input application are installed and stored.

  The processing means 24 is constituted by a processor such as a CPU, and based on the entry information received from the electronic pen 1 by the receiving means 22, displays the entry contents in various forms on the display means 26, or causes the storage means 25 to display it. A stored program for performing calibration processing is activated to perform calibration processing. Further, the processing unit 24 activates and executes an application such as a handwriting input application stored in the storage unit 25. Further, the processing unit 24 causes the transmission unit 27 to transmit to the projector 3 an image signal for causing the projector 3 to project the same image as the image displayed on the display unit 26 in synchronization.

  The projector 3 projects an image similar to the image displayed on the display unit 26 onto the area 403 of the screen 4 based on the image signal received from the transmission unit 27 of the computer apparatus 2.

<Calibration startup process flow>
Next, calibration start processing in the information processing system 10 will be described with reference to FIG.

  FIG. 6 is a flowchart of the calibration activation process. First, the user performs writing or tapping on the paper 402 of the screen 4 using the electronic pen 1. Then, the electronic pen 1 detects contact with the paper 402 when the pressure sensor 107 detects a writing pressure equal to or higher than a predetermined value, and images a dot pattern along the handwriting with the camera 106 while irradiating infrared rays with the LED 105. Then, the processor 108 continuously calculates the position coordinates along the handwriting from the image data of the captured dot pattern, and the position coordinates, the current time transmitted by the real-time clock 110, the writing pressure data (hereinafter referred to as “stroke”). Information ") and the pen ID are transmitted to the computer apparatus 2 immediately and sequentially as entry information (step S101).

  In the computer apparatus 2, when the receiving means 22 receives the entry information (stroke information and pen ID) from the electronic pen 1, the entry information is stored in the storage means 25 (step S201). Then, the processing means 24 checks whether or not a coordinate conversion function is defined (step S202). If the processing unit 24 recognizes that the coordinate conversion function is not defined (step S202: No), it starts a calibration process (step S204, see FIG. 7 described later). Therefore, if the user tries to write on the screen 4 with the electronic pen 1 when the calibration process is not executed, the calibration process is started. On the other hand, when the processing means 24 recognizes that the coordinate conversion function is defined (step S202: Yes), the position coordinates of the received entry information by the dot pattern on the screen 4 are stroked using the coordinate conversion function. By the conversion process, it is converted into position coordinates on the display (display screen) 201 and various processes are performed (step S203, see FIG. 13 described later).

<Calibration process>
Next, the calibration process in step S204 of FIG. 6 will be described with reference to FIGS.

  FIG. 7 is a flowchart of the calibration process. FIG. 8 is a diagram illustrating an image of a calibration mark projected on the screen 4.

  When the processing unit 24 starts the calibration process, the processing unit 24 displays the first calibration mark 601 on the display screen 201 of the display unit 26 and, as shown in FIG. The first calibration mark 601 is projected, and the user is prompted to perform pen-up at the reference point (center of the concentric circle) of the first calibration mark 601 with the electronic pen 1 (step S301). For example, as shown in FIG. 8, the processing means 24 displays an instruction message such as “Please raise when the pen tip touches the center of the target” or output a similar voice instruction.

  Subsequently, the user uses the electronic pen 1 to move the pen tip to the position (center of the concentric circle) where the first calibration mark 601 is projected in the pen-down state, and then performs the pen-up. Then, the electronic pen 1 generates stroke information, associates it with the pen ID, and transmits it as entry information to the computer device 2 (step S401). As described above, when the electronic pen 1 transmits the entry information to the computer device 2, in the computer device 2, the receiving unit 22 receives the entry information (stroke information and pen ID) from the electronic pen 1, and receives the entry information. It memorize | stores in the memory | storage means 25 (step S302). When the receiving means 22 receives the entry information from the electronic pen 1, the processing means 24 extracts the pen-up position coordinates from the entry information and is directed to the position where the first calibration mark 601 is projected. The first recognition reference point on the sheet 402 is stored in the storage means 25 (step S303).

  Further, the processing unit 24 displays the second calibration mark 602 on the display screen 201 of the display unit 26 and causes the projector 3 to project the second calibration mark 602 on the screen 4 ( 8), the user is prompted to perform a pen touch and a subsequent pen-up at the position where the second calibration mark 602 is projected by the electronic pen 1 (step S304).

  Subsequently, when the user uses the electronic pen 1 to pen up after the pen tip comes into contact with the position (center of the concentric circle) where the second calibration mark 602 is projected, the electronic pen 1 displays the stroke information. It is generated, associated with the pen ID, and transmitted as entry information to the computer apparatus 2 (step S402). As described above, when the electronic pen 1 transmits the entry information to the computer device 2, in the computer device 2, the receiving unit 22 receives the entry information (stroke information and pen ID) from the electronic pen 1, and receives the entry information. It memorize | stores in the memory | storage means 25 (step S305).

  The processing unit 24 extracts the pen-up position coordinates from the received entry information, and determines whether or not the pen-up position coordinates are outside the same recognition range of the first recognition reference point (step S306). Here, the same recognition range is a range in which the user can recognize that the same calibration mark has been pen-touched twice or more without being conscious of it, and is displayed on the paper 402 according to the size of the screen 4 or the like. A numerical range (for example, several centimeters) on the printed dot pattern coordinates may be set. When the processing unit 24 determines that the pen-up position coordinate is within the same recognition range of the first recognition reference point (step S306: No), the pen-up position coordinate is adopted as the second recognition reference point. However, if it is determined that the position coordinates of the pen-up are outside the same recognition range of the first recognition reference point (step S306: Yes), the processing unit 24 performs the pen-up. Is stored in the storage unit 25 as the second recognition reference point on the sheet 402 instructed to the position where the second calibration mark 602 is projected (step S307).

  The processing unit 24 then calculates the coordinate values of the first and second recognition reference points on the paper 402 by the dot pattern calculation (the coordinates at the center point of the concentric circles of the projected first and second calibration marks 601 and 602). Value) and the coordinate value at the first and second display reference points that are the center points of the concentric circles of the first and second calibration marks 601 and 602 on the display screen 201 of the display means 26). From the upper position coordinates, a coordinate conversion function for calculating the position coordinates on the display screen 201 in the computer apparatus 2 is obtained and defined, and stored in the storage means 25 (step S308). At the same time, as a result of the calibration process, the processing means 24 determines a coordinate input area 403 on which the position coordinates can be accurately input on the screen 4 by the electronic pen 1 corresponding to the display screen 201, and defines the coordinate area. And stored in the storage means 25.

  Here, a method of obtaining the calculation formula for calibration will be described with reference to FIGS.

FIG. 9 is a diagram showing the position coordinates of the first and second display reference points on the display screen 201 of the computer apparatus 2. As shown in FIG. 9, on the display screen 201 of the display means (display) 26 of the computer apparatus 2, the X axis is the right direction, the Y axis is the downward direction, and the following coordinates are as follows.
Upper left point: Coordinate 202 (Xamin, Yamin),
Lower right corner point: coordinates 203 (Xamax, Yamax),
First display reference point: coordinates 204 (Xa1, Ya1),
Second display reference point: coordinates 205 (Xa2, Ya2),
However, Xamin ≤ Xa1 <Xa2 ≤ Xamax,
Yamin ≤ Ya1 <Ya2 ≤ Yamax,

FIG. 10 is a diagram illustrating the position coordinates of the first and second recognition reference points on the sheet 402 of the screen 4. As the first and second recognition reference points, the first and second display reference points on the display screen 201 of the computer apparatus 2 are projected on the screen 4 and the dot pattern (coded pattern) indicated by the electronic pen 1 is analyzed. The position coordinates based on As shown in FIG. 10, in the coordinate input area 403 projected onto the screen 4 by the projector 3, the X axis is in the right direction, the Y axis is in the downward direction, and the following coordinates are as follows.
Upper left corner point: coordinates 404 (Xb0, Yb0),
Upper left point of coordinate input area 403: coordinates 405 (Xbmin, Ybmin),
Lower right corner point of coordinate input area 403: coordinates 406 (Xbmax, Ybmax),
First recognition reference point: coordinates 407 (Xb1, Yb1),
Second recognition reference point: coordinates 408 (Xb2, Yb2),
However, Xb0 ≤ Xbmin ≤ Xb1 <Xb2 ≤ Xbmax,
Yb0 ≤ Ybmin ≤ Yb1 <Yb2 ≤ Ybmax,

Then, the coordinates 204 (Xa 1, Ya 1) of the first display reference point and the coordinates 205 (Xa 2, Ya 2) of the second display reference point in the display area 201 of the computer device 2 are the first in the dot pattern on the paper 402 of the screen 4. The coordinates correspond to the coordinates 407 (Xb1, Yb1) of the first recognition reference point and the coordinates 408 (Xb2, Yb2) of the second recognition reference point, respectively. Then, the upper left corner point coordinates 405 (Xbmin, Ybmin) and the lower right corner point coordinates 406 (Xbmax, Ybmax) defining the coordinate input area 403 projected on the screen 4 (each vertex of the diagonal in the rectangular coordinate input area 403) Can be expressed as:

[Formula 1] Xbmin = Xb1− (Xa1−Xamin) × (Xb2−Xb1) / (Xa2−Xa1)
[Formula 2] Ybmin = Yb1− (Ya1−Yamin) × (Yb2−Yb1) / (Ya2−Ya1)

[Formula 3] Xbmax = Xb2 + (Xamax−Xa2) × (Xb2−Xb1) / (Xa2−Xa1)
[Formula 4] Ybmax = Yb2 + (Yamax−Ya2) × (Yb2−Yb1) / (Ya2−Ya1)

The coordinate input area 403 can be defined from a rectangular area having the coordinate value as the vertical angle point.

Then, from the position coordinates (Xbn, Ybn) defined by the dot pattern on the paper 402 of the screen 4, the position coordinates (Xan, Yan) on the display screen 201 in the display means 26 of the computer device 2 are expressed by the following equations. Can be sought.

[Formula 5] Xan = Xamin + [Xbn− (Xbmin−Xb0)] × (Xa2−Xa1) / (Xb2−Xb1)
[Formula 6] Yan = Yamin + [Ybn− (Ybmin−Yb0)] × (Ya2−Ya1) / (Yb2−Yb1)

Therefore, the processing means 24 converts the position coordinates (Xbn, Ybn) on the screen 4 to the position coordinates (Xan, Yan) on the display screen 201 of the computer apparatus 2 by the above formula, and performs various processes. Go. Note that a predetermined coefficient for correcting distortion may be applied to the term of the arithmetic expression depending on the angle of the projector 3 with respect to the screen 4.

  After the calibration process, the user writes characters or the like using the electronic pen 1 in the coordinate input area 403 that is defined on the screen 4 so that the position coordinates can be accurately input corresponding to the display screen 201. Then, the electronic pen 1 generates stroke information corresponding to the stroke of the handwriting, and transmits entry information associated with the pen ID to the computer device 2 (step S101). In the computer apparatus 2, when the entry information is received by the receiving means 22, the processing means 24 stores the entry information in the storage means 25 (step S201), and determines that a coordinate conversion function is defined (step S202: Yes), the position coordinates (Xbn, Ybn) on the screen 4 included in the received entry information are converted into the position coordinates (Xan, Ybn) on the display screen 201 of the computer apparatus 2 using the coordinate conversion function obtained by the calibration process. Yan), when the converted position coordinates overlap with the function buttons and icons arranged on the display screen 201, various processes corresponding to the function buttons and the like are performed, and strokes are drawn. (Step S203).

<Handwriting input process>
Next, as an example of various processes performed by the processing unit 24 of the computer device 2 on the entry information received from the electronic pen 1 when the coordinate conversion function is defined, the handwriting input application is started after the calibration process. The process of handwriting input with the electronic pen 1 will be described with reference to the flowcharts of FIGS. 6 and 13 and the image examples of FIGS. As a handwriting input application, for example, a presentation application that can also input text characters by keyboard input may be used.

  First, with reference to FIG.6 and FIG.11, the process which starts a handwriting input application is demonstrated. FIG. 11 is a diagram illustrating an image in which the display screen of the display unit 26 of the computer apparatus 2 is projected by the projector 3 on the coordinate input area 403 of the screen 4. As shown in FIG. 11, icon images for starting various applications are projected on the left side of the coordinate input area 403. Here, the icon image 701 is an image in which an icon of a handwriting input application is projected. On this screen, when the icon image 701 is tapped with the electronic pen 1, the electronic pen 1 reads the dot pattern on the icon image 701, calculates the position coordinates on the paper 402 from the dot pattern, and includes the coordinate data. The entry information is transmitted to the computer apparatus 2. In the computer apparatus 2, the receiving means 22 receives the entry information and stores it in the storage means 25 (step S201). Since the coordinate conversion function is defined (step S202: Yes), the processing unit 24 converts the coordinate data included in the entry information into position coordinates on the display screen 201 of the display unit 26 using the coordinate conversion function. Based on the converted position coordinates, the electronic pen 1 recognizes that the icon image 701 of the handwriting input application has been tapped, and starts the handwriting input application (step S203).

  FIG. 12 is a diagram showing an image projected by the projector 3 on the coordinate input area 403 of the screen 4 on the display screen of the computer apparatus 2 in which the handwriting input application is activated by the processing means 24. The processing unit 24 displays the window 702 by starting the handwriting input application. Further, the processing means 24 displays a writing area 703 in the window 702. The user can enlarge or reduce the writing area 703 by tapping the enlargement button 704 and the reduction button 705 with the electronic pen 1.

  Next, referring to FIG. 13, a process of entering and inputting handwritten characters and the like with the electronic pen 1 in the writing area 703 on the screen 4 of the handwriting input application, which is an example of the process of step S <b> 203 shown in FIG. 6. explain. This process is performed after it is determined that the coordinate conversion function is set by the calibration process (see step S202 in FIG. 6).

  As shown in FIG. 13, in the computer apparatus 2, the processing unit 24 calculates the position coordinates in the dot pattern on the screen 4 included in the entry information received from the electronic pen 1 by the coordinate conversion function obtained by the calibration process. Then, it is converted into position coordinates on the display (display screen) 201 (step S501). Then, the processing unit 24 determines whether or not the converted position coordinates are within the writing area 703 (step S502), and when determining that the position coordinates are within the writing area 703 (step S502: Yes), The stroke is drawn according to the converted position coordinates in accordance with the display magnification / position of 703, and the stroke data is stored as the entry contents in the file of the handwriting input application (step S503). On the other hand, if the processing unit 24 determines that the converted position coordinates are not within the writing area 703 (step S502: No), the processing unit 24 does not capture the stroke as the entry content, but the converted position coordinates are displayed on the display 201. If it overlaps with the position of the function button or icon that has been placed, it is recognized as a tap on the function button or icon, etc., and processing is performed accordingly. The entry information stored in the storage means 25 is erased (step S504).

  For example, when the electronic pen 1 reads the dot pattern on the screen 4 on the enlarge button 704 or the reduce button 705, calculates the position coordinates, and transmits entry information including the coordinate data to the computer apparatus 2 (step S101). In the computer apparatus 2, the receiving means 22 receives the entry information and stores it in the storage means 25 (step S201), and the processing means 24 defines the coordinate transformation function (step S202: Yes), so the entry information A conversion process of the coordinate data included in the data is executed (step S203). Specifically, as shown in FIG. 13, in the computer device 2, the processing unit 24 converts the coordinate data on the screen 4 included in the entry information into the position coordinates on the display screen 201 of the display unit 26 using a coordinate conversion function. (Step S501). Further, the processing unit 24 recognizes that the enlargement button 704 or the reduction button 705 of the handwriting input application is tapped with the electronic pen 1 although the converted position coordinates are not included in the writing area (step S502: No). Thus, the writing area 703 is enlarged or reduced (step S504). At this time, the processing unit 24 similarly enlarges or reduces the stroke written in the writing area 703 and taken into the computer apparatus 2 according to the enlargement / reduction of the writing area 703.

  Subsequently, such processing will be described with reference to FIGS. 14 to 20 by specific writing examples. The process until it is determined that the coordinate conversion function is set by the calibration process (see step S202 in FIG. 6) will be briefly described.

  FIG. 14 shows a state where the user writes “A B C D E” in the writing area 703 projected onto the screen 4 shown in FIG. Whenever a stroke is entered in the writing area 703, the electronic pen 1 generates entry information and transmits it to the computer apparatus 2 (step S101). When the computer apparatus 2 receives the entry information, it is stored in the storage means 25 (step S201). Further, the processing unit 24 determines that a coordinate conversion function is defined (step S202: Yes), and performs the process of step S203. Specifically, as shown in FIG. 13, first, the processing means 24 converts the position coordinates in the dot pattern on the screen 4 included in the received entry information into the position coordinates on the display 201 by the coordinate conversion function. (Step S501). Then, the processing unit 24 determines that the converted position coordinates are in the writing area 703 (step S502: Yes), and according to the converted position coordinates in accordance with the display magnification / position of the writing area 703, The stroke “A B C D E” is drawn and displayed on the display means 26, and the data is stored in a file (step S503). Since the processing unit 24 causes the image signal from the transmission unit 27 to be transmitted to the projector 3 in synchronization with the display on the display unit 26, the same image as the display unit 26 is projected on the coordinate input area 403 of the screen 4. At this time, the stroke drawn on the display means 26 is projected on the screen 4 along the locus of the electronic pen 1.

  FIG. 15 is a diagram illustrating a state in which the window 702 of the handwriting input application is moved from the position illustrated in FIG. 14 by the operation of the electronic pen 1. As shown in FIG. 15, it is assumed that the user drags down the window 702 to the horizontally long bar on the upper edge of the window 702 projected on the screen 4 with the electronic pen 1 and drags the window 702 to the upper left of the image projection area 403. In this case, entry information by the stroke of the drag operation is transmitted from the electronic pen 1 (step S101), and the computer apparatus 2 stores the received entry information in the storage unit 25 (step S201). Further, the processing means 24 recognizes that the coordinate transformation function is defined (step S202: Yes), and then proceeds to the specific processing of step S203, and receives the coordinate transformation function as shown in FIG. The position coordinates in the dot pattern on the screen 4 included in the entry information are converted into the position coordinates on the display 201 (step S501). Then, the processing unit 24 determines that the converted position coordinates are not within the writing area 703 (step S502: No). However, if the processing unit 24 recognizes that the upper edge bar of the window 702 is dragged, the processing unit 24 responds to the drag stroke. The window 702 is moved (step S504). At this time, the processing means 24 moves the characters and the like already written in the writing area 703 together with the window 702. The processing unit 24 causes the projector 3 to project the movement of the window 702 onto the screen 4 in order to transmit the image signal from the transmission unit 27 to the projector 3 in synchronization with the display on the display unit 26.

  FIG. 16 shows a state in which “F G H I J” is further written in the writing area 703 projected onto the screen 4 shown in FIG. 15 by the electronic pen 1. In the electronic pen 1, every time a stroke is entered in the writing area 703, entry information is generated and transmitted to the computer apparatus 2 (step S101). Then, the computer device 2 stores the received entry information in the storage means 25 (step S201), and recognizes that a coordinate conversion function is defined (step S202: Yes). Subsequently, the processing unit 24 converts the position coordinates in the dot pattern on the screen 4 included in the received entry information into the position coordinates on the display 201 using the coordinate conversion function (step S501). Then, the processing means 24 determines that the converted position coordinates are in the writing area 703 (step S502: Yes), and according to the converted position coordinates in accordance with the current display magnification / position of the writing area 703. Then, “F G H I J” and a stroke are added and rendered, displayed on the display means 26, and the stroke data is stored in a file (step S503). Since the processing unit 24 transmits the image signal from the transmission unit 27 to the projector 3 in synchronization with the display on the display unit 26, the same image as the display unit 26 is projected on the coordinate input area 403 of the screen 4. The stroke drawn on 26 is projected on the screen 4 along the locus of the electronic pen 1.

  FIG. 17 is a diagram showing a state in which the writing area 703 is enlarged and displayed by tapping the enlargement button 704 on the screen 4 shown in FIG. 16 with the electronic pen 1. When the user taps the enlargement button 704 on the screen 4 with the electronic pen 1, the electronic pen 1 reads the dot pattern on the enlargement button 704, calculates the position coordinates, and generates entry information. And the electronic pen 1 transmits entry information at any time (step S101). The computer apparatus 2 stores the received entry information in the storage means 25 (step S201) and recognizes that a coordinate conversion function is defined (step S202: Yes). Then, the processing means 24 converts the position coordinates in the dot pattern on the screen 4 included in the received entry information into the position coordinates on the display 201 by the coordinate conversion function (step S501). Then, the processing unit 24 determines that the converted position coordinates are not in the writing area 703 (step S502: No), but recognizes that the enlarge button 704 has been tapped, and enlarges the writing area 703 (step S504). ). At this time, the processing unit 24 enlarges the stroke of the writing area 703 similarly, recognizes that the entire writing area 703 cannot be displayed, and displays the vertical scroll bar 706 and the horizontal scroll bar 707. Display. Since the processing unit 24 causes the image signal to be transmitted from the transmission unit 27 to the projector 3 in synchronization with the display on the display unit 26, the same image as the display unit 26 is projected on the coordinate input area 403 of the screen 4. .

  FIG. 18 shows a state in which the electronic pen 1 continues to write strokes in the enlarged writing area 703 projected on the screen 4 shown in FIG. FIG. 19 shows the display area in the writing area 703 changed by moving the scroll bars 706 and 707 by dragging the electronic pen 1 in the window 702 projected on the screen 4 shown in FIG. This shows a state where a stroke surrounding the letter “E” is being written. In both FIG. 18 and FIG. 19, the stroke is input in accordance with the display magnification / position of the writing area 703 by the processing of the processing means 24 according to the flowcharts shown in FIG. 6 and FIG. FIG. 20 is a diagram showing a state in which the entire writing area 703 is displayed by tapping the reduction button 705 on the screen 4 shown in FIG. 19 with the electronic pen 1. Also in FIG. 20, according to the flowcharts shown in FIGS. 6 and 13, the writing area 703 is similarly reduced by the processing of the processing means 24 and displayed as a whole.

<Operational effect of the first embodiment>
In the information processing system 10 according to the first embodiment, when the calibration process is executed by the processing unit 24, the coordinate input surface and the coordinate display with high accuracy are recognized by recognizing the pen 1 as a reference when the pen-up coordinate is used. Definition of relative position coordinates with the surface (definition of coordinate transformation function) can be realized. Further, by determining whether or not the coordinates acquired as the reference point when executing the calibration process are included in the coordinates within the same recognition range of the coordinates already stored as the reference point, the user's continuous This makes it possible to prevent erroneous settings due to simple writing operations and to perform resetting immediately, improving convenience.

  In addition, a handwriting input application can be started by the processing unit 24 and projected by the projector 3, and handwritten input can be performed with the electronic pen 1 on the screen 4 on which the dot pattern is formed. At this time, when the entry position of the electronic pen 1 is in the writing area 703, the coordinate data included in the entry information transmitted from the electronic pen 1 can be coordinate-transformed and drawn as a stroke.

Second Embodiment
Next, the information processing system 100 in the second embodiment will be described. In the description of the second embodiment, the configuration different from the first embodiment will be mainly described. In the second embodiment, the same reference numerals as those in the first embodiment are the same as those in the first embodiment unless otherwise specified.

  FIG. 21 is a system configuration diagram of the information processing system 100 according to the second embodiment. In the first embodiment, the information processing system 10 is configured by using the screen 4 and the projector 3. However, in the second embodiment, as shown in FIG. 21, the information processing system 100 reads with the electronic pen 1. A possible dot pattern (coded pattern) is printed on the transparent sheet 8, and the transparent sheet 8 is pasted on the display screen 201 of the computer device 2.

  In such a configuration, when the calibration process is performed, the electronic pen 1 can be operated by entering strokes, tapping, and the like on the display screen 201. In the second embodiment, the calibration process is performed in the same manner as in the first embodiment. In addition, when the handwriting input application is started by the processing means 24 and handwritten input is performed, the coordinate conversion function is defined by the calibration process, so that the writing area 703 is placed on the transparent sheet 8 on which the dot pattern is formed. If writing is performed with the electronic pen 1, the same processing as in the first embodiment is performed. Other processes are the same as those in the first embodiment.

  In the case of the second embodiment, a dot pattern is printed on the transparent sheet 8 with ink having a characteristic of selectively reflecting infrared rays. In the electronic pen 1, the processor 108 uses image data captured by the CMOS camera 106 based on image data. In addition, a threshold value may be set so that a partial region having strong infrared reflection is recognized as a dot. The threshold value is set so that the infrared rays transmitted through the transparent sheet 8 from the display can be distinguished from the infrared rays reflected by the dots.

<Operational effects of the second embodiment>
The information processing system 100 according to the second embodiment also has the same effects as the information processing system 10 according to the first embodiment, and a transparent sheet on which a dot pattern (coded pattern) that can be read by the electronic pen 1 is printed. Since 8 is pasted on the display screen 201 of the computer apparatus 2, the user can perform an operation at hand by tapping the computer apparatus 2.

[Modification]
In the first and second embodiments, the calibration process is performed based on the position coordinates of two points. However, the calibration process may be performed based on the position coordinates of three or more points. In the first embodiment, a front-type projector has been described as an example of the projector 3, but the projector 3 can also be applied to a rear projector. Further, the dot pattern (coded pattern) is not limited to the Anoto method.

  In the first and second embodiments, the electronic pen 1 does not have an ink cartridge. However, it is possible to write characters or the like directly on the paper 402 with the electronic pen 1 using the pen unit 104 as an ink cartridge. You may do it. In this case, in the computer device 2, the stroke drawn with the electronic pen 1 is displayed on the display unit 26, but an image signal that is not projected by the projector 3 may be output from the computer device 2 to the projector 3. Further, in order to make it easy to recognize dots from the image data captured by the CMOS camera 106 of the electronic pen 1, an infrared transmission filter is provided in the opening of the electronic pen 1 so that only the infrared light is irradiated to the CMOS camera 106. You may provide near 102.

  The projection input / output system can be used in school classes, conference rooms, lectures, and presentations at companies.

DESCRIPTION OF SYMBOLS 1 ... Electronic pen 2 ... Computer apparatus 21 ... Input means 22 ... Receiving means 24 ... Processing means 25 ... Memory | storage means 26 ... Display means 27 ... Transmission means 201 ... Display screen 202 ... The upper left point coordinate of a display screen 203 ... Right of a display screen Lower end point coordinates 204 ... first display reference point coordinates 205 ... second display reference point coordinates 3 ... projector 4 ... screen 401 ... magnet board (white board)
402: Paper 403 ... Image projection area / coordinate input area 404 ... Upper left end point coordinates 405 ... Upper left end point coordinates of the coordinate input area 406 ... Lower right end point coordinates of the coordinate input area 407 ... First recognition reference point coordinates 408 ... Second recognition Reference point coordinates 601 ... first calibration mark 602 ... second calibration mark 7 ... identical recognition range 701 ... icon image 702 ... window 703 ... writing area 704 ... enlarge button 705 ... reduction button 706,707 ... scroll bar 8 ... Transparent sheet 10,100 ... Information processing system

Claims (4)

A screen on which a coding pattern indicating position coordinates is formed;
Reading means for reading the coded pattern;
A computer device that receives information on the coding pattern transmitted from the reading unit and performs processing according to the information;
An information processing system comprising: a projector that receives an image signal from the computer device and projects an image on the screen;
The computer device includes:
Receiving means for receiving information on the coding pattern transmitted from the reading means;
Conversion enlargement button, reduction button, and then displayed on the display means a moveable window having a write area, the position coordinates of the coded pattern contained in the received information by the receiving means to the position coordinates on the display screen of the display unit and, stored in the coordinate conversion position coordinates by drawing a stroke on the basis of the coordinate conversion position coordinates when in the write area on the writing area file, the coordinate conversion position coordinates the When it is recognized that the enlarge button or the reduce button is designated, the writing area is enlarged or reduced according to the designated button, and the drawn stroke is displayed in accordance with the display magnification and the position of the writing area. When the writing to the writing area at the display magnification and position is received, the coordinates are converted. The information processing system characterized in that it comprises a processing means for storing the same file to draw the stroke based on the location coordinates. A transparent sheet on which a coded pattern indicating position coordinates is formed;
Reading means for reading the coded pattern;
An information processing system comprising: a computer device that receives information related to the coding pattern transmitted from the reading unit and performs processing according to the information;
The transparent sheet is affixed to the display screen of the display means of the computer device,
The computer device includes:
Receiving means for receiving information on the coding pattern transmitted from the reading means;
Conversion enlargement button, reduction button, and then displayed on the display means a moveable window having a write area, the position coordinates of the coded pattern contained in the received information by the receiving means to the position coordinates on the display screen of the display unit and, stored in the coordinate conversion position coordinates by drawing a stroke on the basis of the coordinate conversion position coordinates when in the write area on the writing area file, the coordinate conversion position coordinates the When it is recognized that the enlarge button or the reduce button is designated, the writing area is enlarged or reduced according to the designated button, and the drawn stroke is displayed in accordance with the display magnification and the position of the writing area. When the writing to the writing area at the display magnification and position is received, the coordinates are converted. The information processing system characterized in that it comprises a processing means for storing the same file to draw the stroke based on the location coordinates.   If the processing means determines that a coordinate conversion function for converting the coordinates is not defined when the information is received by the receiving means, the processing means starts a calibration process for obtaining the coordinate conversion function. The information processing system according to claim 1 or 2. A program that causes a computer to function as the computer apparatus according to claim 1.

Images