JP4325332B2 - Pen-type data input device and program - Google Patents

Description

  The present invention relates to an apparatus and a program for capturing characters or figures written by a writer into a computer.

  Currently, an input device including a two-dimensional sensor array, such as an LCD (Liquid Crystal Display) tablet or a digitizer tablet, is widely used for inputting pen writing contents into a personal portable terminal or a computer application device. Since such an input device requires a two-dimensional sensor array having a relatively large area, a separate sensing plane is required. Therefore, it is difficult to carry, occupies a predetermined space, and has a disadvantage in terms of cost.

  Therefore, conventionally, in order to solve such a problem, only a single electronic pen is used on a general plane without a physical tablet. At this time, if this single electronic pen can input a document, it is very effective because a natural input can be made because a wide input space is provided compared to a conventional pen input device.

In order to input a document or a picture using such a self-motion sensing type electronic pen, the position coordinate of the electronic pen tip with respect to a certain reference coordinate system must be continuously obtained.
However, most writing operations are performed in a down state in which the pen is in contact with the writing surface, and when moving, the pen is in an up state in which the pen is not in contact with the writing surface. In order to obtain a continuous coordinate value of a pen, a means capable of accurately measuring the position value even in a contact or non-contact state is required.

There are three types of conventional electronic pen-like input devices: an external coordinate measuring method, an internal coordinate measuring method, and a paper surface coordinate printing method.
The external coordinate measurement method is a method of measuring the coordinates of the pen tip outside the pen. For example, a triangulation method (Patent Document 1), an electromagnetic wave (Patent Document 2), or an ultrasonic wave (Patent Document 3). There are methods that use the time difference of flight.

However, since the above method is such that a transmission signal is transmitted from the pen and received from the outside, in a device such as a portable terminal, the receiver must be attached to the main body of the terminal, so There are inconvenient disadvantages.
The internal coordinate measurement method is a built-in method for measuring the coordinates of the pen tip inside the pen, and is a method for sensing the movement of the pen tip inside the pen. Initially, a 2-axis or 3-axis acceleration sensor mounted inside the pen. The system (patent document 4, patent document 5, patent document 6) which calculates | requires the position motion of an electronic pen through double integration using this was proposed.

  However, there is a problem that it is difficult to mount the acceleration sensor on the pen tip. When the acceleration sensor is mounted at a certain height, the influence on the tilt angle of the pen center axis is not taken into consideration, which may cause a large position error. In addition, since the accumulated error increases due to double integration of the acceleration signal, there is a disadvantage that it is difficult to measure accurate motion.

  In order to correct the influence on the tilt angle of the pen, A.TCross (Patent Document 7) proposes a method of moving the acceleration sensor element of two or more axes to the pen tip and moving the signal processing unit to the upper part of the pen. However, since the sensor element and the signal processing unit are separated and the influence of electrical noise is large, there is a problem that ink cannot be attached to the tip of the pen.

  On the other hand, Seiko Co., Ltd. (Patent Document 8) uses biaxial acceleration and 2-gyro to correct the tilt angle, finds the position of the acceleration by double integration, and calculates the tilt angle of the pen by simply integrating the angular velocity of the pen. A method of measuring is proposed. In addition, Richo (Patent Documents 9 and 10) presents a method for obtaining the position of a pen tip that performs a general three-dimensional writing motion by incorporating a three-axis acceleration sensor and a three-axis gyro sensor inside the pen. .

  However, since the input plane must always be perpendicular to the direction of gravity, there is a limitation in use. In the method using the inertial sensor (acceleration sensor, gyro), acceleration is pen-integrated through double integration and angular velocity is pen-integrated through single integration. The position and angle of the sensor are estimated, but due to noise and drift of the sensor signal, the accumulative error increases in proportion to the square of the time in the acceleration system and proportional to the time in the case of the gyro to estimate the precise pen tip movement. There are difficult problems.

  In order to reduce the accumulated error, Intersense recently added an ultrasonic sensor to a pen device using a three-axis acceleration sensor and a three-axis gyro to reduce the position accumulated error generated from the inertial sensor (acceleration, angular velocity). Presenting technology. However, there is a problem that the addition of such an external sensor is not portable.

  In addition, as a result of these improvements, in Patent Document 11, in addition to a three-axis acceleration sensor, an optical three-dimensional measurement for measuring the tilt angle of the center axis of the electronic pen with respect to the writing surface and the height with respect to the writing surface. Some have improved the accuracy by providing a measuring device and correcting the position coordinates from the triaxial acceleration sensor using the height of the writing surface.

However, this method accurately detects the position in the height direction, but the movement amount detection in contact with the writing surface must be measured with an acceleration sensor. Not suitable.
The paper coordinate printing method is a method in which coordinates are recorded with fine points on one surface of the paper, the coordinates are read by a two-dimensional image sensor (camera) mounted on the pen, and the locus of the pen is calculated based on the coordinates. And patent document 12 etc. are proposed. This method is accurate because the locus of the pen is detected by reading the coordinates written on the paper.

However, this method can operate only on the paper on which the coordinates are written in advance, and there is a problem that the locus of the pen cannot be taken freely on ordinary ordinary paper.
US Pat. No. 5,166,668 US Pat. No. 5,977,958 U.S. Pat. No. 4,478,674 US Pat. No. 5,247,137 International Publication WO 94/09447 Pamphlet US Pat. No. 5,587,558 US Pat. No. 5,434,371 JP-A-6-67799 US Pat. No. 5,902,968 US Pat. No. 5,981,884 JP 2003-029915 A International Publication No. WO92 / 17859 Pamphlet

As described above, the conventional electronic pen has various problems and is not always satisfactory.
Therefore, the present invention provides a pen-type data input device employing a new pen trajectory acquisition method using an acceleration sensor or an image sensor, and a program thereof.

The pen-type data input device according to claim 1 is a pen-type data input device that acquires, as electronic data, a character or a figure written on a writing target medium, and an acceleration generated when the pen-type data input device is moved. Periodically detecting acceleration detecting means, speed calculating means for calculating the moving speed of the pen-type data input device based on the acceleration detected by the acceleration detecting means, and each time the speed is calculated by the speed calculating means Based on the previous speed value stored by the speed storing means and the current speed value calculated by the calculating means, the pen-type data input device First trajectory information acquisition means for acquiring trajectory information indicating the trajectory moved, and writing means for writing the character or the graphic on the writing target medium using ink; An imaging unit that images the character or the graphic written by the writing unit, and the character or the graphic written by the writing unit based on a plurality of images of the character or the graphic captured by the imaging unit Second trajectory information acquisition means for acquiring trajectory information indicating the trajectory of the pen, detection means for detecting whether the pen tip of the pen-type data input device is in contact with or away from the writing target medium, and the detection The first trajectory information acquisition unit is selected when it is detected by the means that the pen tip is separated, and the second trajectory information acquisition unit is selected when the pen tip is detected as touching. a selecting means for obtaining the locus information and the speed of the nib by it in contact with said writing target medium is detected by said detecting means, stored by the speed storage unit Characterized in that it comprises a correcting means for correcting the value to 0, the.

With this configuration, when the pen is away from the paper surface, the trajectory of the electronic pen is analyzed using the acceleration detecting unit, and when the pen is in contact with the paper surface, the pen trajectory is used using the imaging unit. Further, when the pen is placed on the writing surface, it is possible to correct the accumulated error of the speed calculated based on the acceleration obtained from the acceleration detecting means.

According to the invention of claim 2 , the second trajectory information acquisition unit compares the image captured this time by the imaging unit with the image captured last time, and compares the result. and generating the character or locus information indicating a locus of the figure is written by said writing means based on.

By configuring in this way, the trajectory information can be obtained from the difference between continuously captured images.
According to a third aspect of the present invention, when the second trajectory information acquisition means is writing by the writing means, the trajectory information acquired this time by the first trajectory information acquisition means and the previous acquisition. based on the trajectory information, and determines the range of the comparison.

By configuring in this way, it is possible to speed up the processing and reduce the processing load by limiting the comparison range of two images to the vicinity of the portion where writing is performed.
According to a fourth aspect of the present invention, the pen-type data input device further includes transmission means for transmitting the trajectory information acquired by the first or second trajectory information acquisition means to an external device. It is characterized by providing.

With this configuration, the content written with the electronic pen can be transmitted to the host computer via wireless communication or the like.
According to the invention described in claim 5, when the pen-type data input device is connected to the external device, the pen-type data input device transmits the trajectory information by the transmitting means, and the external device and the If the communication is disconnected, characterized by holding the trajectory information.

  By configuring in this way, when the communication with the host computer is possible, the content written with the electronic pen can be transmitted to the host computer via wireless communication or the like in real time, and the communication is impossible. In such a case, the contents can be stored in a memory built in the electronic pen.

By using the present invention, it is possible to accurately detect the movement of the pen by analyzing the image taken by the imaging means and the acceleration detected by the acceleration detecting means, and when the pen is placed on the writing surface. The accumulated error of the speed calculated based on the acceleration obtained from the acceleration detecting means can be corrected.

(Embodiment 1)
The feature of the electronic pen in the present embodiment is that it has the following three functions.
(1) The amount of ink can be controlled electronically according to the writing pressure, and the thickness of the pen character can be adjusted.
(2) The pen trajectory and writing pressure of characters and figures written on plain paper can be stored.
(3) The pen locus can be transferred to a personal computer (hereinafter referred to as PC) by wireless communication. The present embodiment will be described below.

  FIG. 1 is a configuration diagram of an electronic pen 1 in the present embodiment. The inkjet 17 and the inkjet nozzle 2 for spraying ink onto the paper surface are provided so that the thickness of the pen character can be adjusted according to the writing pressure. A pen tip 3 is mounted next to the nozzle 2 so that the nozzle 2 does not directly contact the paper surface. Ink does not come out from the pen tip 3, and a material that does not feel uncomfortable when a figure is written with the pen 1 is selected and mounted.

  A pressure sensor 4 is mounted at the base of the pen tip 3, and the pressure applied to the pen tip 3 is detected by the sensor 4. As a result, the thickness of the pen character can be adjusted according to the pressure. Further, the electronic pen 1 can be equipped with a color ink cartridge 5, and the ink cartridge 5 is made of CMYK (cyan, magenta, yellow, black) so that not only black ink but also a full color can be created. Includes four different colors. The color of the pen 1 can be selected and controlled so that the ink and the amount of ink corresponding to the color of the pen 1 are selected by being sprayed onto the writing surface via the nozzle 2 by the inkjet 17 described above. Drawn.

  In addition, a triaxial acceleration sensor 6 is provided to detect the locus of the pen 1, and a lens 7 and a CMOS image sensor 8 for reading a figure drawn on a paper surface are also provided to compensate for the insufficient accuracy of the acceleration sensor. Have. In addition, a processor 9 for processing the read image or calculating a pen locus and the like and a memory 10 for storing the calculated locus and writing pressure are also provided. A battery 11 for operating the electronic circuit is also mounted. The RF communication circuit 13 is a circuit for transmitting information such as a pen locus to the host computer.

Next, the drawing operation of the electronic pen 1 will be described with reference to FIGS.
FIG. 2 is a control block diagram of inkjet in the present embodiment. FIG. 2 is a detailed circuit diagram of the partial block of FIG. 1, which is a configuration diagram of the electronic pen 1 in the present embodiment, with a signal flow added thereto. When drawing on paper (written material) using the electronic pen 1 according to the present embodiment, the drawing may be performed by sliding the pen tip 3 against the written material in the same manner as a normal pen.

  The pen tip 3 is directly connected to the pressure sensor 4 and is mounted so that the pressure applied to the pen tip 3 can reach the pressure sensor 4 as it is. Therefore, when the user (writer) uses the pen 1, the writing pressure can be detected by the pressure sensor 4. The pressure sensor 4 in the present embodiment uses a piezoresistive pressure sensor, and uses a material whose resistance value changes according to the pressure.

  A constant current is passed through the sensor 4, and the voltage at the end of the sensor 4 is obtained by the AD converter 902 in the processor 9 in accordance with the pressure value. This digital pressure value is read by the CPU 901. The CPU 901 adjusts the amount of ink sent from the color ink cartridge 5 to the inkjet 17 in accordance with this numerical value. The ink amount is adjusted by the following method.

First, assuming that the amount of black ink (K) used for drawing black is 1, the ink mixing ratio of CMYK for creating the current color (C1) to be drawn is set to Kc, Km, Let Ky, Kk.
As for this mixing ratio, the ink mixing ratio for obtaining the color closest to the color to be drawn is examined in advance, and programming of a method for deriving the mixing ratio is performed, or the mixing ratio of the ink corresponding to each color is performed. Is stored in a memory or the like, so that the mixing ratio can be easily obtained.

  Next, assuming that the maximum pressure value applied to the pen tip 3 is Pmax and the current pen pressure is P, the ink mixing ratios are Kc * P / Pmax, Km * P / Pmax, Ky * P / Pmax, respectively. Kk * P / Pmax. In accordance with these mixing ratios, the CPU 901 controls each valve 103 (103a, 103b, 103c, 103d) so as to flow from the ink cartridge 5 to the inkjet 17. Since a small valve cannot normally control the intermediate value of the outflow amount in an analog manner, the intermediate amount is created by controlling the valve open time (open) and OFF time (close) at high speed.

FIG. 3 shows pen line thickness control by duty modulation in this embodiment. This figure shows duty modulation for changing the diameter of ink on the paper (thickness at the time of pen) according to three types of opening and closing times.
FIG. 3A shows the duty modulation when the writing pressure is low. In this case, since the amount of outflow of ink is small, a thin line (the right figure in FIG. 3A) can be drawn. FIG. 3B shows duty modulation when the writing pressure is intermediate. In this case, since the outflow amount of ink is medium, an intermediate thickness line (the right figure in FIG. 3B) is drawn. Can do. FIG. 3C shows duty modulation when the writing pressure is large. In this case, since the amount of ink flowing out is large, a thick line can be drawn (the right diagram in FIG. 3C).

By controlling the opening / closing time of the valve 103 by duty modulation in this way, it becomes possible to eject ink from the inkjet nozzle 2 in accordance with the pressure of the pen, so that the thickness of the pen of a desired color can be controlled. It is.
Next, when drawing is performed on plain paper using the electronic pen 1 in the present embodiment, a trajectory analysis of the pen 1 is performed in order to record the drawing contents.

  FIG. 4 shows a configuration of a position detection circuit using the triaxial acceleration sensor 6 in the present embodiment, and is a block diagram showing an apparatus for performing a basic trajectory analysis of the pen 1. Although the drawing operation on the paper (written material) using the electronic pen 1 of the present embodiment has been described above, the locus of the pen 1 when the drawing is performed can be analyzed and acquired as data.

The triaxial acceleration sensor 6 is a device that detects acceleration according to the movement of the pen 1. In order to specify the coordinates of the pen 1 using this, the position of the pen 1 can be calculated by performing double integration from the acceleration.
The triaxial acceleration sensor 6 includes acceleration sensors 601 (X-axis), 602 (Y-axis), and 603 (Z-axis) using piezoresistors that can detect acceleration in each of three spatial directions. Reference numerals 601, 602, and 603 are arranged at right angles in three axial directions (in FIG. 4, the directions of the respective axes are X, Y, and Z axes).

  Currently, a low-cost two-axis acceleration sensor made of MEMS (micro-electro-mechanical system) has been developed, so two devices may be used to form a three-axis acceleration sensor. Since the resistance value of the piezoresistor changes according to the acceleration, the acceleration can be read out as a voltage value by passing a constant current through the resistor, and the respective voltage values are stored in the processor 9 via the selector 604. The AD converter 902 is configured to be able to continuously read acceleration values in a time division manner.

  In the present embodiment, the image sensor 8 is also provided as another means for analyzing the locus of the pen 1. What is shown in the lower left of FIG. In the electronic pen 1, the image on the writing surface 20 is condensed on the CMOS image sensor 8 by the lens 7 and converted into image data by the image sensor 8. In this embodiment, this image data is sent to the memory 10 by a DMA (Direct Memory Accessing) circuit 903. As a result, the image data can be operated from the CPU 901.

Incidentally, the CMOS sensor 8 is configured to sample the image data at a cycle of 100 sheets per second (at intervals of 10 milliseconds) and sequentially send them to the memory.
FIG. 5 shows an algorithm for obtaining a coordinate position in the processor 9 using the triaxial acceleration sensor 6 in the present embodiment. The processor 9 is programmed so as to be able to execute a plurality of processes called “Task”.

  In this Task, that is, Task 1, first, the current acceleration of each axis is read from the triaxial acceleration sensor 6 (Step S <b> 1, Step is hereinafter referred to as S). The variable a is a three-dimensional vector having acceleration values for the respective axes. The velocity v (three-dimensional vector) is calculated by integrating the acceleration a (S2). That is, if the acceleration a is integrated, v '+ aT is obtained (v' is the previous speed and T is time), and this is the speed v. Further integration is performed to calculate a three-axis absolute position Px (three-dimensional vector) (S3). That is, a value obtained by adding the distance (v ′ + v) T / 2 moved this time to the previous absolute position Px is defined as Px. In this program, since the average of the previous value is used as the speed value in order to reduce the error (S3), the previous value is stored as v '(S4).

  Then, it waits until T second passes (S5). This is because if only this task is processed, other tasks cannot be processed. Thereafter, the above processes S1-S5 are repeated. By reading the memory contents of Px, the current position Px can always be read. This Px can be accessed from Task 2 described later, which can be executed simultaneously.

  However, when a pen trajectory is obtained only by this method, an acceleration detection interval time of a normal acceleration sensor is several milliseconds, and an error in acceleration change between detection times occurs. This error is also integrated to perform the integration to obtain the coordinates, resulting in a fairly large error. With the current acceleration sensor, it is difficult to obtain a resolution locus of 100 DPI (Dot Per Inch).

  Therefore, in this embodiment, the problem is solved using a camera. That is, the moving distance of the pen is calculated by analyzing an image photographed by the camera. As described above, when the writer is drawing a character figure on the paper surface, it is possible to detect whether or not the writing is performed by the pressure sensor.

  FIG. 6 shows a writing example of characters photographed by the camera in the present embodiment. Here, the camera refers to an imaging system including at least the lens 7 and the image sensor 8 in the present embodiment. FIG. 6 shows two images (a) and (b) taken continuously (at intervals of 10 milliseconds) while the writer is drawing the character “A” with the electronic pen 1.

  In the electronic pen 1, such an image on the writing surface 20 is condensed on the CMOS image sensor 8 by the lens 7 and converted into image data by the image sensor 8. The image data is sent to the memory 10 via a DMA (Direct Memory Accessing) circuit 903.

  As a result, the image data can be operated from the CPU 901. Incidentally, the CMOS sensor 8 is configured to sample the image data at a cycle of 100 sheets per second (at intervals of 10 milliseconds) and sequentially send them to the memory. The pen movement distances du and dv can be calculated from the two consecutive images by the algorithm shown in FIGS.

  FIG. 7 shows an algorithm of a pen coordinate position analysis task based on image data in the present embodiment. In order to distinguish from the coordinate system X, Y, Z calculated by the three-axis acceleration sensor 8 described above, a two-dimensional coordinate system on the paper surface calculated from the camera image is set as U, V coordinates. In this embodiment, this task is called Task2.

  First, it is determined whether the electronic pen 1 is in contact with the paper surface 20 (hereinafter referred to as pen-down) or away from the paper surface 20 (hereinafter referred to as pen-up) (S10). This determination is made based on the detection result of the pressure sensor 4. In the case of the pen-up state, the process proceeds to “No” in S10 and waits for the pen-down. When the pen is down, the process proceeds to “Yes” in S10.

  Next, it waits for an image to be input from the camera (S11). Until an image is input from the camera, the process proceeds to “No” in S11. When an image is input from the camera, the process proceeds to “Yes” in S11. Next, image size reduction is performed as pre-processing for matching (matching) two input images (S12). In this method, since the number of pixels is large in order to match an input image as it is, a process of creating a reduced image from the input image is performed (this will be described later).

  Next, the matching calculation routine is called (S13), and the movement vector Du is calculated based on two images, the image input at this time and the image input at the previous timing, among the periodically photographed images. (This will be described later). In this way, the current coordinate position is updated by adding the movement vector Du to the immediately preceding pen coordinate Pu.

  FIG. 8 shows an algorithm for the matching of S13 in FIG. 7 and the calculation of the movement vector Du. In this algorithm, matching is performed by calculating the sum of differences in pixel values between corresponding pixels for two images to be compared. The range of the distance between images to be matched (the amount of deviation between two images) is passed as an argument. The RANGE in the figure shows the maximum value of the range in which the electronic pen 1 is considered to be movable (the unit is the number of pixels). The range can be specified by Du_Begin to Du_End on the U axis side and Dv_Begin to Dv_End on the V axis side . In this example, −Range to + Range are specified for both Du and Dv so that matching is possible within the range of the maximum distance RANGE that can be moved at normal pen speed between image samplings (Du_Begin = −RANGE, Du_End = + RANGE, Dv_Begin). = −RANGE, Dv_End = + RANGE).

In S20 and S21, the ranges of the movement distances (vertical and horizontal deviation amounts) du and dv between the two images designated by this argument are set in order.
In S22, the matching of the two images is performed by calculating the following equation.

  p (n, u, v) is the pixel value of the coordinates u, v of the current image n, and p (n-1, u-du, v-dv) is a coordinate point in the previous image n-1. This is the value of a pixel that is (du, dv) away from (u, v). The pixel value is represented by 8 bits, and black is 0 and white has a maximum value of 255. MU and MV are image sizes in the U and V axis directions of these images, and indicate the range of images to be matched. D (du, dv) represents the result of summing up the differences in pixel values of the overlapping pixels in a state where the two images are shifted by the movement distances du and dv. Since the difference is 0 when the corresponding pixel values are the same, the larger the calculated D is, the closer the two images are.

  The calculation process of S22 is repeatedly executed while changing dv and du one pixel at a time in the range of du and dv shown below and shifting the previous image (S20 to S24).

  When the calculation repeatedly performed from S20 to S24 is completed, the calculated maximum value of D (the one closest to 0) is searched for in S25. Since it can be said that these two images are most similar when the previous image is taken with du and dv at this time, the value at this time represents a vector Du in which the pen moves between the two images. Determine (S26). By adding the value of the movement vector thus obtained to the previous coordinate position data, the coordinates Pu (= Pu + Du) of the pen position can be calculated sequentially.

  Next, consider the resolution of the camera. Using the algorithm of FIG. 8, the moving distance can be calculated in units of pixels from the input image. Therefore, it is sufficient if the resolution of the camera is more than necessary for storing the pen trajectory. Usually, it is said that about 100 DPI is desirable to reproduce the pen trajectory, so the image sensor and the optical design should be designed so that there are four or more scanning lines per mm.

  In this embodiment, the angle of view is 12 mm, and the image sensor is 480 × 480. The reason why the resolution is higher than necessary here is because it is used for other operations described later. Since this number of images is over-spec for matching, the number of pixels of the image to be matched is reduced to about 128 × 128 resolution in S12 of FIG.

  Next, let us consider correction of the movement distance. In the case of the method of analyzing the locus by matching the images taken by the camera used in this embodiment, when the pen is once separated from the writing surface and placed on the paper surface again, while the paper is leaving the paper surface Since the handwritten image on the paper can no longer be obtained with the camera, there is a problem that “the relative coordinates with the previous locus point can no longer be obtained”.

  In the present embodiment, in order to solve this problem, the coordinate position calculated based on the acceleration detected by the triaxial acceleration sensor 6 described above is used to match the pen position even when the pen moves away from the writing surface. Makes it possible to guess the moving coordinates of. First, the task (Task 1) for calculating the position coordinates by the triaxial acceleration is improved.

  FIG. 9 is an improved algorithm of Task 1 of FIG. S30 to S33 and S38 are the same processes as S1 to S5 in FIG. The other processes S34 to S37 calculate where the writing surface (paper surface) is located in the triaxial absolute coordinates. Now, this algorithm will be described in detail.

  First, the process of S30-S33 is performed. These are the same processes as S1 to S4 in FIG. Next, it is determined whether or not the pen 1 is down on the paper surface 20 (S34). When the pen 1 is not down on the paper surface 20 (the process proceeds to “No” in S34), the process proceeds to S38. If the pen 1 is down on the paper surface 20 (proceed to "Yes" in S34), it is determined whether or not there is a triaxial parameter correction request (S35). Whether or not there is a triaxial parameter correction request means whether or not a triaxial absolute coordinate parameter correction request has been made in S55 of FIG.

If there is no correction request, the process proceeds to S37 (the process proceeds to “No” in S35), and if there is a correction request, the process of S36 is performed (the process proceeds to “Yes” in S35). When a pen down occurs in a task (Task 2) for calculating coordinates on the paper, which will be described later, since the pen speed is zero, the pen speed variable v ′ is set to 0 (S36). Since this speed is calculated by integrating the acceleration sensor 6, there is a possibility that an error is accumulated in FIG. 5 and does not match the actual speed. Can be cleared.

Next, when the pen is down, a paper basis vector is created (S37). After the process of S37, it waits for a predetermined time to elapse (S38). This process is the same as S5 in FIG.
FIG. 10 is a diagram illustrating the relationship between the three-axis absolute space and the paper surface coordinates. That is, a coordinate system of absolute coordinates calculated from triaxial acceleration on the X, Y, and Z axes is shown. A rectangular area represented by the paper surface 20 indicates the position of the paper surface in the absolute space. Eu and Ev vectors on the paper surface 20 are basis vectors for creating a coordinate system of the paper surface 20.

  An example in which a line between points P1 and P2 is drawn on the paper surface 20 is shown. This vector is defined as Px in the XYZ coordinate system and as Pu in the coordinate system on the paper surface 20. Since the relationship between the X, Y, and Z axes and the U and V axes is linear, the following relationship is established.

  If the corresponding points of the two coordinates are (x1, y1, z1) → (u1, v1), (x2, y2, z2) → (u2, v2) at two points,

The relationship holds. Than this,

As a result, the basis vectors Eu and Ev are respectively set to (1, 0) and (0, 1) in (u, v) of the equation (B-1) using the coefficients obtained in the equation (B-3). Since it is obtained by substituting, it becomes as follows.

FIG. 11 shows a detailed algorithm for creating the basis vector of the page in S37 of FIG. First, the pen position Pu of the current page coordinate is obtained. Since this is calculated in another task Task2, it is assumed that the latest Pu that has already been calculated is read (S40).
Next, the number of samples Ns corresponding to the two types of coordinates is checked (S41). It is assumed that this is initialized to “0” at the start of Task 1 in advance. When this process is performed for the first time after initialization, since Ns = 0, the process branches to S42. Here, the pen position Pu of the current paper surface coordinate system is stored at (u1, v1), the coordinate value of Px of the three-axis acceleration coordinate system is stored at (x1, y1, z1) (S42), and the corresponding point number Ns is set to 1. (S43), this flow ends.

  Thereafter, when this processing is called again with the pen 1 still down on the paper surface (S37 in FIG. 9), after obtaining Pu in S40, Ns is “1”, so in S41. Branches to S44. For the corresponding point of the current coordinate, Pu is stored in (u2, v2), Px is stored in (x2, y2, z2) (S44), and Δ in the equation (B-5) is calculated (S45).

This Δ is an outer product of two vectors (u1, v1) and (u2, v2), and it is desirable that this value be larger in order to improve the accuracy of calculation of a basis vector to be obtained later. Therefore, it is determined whether or not Δ is larger than a predetermined value (S46). If this Δ does not reach the predetermined value or is equal to or smaller than the predetermined value, it is determined that the second sample is not suitable for obtaining the basis vector, and the process is terminated without performing the subsequent processing.

On the other hand, if Δ exceeds a predetermined amount, the basis vector can be calculated with high accuracy, so that the current sample is valid and the number of samples Ns is set to “2” (S47). Then, the base vectors Eu and Ev are calculated by the equation (B-4) (S48), and the process ends.

Thereafter, when this processing is called again with the pen 1 still down on the paper surface (S27 in FIG. 9), after obtaining Pu in S40, Ns is “2”, so in S41. It is determined that Ns> 1, and the process ends.
As described above, in S37, the coordinates Pu on the paper surface calculated from the current absolute coordinates Px and Task2 are referred to, these two coordinates are stored in pairs, and Eu from the past several coordinate pairs. , Ev. In the present embodiment, calculation of Eu and Ev is performed with Task1, but of course, it may be performed with Task2. As a result, the basis vector can be calculated while writing on the paper surface 20 with the pen 1.

Next, when the pen leaves the paper surface 20 and goes down again on the paper surface 20, how to obtain the coordinates on the paper surface at that point will be described with reference to FIG.
FIG. 12 shows an algorithm obtained by modifying Task 2 in FIG. When the task is started, the FirstDown flag indicating that the pen has never been down on the paper is set to Yes, and PenStatus indicating the pen state is initialized to the pen-up state PEN_UP (S50). .

  Next, the pen pressure is detected by the pen pressure sensor 4, and it is determined whether or not the pen 1 is writing (pen down state) (S51). When the pen 1 is in the down state, the previous pen state is checked (S52). PenStatus for the first pen down is PEN_UP.

  If “PenStatus = PEN_UP” in S52, PenStatus is set to PEN_DOWN (S53), and the three-axis absolute coordinates of the position on the pen-down paper surface 20 are read (S54). Here, Px obtained by Task 1 is substituted into Pxd. Next, as described above, in order to reset the speed error, a correction request for the parameter (speed) of the three-axis absolute coordinate is made (S55).

  Then, it is checked by referring to the FirstDown flag whether the pen is down for the first time on the paper surface 20 (S56). When it is the first time (proceeding to "Yes" in S56), only the FirstDown flag is cleared (FirstDown = No (S61)) and the paper surface coordinate position is initialized (Pu = 0 (S62)). Then, the paper surface coordinate position down on the paper surface 20 is stored (Pu_down = Pu) (S60).

Thereafter, the processing is looped again. While the pen is being drawn on the paper surface, the process passes through S50 and S51. Since “PenStatus = PEN_DOWN” (S52), the process branches to S63.
Next, it is determined whether or not an image has been input (S63). As described above, the image data is taken periodically (every 10 milliseconds) (see FIG. 6), and loops again and waits until image input is made (proceed to “No” in S63). . When a new image is input (proceed to "Yes" in S63), a reduced image is created from the input image because the number of pixels is large in order to perform matching calculation between the input images as they are (S64).

Next, an image matching processing routine is called in S65 to obtain a movement vector Du (detailed operation has already been described with reference to FIG. 8). Then, the coordinate position is updated by adding the difference vector Du to the past pen coordinates Pu (S66).
As described above, while the writer is drawing on the paper surface 20, it is possible to track the pen trajectory and the pen pressure by performing image matching. When the writer performs pen-up, the pen pressure disappears. As a result, the pen pressure is determined to be up in S51, and the process branches to S67.

  Whether or not the pen-up state is the first time is determined based on PenStatus (S67). If this is the first time, PenStatus is set to PEN_UP (S68), and the three-axis absolute coordinates at that time are stored (S69). Here, Px obtained by Task 1 is substituted into Pxu. Thereafter, the up position of the paper surface coordinates is stored (S70). Here, Pu is substituted into Pu_up. After pen-up, the amount of movement of the pen 1 in the air is measured by Task 1.

  If the pen goes down on the paper again, it is determined in S56 that the second time or later, and the movement vector (du, dv) is determined in S57-S59 using the previous pen-up coordinate and the current absolute coordinate value. Calculated. First, an aerial absolute movement vector Dx is obtained by subtracting the 3-axis absolute coordinate Pxu acquired in S70 from the 3-axis absolute coordinate Pxd acquired in S54 (S57).

  Next, a process for converting the aerial absolute movement vector Dx into coordinates on the paper is performed. Eu and Ev are vectors corresponding to the XYZ coordinate systems of the U and V component unit vectors of the U and V coordinates, respectively. Now, converting the X-Y-Z system absolute air movement Dx to the U-V coordinate system is nothing other than decomposing the Dx vector into Eu and Ev. Therefore, if Dx = (x, y, z), Eu = (xu, yu, zu), Ev = (xv, iv, zv), the component (du, dv) of the UV coordinate system of Dx to be obtained is

It is obtained by
Then, the paper surface coordinate position Pu when pen-down is performed again may be obtained by adding (du, dv) obtained in S58 to the coordinates when pen-up is performed in S70 (S59). This Pu is stored as Pu_down (S60). In this way, the movement vector of the paper surface coordinates can be calculated from the absolute coordinates of the three axes.

  Next, the pen trajectory host transfer in this embodiment will be described. When the pen 1 performs drawing on the paper surface 20 by the method described above, the pen trajectory and writing pressure are stored in the memory in real time. The contents of the memory are read out in real time when the radio is being serviced and sent to a host such as a PC by radio. When the radio is not being serviced, it is stored in the memory 10 until the radio is serviced. In addition, the data stored when the radio is serviced is called at once and transferred to a host such as a PC.

  This is because the wireless nature makes communication unstable, so it is designed with that in mind. Of course, if the host does not operate for some reason, for example, you can take out only the pen without having the host PC at the business trip and write the proceedings on the business trip in the pen. Data can be saved and read out on the office PC after returning to work. In the present embodiment, data is transferred wirelessly by a PC. However, the present invention is not limited to this and may be wired.

As described above, the movement of the electronic pen can be detected with high accuracy by using the image photographed by the camera and the triaxial acceleration sensor signal.
(Embodiment 2)
In this embodiment, speeding up by limited matching will be described. When matching the camera images input in FIG. 7 or FIG. 12 (S13 or S65), all matching is performed at a distance within the maximum range that the pen is supposed to move between image sampling of the camera from FIG. It was.

In the present embodiment, the moving direction and the moving amount of the pen 1 are estimated from the absolute coordinates and speed of the three axes obtained from the acceleration sensor 6, the matching range is limited, and no dedicated hardware is added. It is also possible to calculate matching at high speed.
In other words, if the absolute coordinate vector moved between the two sampled images is Dx, the inner surface is calculated by the same method as S57 and S58 in FIG. An on-axis movement vector Du = (du, dv) can be calculated. What is necessary is just to obtain | require an image matching and a movement vector in the range of the vicinity of this du and dv. Since this value is an approximate movement vector moved between the two images, if the image matching is performed in the vicinity, the movement vector can be obtained efficiently.

  FIG. 13 shows an algorithm for obtaining the coordinate position in the processor 9 using the limited matching in this embodiment. Each process of S80 to S84 is the same as S10 to S14 of FIG. At this time, when calling the algorithm of FIG. 8 in S83, Du_Begin = du−δ, Du_End = du + δ, Dv_Begin = dv−δ, Dv_End = dv + δ (δ is sufficiently small compared to RANGE in the vicinity). Just pass an argument. This significantly reduces the amount of matching.

  As described above, the movement direction can be predicted by comparing the past photographed image with the current photographed image and using the information obtained from the acceleration sensor to perform the movement matching and knowing the approximate pen movement direction. Therefore, the matching direction and the amount of movement of the two images can be limited, so that the matching time can be shortened.

(Embodiment 3)
In the present embodiment, high-precision matching based on pen tilt correction will be described.
FIG. 14 shows an example of the movement of the pen 1 in the present embodiment. In the same figure, when the pen 1 is moved along the paper surface, the position indicated by the pen shaft in addition to the two-dimensional movement that moves along the paper surface while maintaining the inclination formed by the pen shaft and the paper surface. There is also a movement in which the tilt of the pen axis and the paper changes as it is.

The image matching for estimating the movement of the pen shown in FIG. 8 assumes only the movement of the pen in a two-dimensional direction. This is because the inclination of the pen axis during normal writing does not change much, and only the resolution of the movement amount is required. It can be ignored when matching images.
However, in order to record the trajectory of the details of the character, it is necessary to perform matching in consideration of the inclination of the pen axis as well as increasing the resolution of the camera. To match images while changing the distance and the tilt of the pen axis between two images, a large amount of calculation is required, which is unrealistic in real time.

  In this embodiment, in order to solve this problem, another three-axis acceleration sensor or three-axis gyro is mounted in addition to the three-axis acceleration sensor 6 provided for detecting the position coordinates, and these two accelerations are mounted. Is used to obtain the angular acceleration of the pen, and as a result, the change in the tilt of the pen is detected.

As a result, two effects can be obtained. One is that the tilt of the pen can be obtained, so the above algorithm has a problem that the pen trajectory in the air cannot be obtained accurately when the pen is rotated while the pen is flying. This problem can be solved. did it.
The other is that a change in the tilt of the pen can be detected even while the pen is being drawn, and a change in the tilt of the pen axis can be estimated. Therefore, in the above-described two image matching, since the range in which the tilt of the pen is matched in addition to the moving distance can be limited, image matching within the range can be performed within a predetermined time.

  As described above, by providing another acceleration sensor or another three-axis gyro and measuring the rotation angle of the pen, the accuracy of the movement vector when the pen moves away from the paper surface is improved, and By performing measurement in consideration of the inclination of the pen axis in image matching when the pen performs drawing on the paper surface, it is possible to improve the estimation of the pen movement vector.

(Embodiment 4)
In the similarity calculation processing in S22 of FIG. 8, the similarity is calculated by taking the absolute value error of the pixel values of the two images, so that the two images are different from white (background) to black (line color). But the change from black to white is treated in the same way.

  Accordingly, in the difference between the two images, if there are a lot of newly drawn lines, the similarity will be separated. Therefore, in S22, the similarity between the two images is not simply taken, but the background color (white). Since the change from 1 to black (line color) is not similar between the two images, the movement distance can be extracted more accurately if the change in the pixel value is excluded from the measurement target.

  That is, in the first embodiment, the change from white to black is treated as “not similar” and the similarity is lowered, or the character is newly written on the paper, If this part is set to “not similar”, the evaluation will cause a negative effect that the matching score will be worse if many new characters are written. Therefore, it is better to operate this part without considering it in the similarity evaluation. Can be matched.

  Therefore, S13 is calculated by the following formula.

  As described above, in this embodiment, it is possible to extract a movement distance that is more accurate than in the first embodiment.

1 is a configuration diagram of an electronic pen in Embodiment 1. FIG. FIG. 2 is a control block diagram of inkjet in Embodiment 1. 5 is a diagram illustrating pen line thickness control by duty modulation in Embodiment 1. FIG. FIG. 3 is a diagram illustrating a configuration of a position detection circuit using a triaxial acceleration sensor in the first embodiment. It is a figure which shows the algorithm which obtains the coordinate position using the triaxial acceleration sensor in Embodiment 1. FIG. It is a figure which shows the example of writing of the character image | photographed with the camera in Embodiment 1. FIG. 6 shows an algorithm of a pen coordinate position analysis task based on image data in the first embodiment. 3 is a diagram illustrating an image matching algorithm according to Embodiment 1. FIG. It is a figure which shows the algorithm which improved the algorithm of FIG. 5 in Embodiment 1. FIG. FIG. 3 is a diagram illustrating a relationship between a three-axis absolute space and paper plane coordinates in the first embodiment. FIG. 6 is a diagram illustrating a basis vector creation algorithm in the first embodiment. It is a figure which shows the algorithm which added correction to Task2 of FIG. 7 in Embodiment 1. FIG. It is a figure which shows the algorithm which acquires the coordinate position by limited matching in Embodiment 2. FIG. FIG. 10 is a diagram illustrating an example of movement of an electronic pen according to a third embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Electronic pen 2 Inkjet nozzle 3 Pen tip 4 Pressure sensor 5 Color ink cartridge 6 Triaxial acceleration sensor 7 Lens 8 CMOS image sensor 9 Processor 10 Memory 11 Battery 13 RF communication circuit 17 Inkjet 20 Paper surface 103 (103a, 103b, 103c, 103c ) Valve 604 Selector 901 CPU
902 AD converter 903 DMA

Claims (7)

In a pen-type data input device that acquires characters or figures written on a writing target medium as electronic data,
Acceleration detecting means for periodically detecting acceleration generated when the pen-type data input device is moved;
Speed calculating means for calculating the moving speed of the pen-type data input device based on the acceleration detected by the acceleration detecting means ;
A speed storing means for updating the speed value every time the speed is calculated by the speed calculating means;
First trajectory information for acquiring trajectory information indicating the trajectory moved by the pen-type data input device based on the previous speed value stored by the speed storing means and the current speed value calculated by the calculating means. Acquisition means;
Writing means for writing the character or the figure on the writing target medium using ink;
Imaging means for imaging the character or the figure written by the writing means;
Second trajectory information acquisition means for acquiring trajectory information indicating a trajectory of the character or the graphic written by the writing means based on a plurality of images of the character or the graphic imaged by the imaging means;
Detecting means for detecting whether the pen tip of the pen-type data input device is in contact with or away from the writing target medium;
The first trajectory information acquisition unit is selected when the detection unit detects that the pen tip is separated, and the second trajectory information acquisition unit is detected when the pen tip is detected to be in contact. Selecting means for acquiring the trajectory information by selecting
Correction means for correcting the value of the speed stored by the speed storage means to 0 when the detection means detects that the pen tip is in contact with the writing target medium;
A pen-type data input device.   The second trajectory information acquisition unit compares the image captured this time by the imaging unit with the previously captured image, and based on the comparison result, the character or the character written by the writing unit The pen-type data input device according to claim 1, wherein trajectory information indicating the trajectory of the figure is generated.   The second trajectory information acquisition means, when writing by the writing means, a range for the comparison based on the trajectory information acquired this time by the first trajectory information acquisition means and the trajectory information acquired last time. The pen-type data input device according to claim 2, wherein the pen-type data input device is determined. The pen-type data input device further includes:
The pen-type data input device according to claim 1, further comprising: a transmission unit that transmits the trajectory information acquired by the first or second trajectory information acquisition unit to an external device.   The pen-type data input device transmits the trajectory information by the transmission means when communication with the external device is connected, and the trajectory information when the communication with the external device is disconnected. The pen-type data input device according to claim 4, wherein the pen-type data input device is held. In a data input program for causing a computer in a pen-type data input device that acquires characters or figures written on a writing target medium as electronic data to execute processing for acquiring the electronic data,
Acceleration detection processing for periodically detecting acceleration generated when the pen-type data input device is moved;
A speed calculation process for calculating the moving speed of the pen-type data input device based on the acceleration detected by the acceleration detection process ;
A speed saving process that saves while updating the speed value every time the speed is calculated by this speed calculation process,
First trajectory information for acquiring trajectory information indicating the trajectory moved by the pen-type data input device based on the previous speed value saved in the speed saving process and the current speed value calculated by the calculating means. Acquisition process,
A writing process for writing the character or the figure on the writing target medium using ink;
An imaging process for imaging the character or the graphic written by the writing process;
A second trajectory information acquisition process for acquiring trajectory information indicating a trajectory of the character or the graphic written by the writing process based on a plurality of images of the character or the graphic captured by the imaging process;
A detection process for detecting whether the pen tip of the pen-type data input device is in contact with or away from the writing target medium;
The first trajectory information acquisition process is selected when it is detected that the pen tip is separated by the detection process, and the second trajectory information acquisition process is performed when it is detected that the pen tip is in contact. A selection process for obtaining the trajectory information by selecting
A correction process for correcting the value of the speed stored by the speed storage process to 0 by detecting that the pen tip is in contact with the writing target medium by the detection process;
Data input program for causing a computer to execute.   In the second trajectory information acquisition process, the image captured this time by the imaging process and the image captured last time are compared, and the characters or characters written by the writing process based on the comparison result are compared. The data input program according to claim 6, wherein trajectory information indicating the trajectory of the figure is generated.

Images