JP2011034548A - System and method for acquiring handwritten pattern - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method capable of taking a content handwritten on paper into a database without needing special paper. <P>SOLUTION: The handwritten pattern acquisition system includes a pen unit 1 having a pen tip and a moving image camera 3 for imaging the vicinity thereof, and a processing unit which determines a trace of the moving pen tip between a series of frames of taken moving image. In the pen unit, a paper fingerprint that is an irregular pattern formed on a surface of paper by paper making is taken by the camera. The processing unit includes an extraction processing unit which extracts a plurality of feature points each showing a local feature of the paper fingerprint on each frame image; a correspondence processing unit which determines each corresponding feature point between sequential frame images; and a trace processing unit which determines a position change of the pen tip to the paper surface based on a position change of each corresponding feature point in the sequential frames to determine a handwritten pattern as the trace of the pen tip. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a handwritten pattern acquisition system and acquisition method for acquiring a handwritten pattern in real time using a pen equipped with a camera.

  Handwriting (handwritten content) with "paper and pen" is the most historical information generation and recording medium for humanity. There is a lot of people who use paper notebooks, and many people take lecture notes as paper notes. Therefore, if such handwritten content can be digitized and imported into a database, it will be easy to save, search, and duplicate data, and life log that can save everyday handwritten content, so-called writing life log ("writing-life-log") ") Can be realized.

  Based on this idea, several devices that acquire handwritten patterns in real time have already been proposed. An example of this is a digital pen developed by Sweden's Anoto, the product name Anoto penDocuments (hereinafter referred to as Anotopen or Anoto System for simplicity) (eg [Search February 12, 2009] Internet <URL: http: //www.anoto.com/>, see) The Anoto system reads a fine dot pattern pre-printed on a paper surface with a pen camera, determines which position on which paper is described, and saves the locus (handwriting) of the pen tip. When the document is printed on paper, the handwriting can be correctly arranged on the document by associating the paper with the document.

More specifically, the Anoto system is a system consisting of “ANOTO paper”, a special paper with a dot pattern printed on it, and an “ANOTO pen” with a small camera and Bluetooth-compatible communication function. Handwriting information is obtained by recognizing the dot pattern of the special paper with a pen camera. The dot patterns are arranged in a grid pattern at intervals of about 0.3 mm, and are slightly displaced from the orthogonal grid. The deviation is in four directions, up, down, left, and right, and this deviation pattern is read by the camera built in the pen. The range that the camera reads at one time is 6 × 6 36 dots, and different positional information can be obtained by combining 36 shifts. This feature is a combination of ²⁷² ways, and one point can be recognized from an area as wide as the Eurasian continent. Further, various functions can be given to the paper by combining the dot patterns. For example, it may be possible to easily transmit data by creating an area for transmitting accumulated data to a PC or the like.

  Another example of a pen with a camera is PaperLink by Arai et al. (See Non-Patent Document 1, for example). This is a combination of a pen-type small camera and a fluorescent line marker, and can handle a paper document like hypertext. Specifically, the user presses the button on the pen and traces on the paper with the fluorescent line marker, and cuts out the pattern of the portion with the camera. If an action is defined for the clipped area, the defined action is activated by photographing the area later.

  In addition, Iwata et al. Have proposed a method for restoring a handwriting using a document image retrieval technique for retrieving a document and / or an image (hereinafter referred to as a document image) written on a paper and arranging it on the document (for example, Non-Patent Document 3).

  In order to actually obtain handwritten content from an image captured by a camera, it is necessary to restore the entire image from a handwritten pattern captured in fragments in each frame of the moving image. The technique used for this is so-called video mosaicing (see, for example, Non-Patent Document 2). Specifically, the correspondence relationship between the feature points is obtained between two adjacent frames of the current frame image and the immediately preceding frame image, and the posture change between the frames is obtained based thereon. If this posture change is obtained over all adjacent frames, the posture change series for all frames can be grasped. Then, by combining the frame images using them, it is possible to generate one image in which the entire image of the handwritten pattern is restored.

T. Arai, D. Aust, S. E. Hudson, "PaperLink: a technique for hyperlinking from real paper to electronic content," Proc. ACM Conf. Human Factors in Computing Systems (CHI'97), pp. 327-334, 1997. M. Irani and P. Anandan, "Video indexing based on mosaic representations," Proc. IEEE, vol. 86, no. 5, pp. 905-921, 1998. Kazumasa Iwata, Koichi Kise, Tomohiro Nakai, Masakazu Iwamura, Seiichi Uchida, Shinichiro Omachi, "Capturing Digital Ink as Retrieving Fragments of Document Images," Proceedings of the 10th International Conference on Document Analysis and Recognition (ICDAR2009), pp.1236-1240, (2009-7).

As described above, if the handwritten pattern can be automatically registered in the database as an image, the above-described writing life log can be realized.
However, since the Anoto system cannot use paper on which a special dot pattern is not printed, there is a problem that convenience is impaired. That is, the problem is a decrease in convenience due to the necessity of dedicated paper. The dot pattern can be used by printing on plain paper, but it is arranged approximately every 0.3mm, and a high-precision dedicated printer is required. Moreover, even if it is purchased as a notebook, it is expensive and the method for obtaining it is limited. For this reason, daily use of the Anoto system is expensive and time consuming.

There is a demand for a camera pen system that can perform the same function as described above without using a special dot pattern, that is, the restoration of handwriting on white paper and the placement of handwriting on a document.
In addition, the above-described methods of PaperLink and Iwata are based on the premise that a document or the like is described on the paper traced by the line marker.

  The present invention has been made in consideration of the above-described circumstances, and as a first problem, it is possible to capture a handwriting handwritten on a sheet as a content into a database without requiring a special sheet. Is to provide.

  As a second problem, the present invention provides a skillful integration between a technique for solving the first problem of obtaining a handwriting on paper and a technique for arranging on a document using a document image search technique. It is an object of the present invention to provide a technique capable of restoring a handwriting on a white paper and obtaining a writing position with respect to the document image when writing is performed on a paper on which the document image is previously recorded.

  In order to solve the first problem, the present invention focuses on paper fingerprints when detecting feature points from each frame image. The paper fingerprint is a concavo-convex pattern due to the fine structure of the paper observed on the surface of the paper. If a close-up of the paper surface is taken, it can be seen that the entanglement of the plant fibers forming the paper produces a random pattern. If, for example, a corner point or an edge point in this pattern can be detected, it can be used as a feature point for association. Such feature points that exist naturally on the paper have not been used so far. Therefore, this is one of the features of the present invention.

  The name “paper fingerprint” is derived from a technique for verifying the identity of paper by its pattern. For example, in the paper fingerprint verification technology XAYA developed by Fuji Xerox (for example, [Search February 12, 2009] Internet <URL: http://www.fujixerox.co.jp/company/technical/xaya/>) The paper surface pattern, that is, the paper fingerprint, is optically read by a scanner and recorded in a database or the like, and compared with the paper fingerprint of the inputted paper image at the time of identification. The paper can be identified with high accuracy.

  As a first invention for solving the first problem, the present invention moves between a pen unit having a pen tip and a moving image camera for photographing the vicinity of the pen tip and a series of frames of the captured moving image. And a processing unit that obtains a locus of a pen tip to be used, wherein the camera captures a paper fingerprint that is a concavo-convex pattern formed on the paper surface after the paper is made, and the processing unit is configured to display each frame image. An extraction processing unit that extracts a plurality of feature points each representing a local feature of a paper fingerprint reflected in the image, a corresponding processing unit that determines corresponding feature points between the preceding and following frame images, and a frame before and after each corresponding feature point A handwriting pattern acquisition system is provided that includes a locus processing unit that determines a change in the position of the pen tip with respect to the paper surface based on the change in position and obtains a handwritten pattern as a locus of the pen tip.

  Further, from a different point of view, the first invention relates to a paper surface near the pen tip when handwritten using a pen unit having a pen tip and a moving image camera that captures the vicinity of the pen tip. A photographing step, and a processing unit that obtains a locus of a pen tip that moves between a series of frames of the captured moving image, wherein the camera unit is formed with unevenness formed on the paper surface by the paper being made. A paper fingerprint that is a pattern is photographed, and the processing unit extracts a plurality of feature points that respectively represent local features of the paper fingerprint that appear in each frame image, and determines corresponding feature points between the preceding and following frame images. And providing a handwritten pattern acquisition method characterized by determining a position change of the pen tip relative to the paper surface based on a position change of each corresponding feature point in the front and back frames and obtaining a handwritten pattern as a locus of the pen tip. .

  In order to solve the second problem, the present invention integrates the first invention for solving the first problem and a technique for obtaining a position on a document using a document image search technique. The biggest obstacle to integration is the difference in shooting range required for operation. That is, the technique according to the first invention requires a high-definition image capturing a narrow range, whereas the technique of Iwata et al. Requires a wide range image capturing many characters. Although it is conceivable to use a wide-angle camera to obtain a wide range of images while performing close-up photography, geometric distortion due to the wide-angle camera is a big problem, and expensive cameras are required to meet the requirements for resolution, distortion, chromatic aberration, etc. Required. As a result of repeated investigations to find different solutions for the conflicting problems, the inventors have found that they can be handled by image mosaicing technology. That is, a larger image is constructed by mosaicking the image while photographing a narrow range and restoring the handwriting. When a sufficiently large image is obtained, the position of the handwriting in the document is obtained using document image retrieval. As will be described later, the inventors evaluated the effectiveness of the technique for solving the second problem based on experiments using a prototype system.

  As described above, the present invention also has, as a second invention for solving the second problem, an uneven pattern formed on a paper surface by having a pen tip and a moving image camera for photographing the vicinity thereof. A pen part that captures a paper fingerprint and a document image written on the paper surface with the video camera, and a local feature of the paper fingerprint is extracted as a paper surface feature point from each frame of the captured video, and is supported by the previous and next frames A first trajectory processing unit that obtains a trajectory of the pen tip with respect to the paper surface based on a movement amount of the paper surface feature point, and when a document image is captured in each frame, the local feature of the document image is used as the document image feature point. A second trajectory processing unit that extracts and obtains the position of the trajectory with respect to the document image based on the amount of movement of the corresponding document image feature point in the preceding and following frames, and the second trajectory processing unit includes the corresponding document image feature points. To overlap A combination of frames to determine the placement of one frame area larger than the document image feature points, to provide a handwriting pattern acquisition system characterized by obtaining a position of the track with respect to the document image based on the arrangement.

In the handwritten pattern acquisition system according to the first invention, the pen unit captures a paper fingerprint, which is a concavo-convex pattern formed on the paper surface after the paper is made, and the processing unit is captured in each frame image. An extraction processing unit for extracting a plurality of feature points each representing a local feature of a paper fingerprint, a corresponding processing unit for determining each corresponding feature point between the preceding and following frame images, and a position of each corresponding feature point in the preceding and following frames Since it has a trajectory processing unit that determines the change in the position of the pen tip relative to the paper surface based on the change and obtains a handwritten pattern as the trajectory of the pen tip, no special paper is required and the content handwritten on the paper is stored in the database. Can be captured. Moreover, since it is not a method of determining a locus from the handwritten pattern itself, it is strong against concealing the handwritten pattern. Furthermore, there is an excellent advantage that a so-called opening problem can be avoided in a state where a handwritten pattern captured between frame images does not change even when the pen tip moves.
The handwritten pattern acquisition method according to the first invention has the same advantages.

  In the handwritten pattern acquisition system according to the second aspect of the invention, the second trajectory processing unit combines document frames so that corresponding document image feature points overlap, and arranges document image feature points in a wider area than one frame. Since the position of the trajectory with respect to the document image is obtained based on the arrangement, it is possible to shoot a document image that requires shooting of a paper fingerprint and a wider area with a single moving camera. Therefore, the locus of the pen tip can be acquired even on white paper without performing special processing on the paper surface, and when a mark, a note, or the like is written on the document image previously recorded on the paper surface, the locus for the document image is obtained. Can be determined.

  In the present invention, the pen unit is actually used as a writing instrument for drawing letters, numbers, figures, patterns, etc. on general paper. That is, it is a ballpoint pen, a pencil or a felt pen. The moving image camera included in the pen unit is preferably a small camera built in, for example, a mobile phone or a computer. However, if the camera can capture time-series frame images, the size and the imaging device are not particularly limited. Absent.

  The processing unit according to the first invention and the first and second trajectory processing units according to the second invention may be integrated with the pen or may be separate. In the case of the integrated type, it has a storage device for storing the obtained handwritten pattern data, and the stored data may preferably be input to the host by communicating with the host computer wirelessly or by wire. Alternatively, the data may be configured to be copied or moved to the host via a standardized medium such as a memory card. The processing unit may be configured by hardware mainly including a microcomputer and a memory, and the function may be realized by the microcomputer executing a predetermined control program.

  When the pen unit and the processing unit are separate, they may be configured to communicate with the processing unit, preferably wirelessly or preferentially. The processing unit may be configured by hardware mainly including a microcomputer and a memory, and the function may be realized by the microcomputer executing a predetermined control program. Alternatively, the function may be realized by the CPU executing a predetermined application program on a general-purpose information processing apparatus such as a personal computer.

  The handwritten pattern acquired by the handwritten pattern acquisition system according to the first and / or second invention of the present invention may be stored and used as an external database as pattern data. For example, it is publicly known outside the system. The character may be converted into character data by the character recognition process and used as character data.

It is explanatory drawing which shows the concept of writing life log realization by this invention. FIG. 2 is an example of an image actually taken by the pen tip camera in this embodiment. It is an image which shows an example of the paper fingerprint based on this invention. In embodiment of this invention, it is explanatory drawing which shows the result of having actually calculated | required the point correspondence between adjacent frame images. It is an image showing a frame series of a nib image when the numeral “2” is written using the camera-equipped pen according to the present invention. It is explanatory drawing which shows the result of the video mosaic based on each frame image of FIG. FIG. 6 is an image obtained by scanning “2” actually written in the frame sequence of FIG. 5 with an image scanner. 6 is a graph showing the number of SURF feature point pairs in each frame of FIG. 5. It is explanatory drawing which shows the result of the video mosaic at the time of writing handwritten character "2" (it is the handwritten character of FIG. 10) different from FIG. 8 is an image obtained by scanning a handwritten character “2” different from FIG. 7 with an image scanner. 10 is a graph showing the number of feature point correspondences in each frame of the handwritten character of FIG. 9. It is explanatory drawing which shows the result of having extracted the SURF feature point of the paper fingerprint used by this invention from white paper. It is explanatory drawing which visualized the correspondence of the SURF feature point between two frames by this invention. It is explanatory drawing which shows the mode of the LLAH feature point extracted from the mosaic image which concerns on 2nd invention among this invention. It is explanatory drawing which shows the external appearance of the camera pen used by the embodiment which concerns on 2nd invention. It is explanatory drawing which shows the flow of the process which concerns on 2nd invention. In Experimental example 2 which concerns on 2nd invention, it is a graph which shows the result of the experiment which evaluates the handwriting which restored by performing 50 writing in the document area | region. In Experimental example 2, it is explanatory drawing which shows an example which the restoration result of the handwriting rotated. In Experimental example 2, it is explanatory drawing which shows the restoration result of a handwriting when writing in the margin in a document. In Experimental Example 2, it is explanatory drawing which shows the restoration result of a handwriting when writing widely in a document area. In Experimental example 2, it is explanatory drawing which shows the restoration result of a handwriting when writing in a small area | region.

Hereinafter, preferred embodiments of the present invention will be described.
The extraction processing unit determines a feature point whose position changes between different frame images as a feature point on the paper surface, determines a feature point whose position does not change as a feature point of the pen tip part, and displays the pen point part in each frame image The feature points may not be extracted from. In this way, processing is performed so as not to extract feature points from the pen tip, so that extraction of feature points that become noise when determining a change in position with respect to the paper surface is suppressed, and correspondence between feature points is obtained more accurately. Can do.

  Further, the correspondence processing unit may not correspond to the feature points in an area within a predetermined range from the pen tip of the subsequent frame. In this way, processing is performed so as not to extract feature points from a handwritten pattern portion that does not have a feature point corresponding to the previous frame, so that extraction of feature points that can be noise when suppressing position changes with respect to the paper surface is suppressed, The correspondence between feature points can be obtained more accurately.

  Furthermore, when the determination is made, the correspondence processing unit may determine the positional change of the corresponding feature point between the previous and subsequent frame images after correcting the projection distortion of the paper image that appears in each frame image. In this way, even when the pen portion is tilted with respect to the paper when handwritten on the paper, the projection distortion due to the correction is corrected. Therefore, the position of the trajectory can be obtained more accurately than without correction. be able to.

  The correspondence processing unit calculates a projection transformation matrix corresponding to each frame image in order to correct projection distortion, and obtains a pen tip locus for a frame whose elements are discontinuous with the projection transformation matrix of the previous and / or subsequent frames. It may be excluded from the frame. In this way, since a frame in which the pen tip locus is discontinuous is removed as noise, a handwritten pattern can be extracted more appropriately than when the processing is not performed.

  In addition, the correspondence processing unit may perform noise removal processing with an incorrect correspondence. In this way, it is possible to determine the position change between the previous and next frame images more accurately than when the processing is not performed.

Furthermore, the extraction processing unit may use a SURF algorithm as a feature point extraction method. The feature points extracted by the SURF method have the property that the feature quantity is invariant to rotation and scale change, and robust to illumination changes. In addition, since the processing is lighter than the SIFT method that is the basis of SURF, it is more suitable for application to video than SIFT. However, the feature point extraction method is not necessarily limited to SURF, and other local feature amounts such as SIFT and PCA-SIFT can be used.
Furthermore, the above-mentioned SIFT, SURF, etc. have both functions as a region detector and a feature descriptor. However, the functions of both are separated and either or both are used. May be substituted. Applicable region detectors include harris-affine, hessian-affine, MSER, and the like, and feature descriptors applicable to the present invention include shape context in addition to SIFT.

  In the second invention, a document image search unit for searching for a document image corresponding to the arrangement from a document image database in which a plurality of document images are registered in association with document image feature points extracted from the document image. The second trajectory processing unit further includes: a geometric image of the document image captured in each frame based on a correspondence relationship between the arrangement and the document image feature point associated with the document image searched by the search unit. The amount of distortion may be determined, and the locus may be corrected using the amount of distortion. In this way, when the search for the document image is successful, projective conversion to a plane that faces the paper surface can be performed, so that a handwriting with a smaller error than the handwriting obtained only from the paper fingerprint can be obtained.

The first trajectory processing unit preferably uses a SURF algorithm as a paper feature point extraction method. However, the feature point extraction method is not necessarily limited to SURF, and other local feature amounts such as SIFT and PCA-SIFT can be used.
Furthermore, the above-mentioned SIFT, SURF, etc. have functions as a region detector and a feature descriptor. However, the functions of both are separated, and either one or both are separated by other methods. May be substituted. Applicable region detectors include harris-affine, hessian-affine, MSER, and the like, and feature descriptors applicable to the present invention include shape context in addition to SIFT.

Further, the second trajectory processing unit may extract the centroid of the connected component and represent each feature point using the LLAH algorithm as a document image feature point extraction method. Note that the present invention is not necessarily limited to LLAH, and it is not possible to use the feature amount extracted by applying the SURF or SIFT algorithm and the method of Noguchi et al. It is not possible. However, the feature quantity obtained by applying LLAH (LLAH feature point) is compact and does not require capacity, whereas the feature quantity obtained by applying SURF or SIFT requires a large capacity. When using feature values obtained by applying SURF, SIFT, etc., it is expected that processing will take a longer time and the database will be larger than LLAH feature values. Therefore, it is preferable to use the LLAH feature amount.
The various preferable aspects shown here can also be combined.

Hereinafter, the present invention will be described in more detail with reference to the drawings. In addition, the following description is an illustration in all the points, Comprising: It should not be interpreted as limiting this invention.
First, an embodiment corresponding to the first invention of the present invention will be described.
In this embodiment, the principle for obtaining the entire handwritten pattern from the camera-equipped pen will be described. Next, feature point detection from paper fingerprints will be described, and a method for estimating posture change between frames using the feature points as a clue will be described. Finally, a method for obtaining the entire pattern as one image by video mosaicing will be described. . The experimental results are shown, and finally future issues are described.

≪Overview of configuration and processing of pen with camera≫
FIG. 1 is an explanatory diagram showing the concept of realizing a writing life log according to the present invention. Fig.1 (a) is a perspective view which shows schematic structure of the pen with a camera which concerns on this embodiment.
FIG. (B) is an explanatory diagram showing how handwritten content is captured into a database using video mosaicing technology. As shown in FIG. 1A, in this embodiment, a pen with a camera equipped with a micro CCD camera (product name: Plumnet Handymini, model name: CCN3412Y) is used at the pen tip. The pen tip camera captures a handwritten pattern and a paper surface during writing. Since the camera is fixed to the pen, the pen is always in the same position in the field of view. FIG. 2 shows an example of an image actually taken by the pen tip camera in this embodiment.

  In order to actually obtain the entire handwritten content, as shown in FIG. 1B, it is necessary to restore the entire image from the fragmented handwritten pattern photographed as each frame image of the moving image. This process is called a so-called video mosaicing process. For this reason, in this embodiment, the feature points corresponding to the adjacent two frame images of the frame image to be processed and the immediately preceding frame image are known SIFTs (more specifically, details of SURF, SURF, for example, H. Bay, T. Tuytelaars, and LV Gool, "SURF: speeded up robust features," Proc. ECCV2006 (LNCS volume 3951), part 1, pp. 404-417, 2006.) . If a feature point common to both frame images can be extracted, a positional change with respect to the paper surface between frames can be estimated from their correspondence. Specifically, a projective transformation matrix indicating the correspondence between each frame image in a state where the paper surface has undergone projection distortion and the paper surface is obtained between all adjacent frames. Then, after correcting the projection distortion of each frame image, the positional change on the paper surface is estimated. Based on these results, a positional change on the paper surface over a series of frames, that is, a locus of movement of the pen tip can be obtained.

  By the way, various forms of the position of the camera can be considered. For example, if a camera is attached in the vicinity of the pen butt, it can be assumed that a wider character area can be photographed. Therefore, there is a possibility that the entire character (and the entire paper surface) can be captured at a time, which is advantageous over a pen tip camera that requires mosaicing. However, the pen butt camera may be occluded (closed, concealed) by the hand holding the pen. For applications that require pen movement itself, it may be difficult to estimate movement with a pen butt having a large amplitude.

  Thus, the nib camera plays a complementary role depending on the mounting position. By mounting a camera on the pen tip, the vicinity of the handwritten pattern can be photographed more reliably. Furthermore, since the paper fingerprint can be used effectively as will be described later, it is possible to estimate the detailed movement of the pen tip. In this way, it is necessary to determine the mounting position suitable for the processing purpose while keeping in mind that the roles of the pen tip and pen butt cameras are different.

≪Feature point detection from paper fingerprint≫
In the present invention, attention is paid to the paper fingerprint as a reference for acquiring the position on the paper surface. As mentioned above, paper has a geometric pattern on its surface. FIG. 3 is an image showing an example of a paper fingerprint according to the present invention. The image in FIG. 3 is an enlarged view of a part of FIG. However, the contrast is emphasized after adjusting the brightness for easy viewing. If the positional relationship (that is, the movement amount and direction between frames) of each frame image can be grasped by using this paper fingerprint, the locus handwritten on the paper surface can be acquired even if the handwritten pattern is not successfully photographed. This method is extremely effective in the following two ways.

  First, there is avoidance of the concealment problem caused by the pen tip. For example, when the pen tip is visible as shown in FIG. 2, if the pen moves from the right to the left of the paper, the handwritten pattern is hidden by the pen tip portion and cannot be seen at all. Therefore, it is impossible to estimate the movement amount by paying attention to the handwritten pattern. On the other hand, there is no problem even if the handwritten pattern cannot be seen if the amount of movement is known from the paper fingerprint.

  Secondly, there is an avoidance of a handwritten pattern opening problem. Consider a situation in which a horizontal line continues to be written as a handwritten pattern from left to right on the page. In this case, the same handwritten pattern on the screen can always be seen, so it cannot be determined whether the pen is moving or stationary. This is an aperture problem in motion estimation. The horizontal line is an extreme example, but when writing a character, a pattern that does not change locally often occurs, so an unnatural amount of movement estimation occurs at that point, resulting in a handwritten pattern. The shape can be non-linearly stretched. On the other hand, focusing on the paper fingerprint, the paper fingerprint moves when the pen is moving, and conversely, the paper fingerprint is stationary when the pen is stationary.

  The feature points are preferably rotation / scale invariants and robust to brightness changes. The former is because the frame image is rotated by the rotation of the pen, and further, the distance relationship between the camera position and the paper surface changes depending on the stroke. In order to extract feature points stably even in such a situation, invariants with respect to rotation and scale changes (rotation / scale invariants) are desirable. Strictly speaking, an invariant with respect to the projective transformation is desirable, but since the displacement between adjacent frames is not so large, it is considered that a rotation / scale invariant can be approximated. On the other hand, the latter is for eliminating the influence of shadows appearing in a part of each frame image.

  In this embodiment, feature point detection and feature description based on the framework of SIFT (Scale-Invariant Feature Transform) are used to satisfy these requirements. As is well known, features described in SIFT are invariant to rotation and scale changes, and are robust to changes in lighting. In this specification, SURF (Speed-up Robust Features), which is a high-speed version of SIFT, is used.

≪Video mosaic ≫
By obtaining a plurality of pairs of points having SURF features that are very similar between adjacent frames, it is possible to estimate the posture change between adjacent frames. When the paper surface is flat, the frame images taken with the camera pen are projected and converted to each other (a kind of geometric conversion. In addition to the conversion and projection conversion represented by (3), there is an affine transformation in which a rectangle in the depth direction is represented by a parallelogram. Therefore, if projective transformation is estimated from the point correspondence between adjacent frames, the adjacent frames can be superimposed using the projection transformation. If this process is performed over all frames, a series of frames can be superimposed. This is so-called video mosaicing.

The technical significance of obtaining the point correspondence based on the similarity of the SURF features when obtaining the corresponding points between adjacent frames will be described.
As mentioned in the description of video mosaicing, the first step in the process is to establish several point correspondences between adjacent frames.
That is, when a certain point A in the t-th frame image is noticed, the point B in the t + 1-th frame image corresponding to the point B is found by using the similarity between the two points. At this point in time, the projective transformation between the two frames has not been estimated yet, so even if the two points A and B are originally the same point on the paper, their surroundings are not the same. Accordingly, it is necessary to find a pair of similar points as stably as possible even if there is an influence of such projective transformation.
In this respect, since SURF is rotation and scale invariant, if the features near points A and B are described in SURF, respectively, the SURF features are almost similar unless projective transformation is large. Therefore, it can be expected that the corresponding point B can be found by searching for a point having characteristics similar to A in the t + 1-th frame image.
In actual processing, a large number of SURF feature points are detected once in both frames, and point correspondences between frames are determined by simply finding a plurality of point pairs whose Euclidean distance between features is less than or equal to a threshold value. It will be.

  FIG. 4 is an explanatory diagram showing a result of actually obtaining a point correspondence relationship between adjacent frame images in the embodiment of the present invention. Points with similar SURF features are connected by line segments. Since many SURF feature points are detected on the paper surface, it is difficult to see as a figure. However, if you look closely, you can see that many of them are associated with the same position on the paper surface.

  From the figure, it can be seen that there are cases in which a completely wrong point correspondence is given. When a projection transformation matrix is obtained including this large and incorrect point correspondence, the adverse effect is magnified depending on the method, and an erroneous projection transformation matrix may be obtained. Since paper fingerprints are expected to be limited, it would be appropriate to consider such an incorrect point correspondence rather unavoidable.

  For this reason, a so-called robust estimation method is required for the projection transformation matrix estimation. Therefore, in this specification, RANSAC (MA Fischler and RC Bolles, "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography," Comm. Of the ACM, vol. 24, no .6, pp. 381 -395, 1981.). RANSAC is a method for obtaining a projection transformation matrix with a small number of point correspondences and evaluating how much other point correspondences can be explained by the projection transformation matrix. By performing this evaluation while randomly changing the point-corresponding set for obtaining the projective transformation matrix, it is possible to robustly obtain the projective transformation matrix.

There are three devices specific to the present invention when projective transformation is estimated from the correspondence between feature points in an inter-frame image. These are listed below.
The first is the removal of feature points that appear in the nib portion. Because the camera is fixed to the pen, the pen portion is always in a fixed position in the image. Accordingly, even if the feature point of the paper fingerprint portion moves between frames, the feature point of the pen portion appears to be stationary. For this reason, if the movement amount is estimated by combining these, the feature point of the pen has an adverse effect, and an incorrect result is obtained. For this reason, SIFT feature points appearing in the pen part need to be ignored.

  The second is removal of a handwritten pattern (black ink) near the pen tip. The handwritten pattern near the pen tip is a newly written portion during the time from the immediately preceding frame to the current frame. For this reason, there is no corresponding point in the immediately preceding frame, and in the worst case, an erroneous correspondence occurs. Therefore, it is necessary to ignore this part as well as the pen tip part. The difference between the current video mosaic of the nib image and the normal video mosaic is that the present invention is intended for patterns that are being dynamically generated.

  The third is ignoring the wrong projective transformation matrix. Basically, handwriting is performed continuously, so that the projective transformation itself should be continuous. However, there is a case where a projective transformation matrix suddenly completely wrong may be obtained due to instability of correspondence with SURF feature points. In such a case, the current frame may be skipped and projective transformation may be performed between the previous frame and the next frame. Since most of the frames overlap, it can be said that skipping a few frames has little effect. However, if the gap is too long due to continuous skipping, a large pose change occurs between the two frames, and it is difficult to handle SURF features.

≪Experimental example 1≫
FIG. 5 is an image showing a frame sequence of a pen tip image when the numeral “2” is written using the camera-equipped pen according to the present invention. The paper used was copy paper (non-recycled paper), and the size of “2” written was approximately 3.5 cm 2.5 cm. The number of frames was approximately 340 frames. Since the camera's frame rate is 30 fps, this corresponds to approximately 11 seconds, because it was written slowly to avoid motion blur.

  FIG. 6 is an explanatory diagram showing the result of the video mosaic based on each frame image of FIG. On the mosaic image, a small black circle (●) is plotted at the pen tip position in each frame. This series of black circles is a restored handwritten pattern. FIG. 7 is an image obtained by scanning “2” actually written in the frame sequence of FIG. 5 with an image scanner.

  Compared to FIG. 7, the restored image of FIG. 6 is quite jaggy, but it can still be seen as “2”. In this case, since the mosaic image is simply created by sequentially pasting, the projection conversion estimation error accumulates. If the initial frame does not face the paper, it affects the whole. For this reason, the shape tends to be unstable in principle. Still, this level of restoration indicates that there is a promise for mosaicing using the features of paper fingerprints.

  FIG. 8 is a graph showing the number of SURF feature point pairs in each frame of FIG. The horizontal axis is the ID of each frame, and the vertical axis is the number of SURF feature point pairs. The vertical line in the figure represents that skipping was performed because an unnatural projection transformation matrix was found in that frame. Detailed examination is an issue for the future, but when considered in conjunction with FIG. 6, there appears to be a tendency that the number of feature point pairs is reduced and the accuracy tends to be reduced near skipping. Specifically, many skips are observed near the inflection point and the upper part of "2", and the restoration pattern near the latter is still jaggy.

  FIG. 10 is an image obtained by scanning a handwritten character “2” different from that in FIG. 7 with an image scanner. FIG. 9 is an explanatory diagram showing the result of the video mosaic when “2” in FIG. 10 is written. This “2” is approximately 1.6 cm 1.1 cm in size and smaller than that of FIG. It is written in 118 frames in total, that is, about 4 seconds. Although some nonlinear expansion / contraction is observed, the accuracy of “2” is clearly understood. FIG. 11 is a graph showing the number of feature points corresponding to each frame of the handwritten character of FIG.

  This time, no processing such as placing a weight on the black ink part is performed, and the projective transformation is estimated based on the SURF feature obtained without distinction between the paper and the ink. The inventors have published the following article, “Katsuhiro Ito, Seiichi Uchida, Masakazu Iwamura, Shinichiro Omachi, Koichi Kise,“ Restoring handwritten patterns from nib camera images, ”ISS Special Meeting 2008 ISS Special Program As shown in “Student Poster Session, ISS-P-323, 2008.”, in fact, mosaicing is possible with considerable accuracy by just evaluating the overlay of black ink. Therefore, in the future, it may be possible to use a paper fingerprint only when the black ink is hidden behind the pen tip.

  In addition, the technology according to the present invention is disclosed in the following literature, “Kazuhiro Tanaka, Seiichi Uchida, Masakazu Iwamura, Shinichiro Omachi, Koichi Kise,“ Basic study on data embedding pen, ”Journal of Human Interface Society, vol. 10, no. 4, pp. 559-567, 2008. ”can be used in combination with the“ information embedding pen ”proposed. With this information embedding pen, various information (for example, URL, writer ID, etc.) can be embedded in handwritten content by applying small ink dots simultaneously with writing on paper. By registering the entire shape of the handwritten content obtained in this invention and the embedded information as a pair in the writing life log, it is possible to add a cybermedia function to the handwritten content.

  As described above, the present invention provides a specific method of a video mosaicing method for restoring a handwritten pattern from an image from a camera attached to a pen tip. The greatest feature of the present invention is to use a pattern (paper fingerprint) on the paper. By extracting SURF feature points from paper fingerprints and using them for video mosaicing, for example, even in a situation where the handwritten pattern itself is concealed by the pen tip, the pen movement, that is, the handwritten pattern can be restored. Although it was an extremely early study stage, it was found that the outline of a handwritten pattern could be restored even with such a simple method.

  Possible future improvements include: First, the image quality conversion for emphasizing the paper fingerprint before finding the SURF feature, or combining with the feature with the paper fingerprint feature point while emphasizing the correspondence relationship of the black ink part, etc. Stabilization can be considered. Furthermore, it is considered important to improve the method of video mosaicing. In this specification, since the method of repeating alignment between adjacent frames is employed, errors may accumulate as the frame is later, and the mosaicing result may be corrupted. Use of re-appearance points (Akihiko Ikeya, Tomokazu Sato, Kiyoshi Ikeda, Noriyuki Kanbara, Noboru Nakajima, Naokazu Yokoya, "Super-resolution video mosaic for paper based on camera parameter estimation," Theory of Science, vol. J88- D-II, no. 8, pp. 1490-1498, 2005)) is considered essential. It may be important to consider removing motion blur. Depending on the moving speed of the pen, motion blur becomes significant, and SURF feature points cannot be detected, and as a result, matching cannot be performed. Therefore, it is important to remove motion blur, but in the frame where the handwriting is shown, the blurring that makes use of the fact that the handwriting is a linear pattern (for example, XY Qi, L. Zhang, CL Tan, "Motion deblurring for optical character recognition, "Proc. ICDAR2005, pp. 389-393, 2005.) is also conceivable.

  As for evaluation, quantitative evaluation of restoration accuracy and resistance measurement against writing speed can be considered. Further, in the present invention, since a mosaic image is obtained using a paper fingerprint as a clue, if the influence of paper quality is also investigated, an improvement based on the knowledge obtained as a result can be considered.

  Next, the second invention will be described. In order to make the explanation easy to understand, first, the first invention as the basis of the second invention and the technique using the document image search technique will be described.

<< Method according to the first invention >>
The technique of the first invention will be briefly summarized again. In the first invention, a paper fingerprint is used to deal with a blank paper or a blank portion. A paper fingerprint can be taken by taking a close-up of the paper surface. By extracting feature points from this pattern and determining the amount of movement of the pen tip between each frame, the movement of writing can be grasped. Thereby, it is possible to restore the handwriting that does not require special paper.

  Here, the extracted feature points need to be invariant to rotation and scale changes. This is because the rotation of the pen and the change of the angle with respect to the paper surface occur during writing. Therefore, SURF is used as feature point extraction and feature description. SURF features are invariant to rotation and scale changes, and are robust to lighting changes. FIG. 12 shows the result of extracting SURF feature points from white paper.

  The amount of movement of the nib coordinate can be calculated by obtaining a projective transformation matrix from the point correspondence of the SURF feature points. FIG. 13 is a visualization of the point correspondence. Similar SURF feature points are connected by a line, and it can be seen that many points are correctly handled. However, there are cases where incorrect point correspondences are taken. If a projective transformation matrix is obtained including such an incorrect point correspondence, an erroneous projective transformation matrix may be obtained. Therefore, RANSAC is used as a robust estimation method. RANSAC is a technique for obtaining a projective transformation matrix from randomly selected point correspondences and evaluating how much the other point correspondences can be explained by the projection transformation matrix. By repeating this evaluation, a projective transformation matrix is robustly obtained.

≪Method using document image search technology≫
There is a method using LLAH (Locally Likely Arrangement Hashing) as a pen system with a camera that restores writing on a document (Tomohiro Nakai, Koichi Kise, Masakazu Iwamura, “Real-time document image search using a Web camera”, (See IEICE Transactions D, J90-D, 8, pp.2262-2265, Aug. 2007.) In this method, writing information is obtained by the following procedure.
1. Take a document on paper with a camera attached to the pen.
2. Extract the center of gravity (LLAH feature point) of the connected component from the obtained image, and perform document image search using LLAH. As a result, a corresponding document image and a projective transformation matrix for the image are obtained.
3. Estimate the coordinates of the pen tip in the document image from the projective transformation matrix.
4. Evaluate whether the estimated pen tip coordinates are valid, and if so, record the coordinates.

  Handwriting information is obtained by repeating the above process and connecting the estimated pen tip positions (see Non-Patent Document 3). Since this system obtains feature points from a document printed on paper, it does not require special paper as an object of writing. In addition, the written document and the writing position can be specified.

≪Problems when integrating the two≫
The problem with both is that they cannot handle all of their daily writing alone. There are various daily writings, from writing notes on blank paper to underlining the document. Therefore, there is a need for a system that can determine the document name and the writing position on the document when writing on the document while supporting writing on a blank sheet. The technique according to the first invention restores writing using feature points extracted from a paper fingerprint. By using paper fingerprints, it is possible to restore writing on general paper, but it does not take into account the relationship between writing on a document printed on paper and writing. On the other hand, the technique using the document image search technique makes it possible to see the relationship between writing and a document. However, since LLAH used as a document image search method extracts LLAH feature points necessary for search from printed characters, it is impossible to restore writing unless the area where the document is printed is photographed.

≪Solution according to the second invention≫
The system proposed in the second invention can acquire handwriting even on white paper without performing special processing on the paper surface, and when the writing destination is a document, the writing position on the document can be obtained. . In order to implement the second invention, it is necessary to solve the problem between the technique according to the first invention and the technique using document image retrieval.

  A major problem between the two methods is the problem of the camera installation position. The camera installation position is a problem because different fields of view are required when obtaining a large amount of information from a blank sheet and when obtaining a large amount of document information. For example, a paper fingerprint is a fiber of paper and has very fine features, so that a high resolution image is required for photographing. Therefore, by installing the camera close to the pen tip, it is possible to stably capture a paper fingerprint and to restore handwriting with high accuracy. On the other hand, when a document image search is performed, if a large number of document areas can be photographed, it becomes easy to distinguish between documents. Therefore, the search accuracy is improved by installing the camera away from the pen tip. Thus, in order to improve the accuracy of handwriting restoration and document image search, the opposite camera installation positions are required. At this time, it is possible to attach two cameras to the pen, but in terms of practicality, it can be said that there should be one camera.

  Further, as a problem of the technique according to the first invention, there is an accumulation of errors while continuously obtaining the projective transformation matrix. If projective transformation is repeated using information only between adjacent frames, errors are accumulated in later frames, and the accurate pen tip position cannot be estimated. For this reason, the shape of the characters is lost compared to actual writing.

1. Coping with the camera installation position problem The camera installation position problem is addressed by using image mosaicing technology. Image mosaicing technology is a technology that combines frames taken to create images that are equivalent to a wide area (for example, Tomokazu Sato, Akihiko Ikeda, Kiyoshi Ikeda, Noriyuki Kambara, Noboru Nakajima, Naokazu Yokoya, "Super-resolution video mosaicing for planes based on camera external parameter estimation," Proceedings of the 9th Pattern Measurement Symposium, pp. 13-20, Nov. 2004.). In the second invention, a projection transformation matrix for each frame is obtained from the correspondence of the SURF feature points. Thus, each frame image is projectively transformed to obtain a mosaic image, thereby obtaining a captured image with a wide field of view. FIG. 14 shows the state of LLAH feature points extracted from the mosaic image. As a result, the number of LLAH feature points can be increased, and the accuracy of document image search can be improved.

  The camera pen actually used in the second invention is shown in FIG. A micro CCD camera (Asahi Electronics Laboratory NCM03-K) is attached to the middle of the pen body 11 (camera 13 in FIG. 15). An ultraviolet light 15 is attached to the upper part of the camera 13. The reason for attaching the ultraviolet light 15 is that it is difficult to obtain a feature point from a paper fingerprint in a white light environment. Since white light such as natural light or fluorescent light is strongly reflected on the paper surface, the paper surface photographed by the camera 13 becomes bright, and the paper fingerprint is hidden by the light. Therefore, in order to make it easy to extract feature points from a paper fingerprint, it is necessary to apply light that does not reflect strongly to the paper surface. Therefore, by attaching an ultraviolet light 15 to the pen body 11, a feature point can be obtained from the paper fingerprint even if the camera 13 is installed at a position away from the paper surface.

2. Reappearance of feature points The problem of handwriting misalignment due to accumulated errors is addressed by examining the reappearance of SURF feature points. This is because characters often intersect or return to their original positions, so if you can take correspondence with SURF feature points extracted in the past when shooting the same area, projection with less accumulation error This is because conversion is possible. For example, for a character that intersects halfway and is connected to the original line, such as “8”, the camera 13 captures the same area. For this reason, once the SURF feature points that have appeared once are stored, and when passing through the same region again, errors can be corrected by taking correspondence with all the stored SURF feature points.

Here, if all the past SURF feature points are preserved, the number will increase over time. As the number of SURF feature points increases, the time required for point correspondence calculation increases. Therefore, it is possible to speed up the search for corresponding points using a hash (for example, Kazuhito Noguchi, Koichi Kise, Masakazu Iwamura, "High-speed object recognition by multi-step approximate nearest neighbor search," Image Recognition and Understanding Symposium) (See MIRU2007), OS-B2-02, pp.111-118, July, 2007. The hash value is calculated using 16 dimensions of the SURF feature quantity having 64 dimensions. The SURF feature is

When

Binarization using, bit vector
Create Here, μ _j is the median value of each dimension of the feature amount obtained from the image taken for the preliminary experiment. And
The hash value is obtained by Here, mod is the remainder operation, and H _size is the _size of the hash table.

≪Process flow≫
A specific processing flow in the second invention will be described. The flow of processing is shown in FIG. First, the SURF feature points are extracted from the paper fingerprints and printed characters of the photographed paper. Next, a point correspondence is obtained by comparing with the feature value extracted from the hash table. From this point correspondence, a projective transformation matrix with a reference frame image is obtained. Then, using this matrix, the position of the pen tip 17 on the plane is obtained. At the same time, for the feature points for which no corresponding points are found, the coordinates of the feature points are projected and registered in the hash table. By repeating this process, a series of coordinates of the nib 17 can be obtained. In addition, image mosaicing is performed every m (> 1) frame intervals. Then, LLAH feature points are extracted from the mosaic image every n (> m) frame intervals, and a document image search is performed if a certain amount of feature points is obtained. If the feature points do not exceed a certain amount, image mosaicing is continued. When the search is performed, if it is determined that the result is correct, the handwriting is restored on the document image by projective transformation of the obtained coordinates of the pen tip 17 and connecting them. When a document image search is performed, the mosaicing image is initialized. This is to prevent accumulation of projection transformation errors.

≪Experimental example≫
Written on the document and evaluated the results. Copy paper (recycled paper) was used as the target paper. The frame rate of the camera 13 attached to the pen body 11 is 30 fps. Mosaiking was performed every 10 frames, and document image retrieval was performed every 30 frames. This value was set based on knowledge obtained from preliminary experiments. Moreover, the writing speed per character was 5 to 10 seconds, and writing was performed relatively slowly. This is to avoid motion blur. Letters and figures were written with a single continuous line. The reason for the single continuous line is that the up / down of the nib 17 is not considered at present. In addition, the size and shape of the characters and figures are not unified but arbitrary. The writing during the experiment includes photographs taken over a blank area and a document area.

  Evaluation of the experimental results was performed by comparison with handwriting information obtained with a pen tablet. As an evaluation value, the coincidence ratio between the answer created by the pen tablet and the handwriting made from the second invention was obtained. However, the handwriting information obtained from the pen tablet is misaligned compared to the actual writing. This is because the coordinates of the image file and the possessed paper do not coincide with each other due to the inclination of the paper surface and the setting of the margin that occur during printing. Therefore, the evaluation was performed while allowing a deviation that can be visually judged to be the same level of restoration. In the experiment, the allowable deviation range was about 3 mm.

FIG. 17 shows the result of writing 50 pieces in the document area. In FIG. 17, the degree (accuracy) that the handwriting coincides with the answer as an experimental result is shown in 10 steps in 10% increments over a range of 0 to 100%. The average processing time per frame was 128 ms.
FIG. 15 is from the point correspondence of SURF feature points. It is the result of evaluating the restored handwriting. Here, the reason why the evaluation value of many results is around 30% is that the obtained handwriting cannot correct the tilt of the pen body 11 and the rotation of the camera 13. FIG. 18 shows an example in which the handwriting restoration result is rotated. In the experiment, a reference frame image is determined, and handwriting is restored with respect to the plane of the frame image. When the reference frame is inclined with respect to the vertical and horizontal directions of the paper, the result is expressed with an inclination as shown in FIG. For this reason, the evaluation value in FIG. 15 decreased due to not only handwriting disturbance due to projective transformation error but also distortion and rotation due to the positional relationship of the camera 13.

  FIG. 15 shows the ratio of matching with the answer image when the document image search is performed and the handwriting is restored on the document. In FIG. 15, the evaluation value increased compared to FIG. 15. This is because when the document image search is successful, projective transformation to a plane that faces the paper surface can be performed. By projective transformation on the paper, rotation and distortion of the handwriting as shown in FIG. 18 can be corrected, and the answer image is approached. FIG. 19 shows a restoration result obtained by writing in the margin in the document. As shown in FIG. 19, if the surrounding document area can be photographed, the position of the margin area can also be obtained on the document. Moreover, FIG. 20 is shown as an example of a result with a high evaluation value. As shown in FIG. 20, the accuracy of the document image search is increased when writing widely in the document area. This is because many feature points can be extracted by mosaicing because the nib 17 moves greatly. On the other hand, an example in which the evaluation value is lowered is shown in FIG. As shown in FIG. 21, when writing in a small area, the accuracy of the document image search was reduced, and a positional shift occurred. This is because there are many frame images that capture the same region, and the number of feature points does not increase even when mosaicing is performed. To solve this problem, LLAH needs to be improved. Specifically, it is necessary to improve the feature amount so that a highly accurate search is possible even in a situation where the number of feature points is small.

  In FIG. 17, the result that the evaluation value becomes 0% is that the tracking of the handwriting or the document image search by LLAH has failed. In this experiment, the biggest reason for failure in tracking and searching is the large change in the captured image due to the experimental environment. Natural light and strong light from a fluorescent lamp are the ones that most affect a photographed image. In the experiment, a camera pen with an ultraviolet light 15 was used as a countermeasure. However, the camera pen in FIG. 15 is not designed to block light from the outside. Therefore, in the experiment, the degree of light entering from the outside differs for each image to be photographed, and the result depends on it. This problem can be solved by making the camera pen unaffected by disturbances based on the results obtained from this experiment.

  In addition to the embodiments described above, there can be various modifications of the present invention. These modifications should not be construed as not belonging to the scope of the present invention. The present invention should include the meaning equivalent to the scope of the claims and all modifications within the scope.

1: Pen part
2: Video camera
5: Database
11: Pen body
13: Camera
15: UV light
17: Nib

Claims (12)

A pen unit having a pen tip and a video camera for photographing the vicinity thereof;
A processing unit that obtains a locus of a pen tip that moves between a series of frames of a captured moving image;
In the pen unit, the camera shoots a paper fingerprint which is a concavo-convex pattern formed on the paper surface after the paper is made,
The processing unit is an extraction processing unit that extracts a plurality of feature points each representing a local feature of a paper fingerprint captured in each frame image, a corresponding processing unit that determines corresponding feature points between the preceding and following frame images, A handwriting pattern acquisition system comprising a trajectory processing unit for determining a pen tip position change with respect to a paper surface based on a position change of each feature point in the front and back frames and obtaining a handwritten pattern as a pen tip trajectory .   The extraction processing unit determines a feature point whose position changes between different frame images as a feature point on the paper surface, determines a feature point whose position does not change as a feature point of the pen tip part, and displays the pen point part in each frame image The system according to claim 1, wherein no feature points are extracted from the system.   3. The system according to claim 1, wherein the correspondence processing unit does not correspond to feature points in an area within a predetermined range from a pen tip of a subsequent frame.   The said corresponding | compatible process part determines the position change of the corresponding feature point between back-and-front frame images, after correcting the projection distortion of the paper surface which appears in each frame image, when determining. The described system.   The correspondence processing unit calculates a projection transformation matrix corresponding to each frame image in order to correct projection distortion, and obtains a pen tip locus for a frame whose elements are discontinuous with the projection transformation matrix of the previous and / or subsequent frames. The system according to claim 4, wherein the system is excluded from the frame.   The system according to claim 1, wherein the correspondence processing unit performs noise removal processing of an erroneous correspondence relationship.   The system according to claim 1, wherein the extraction processing unit uses a SURF algorithm as a feature point extraction method. A step of photographing the paper surface near the pen tip when the handwriting using the pen unit having the pen tip and the moving image camera for photographing the vicinity of the pen tip is made on the paper;
A processing unit comprising a step of obtaining a locus of a nib that moves between a series of frames of a captured moving image;
The camera unit photographs a paper fingerprint which is a concavo-convex pattern formed on the paper surface after the paper is drawn, and the processing unit is a plurality of feature points respectively representing local features of the paper fingerprint shown in each frame image The corresponding feature points between the preceding and following frame images are determined, the position change of the pen tip relative to the paper surface is determined based on the position change of the corresponding feature points in the previous and next frames, and the locus of the pen tip is determined. The handwritten pattern acquisition method characterized by calculating | requiring the handwritten pattern of this. A pen unit that has a pen tip and a moving image camera that captures the vicinity of the pen tip, and that captures a paper fingerprint that is a concavo-convex pattern formed on the paper surface by paper and a document image recorded on the paper surface by the moving camera; ,
First trajectory processing for extracting a local feature of a paper fingerprint from each frame of a captured moving image as a paper surface feature point and obtaining a trajectory of the pen tip with respect to the paper surface based on a movement amount of the corresponding paper surface feature point in the preceding and following frames And
When a document image is captured in each frame, a local feature of the document image is extracted as a document image feature point, and the position of the trajectory with respect to the document image based on the amount of movement of the corresponding document image feature point in the preceding and following frames A second trajectory processing unit for obtaining
The second trajectory processing unit determines the placement of document image feature points in a wider area than one frame by combining the frames so that corresponding document image feature points overlap, and the trajectory for the document image based on the placement. Handwritten pattern acquisition system characterized by obtaining the position of A document image search unit for searching for a document image corresponding to the arrangement from a document image database in which a plurality of document images are registered in association with document image feature points extracted from the document image;
The second trajectory processing unit calculates a geometric distortion amount of the document image captured in each frame based on a correspondence relationship between the arrangement and the document image feature point associated with the document image searched by the search unit. The system according to claim 9, wherein the system determines and corrects the trajectory using the distortion amount.   The system according to claim 9 or 10, wherein the first trajectory processing unit uses a SURF algorithm as a paper feature point extraction method.   The system according to any one of claims 9 to 11, wherein the second trajectory processing unit extracts, as a document image feature point extraction method, a centroid of a connected component and represents each feature point using an LLAH algorithm.