JP4985436B2 - Position detection apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To identify an instructed position on a medium even if there is a gap between the instructed position on the medium and a position found by photographing a printed image. <P>SOLUTION: A position detection device includes; an image reading part 21 which reads out images; a dot sequence generation part 22 which detects dots from the images to generate a dot sequence; a block detection part 23 which detects blocks from the dot sequence to generate a code sequence; a synchronous code detection part 24 which detects synchronous codes from the code sequence; an identification code detection part 30 which detects identification codes on the basis of the synchronous code; an identification code decoding part 32 which decodes the identification codes to obtain identification information; an X-coordinate code detecting part 40 and a Y-coordinate code detecting part 45 which detect coordinates codes on the basis of the synchronous codes; an X-coordinate code decoding part 42 and a Y-coordinate code decoding part 47 which decode the coordinates codes to obtain coordinates information; and a coordinates correcting part 50 which converts the relative coordinates of the tip of a pen on a photographed image to those on a medium to find the coordinates of the tip of the pen on the medium in addition to its coordinates information. <P>COPYRIGHT: (C)2009,JPO&amp;INPIT

Description

  The present invention relates to a position detection device and a program.

  A method for providing a position code for encoding a plurality of positions on a surface is known (see, for example, Patent Document 1). In this patent document 1, a cyclic numeral sequence is printed a plurality of times along the surface. At that time, using different rotations of the cyclic numeral sequences so that a predetermined deviation occurs between the adjacent numeric strings, a plurality of code windows divided on the surface include at least three cyclic numeral sequences and adjacent codes. It has one numeric string that overlaps one numeric string of the window, and the position of the code window is encoded by a shift between the cyclic numeric strings belonging to the code window.

  There is also known a technique that enables spatially asynchronous reading of encoding from a recording medium that records machine-readable encoding of digital quantum that is logically ranked (see, for example, Patent Document 2). In this document, multiple copies of an encoding that are essentially identical can be formed, and a machine-recognizable spatial synchronization index can be combined with the print position within each of the copies of the encoding to allow spatial synchronization of the encoding. A large number of cases are provided, and these cases are written in a spatially periodic center grid in the recording medium according to the layout rules.

  Systems and methods for determining the position of captured images from larger images are also known (see, for example, Patent Document 3). In this Patent Document 3, a non-repetitive sequence is folded into a non-repetitive sequence unique to each sub-window of a predetermined size, and the captured position is determined for each sub-window in the non-repetitive sequence. Thus, the position of the captured image is determined from the image of the larger subwindow.

  A technique that enables long-time recording and repeated reproduction of multimedia information that is optically readable is also known (see, for example, Patent Document 4). In this patent document 4, a recording apparatus uses a printer system or a printing plate making system as a dot code that can optically read so-called multimedia information including audio information, video information, digital code data, etc. And record on a medium such as paper. The pen-type information reproducing device sequentially captures the dot code according to the manual scanning of the dot code, and the original audio information is output by the audio output device, the original video information is displayed by the display device, and the original digital code data Is output by a page printer or the like.

  There is also known a technique that enables precise detection of coordinates on a medium without using a tablet (see, for example, Patent Document 5). In this patent document 5, a pen-type coordinate input device optically reads a code symbol formed on a medium and indicating the coordinates on the medium, decodes the read code symbol, and reads the code symbol in the read image. The position, orientation, and amount of distortion are detected. Then, the coordinates of the position of the tip on the medium are detected based on the decoded information and the position, orientation, and distortion amount of the code symbol.

JP-T-2003-511762 JP-A-9-185669 JP 2004-152273 A JP-A-6-231466 JP 2000-293303 A

Here, in general, the position obtained by capturing the print image is strictly different from the instructed position on the medium.
An object of the present invention is to make it possible to specify an instructed position on a medium even if there is a deviation between the instructed position on the medium and a position obtained by capturing a print image.

According to the first aspect of the present invention, an instruction unit that indicates a position on a medium and printed on the medium , unit images are arranged at n (1 ≦ n <m) of m (m ≧ 3). Image acquisition means for acquiring a picked-up image obtained by picking up a print image in which a group of pattern images in which Ax (Ax ≧ 2) in the horizontal direction and Ay (Ay ≧ 2) in the vertical direction are arranged periodically are arranged Extraction means for extracting a partial image constituting a portion of the captured image from the captured image, and a first array generation for generating a first array that stores the presence or absence of the unit image in the partial image as an element And first information acquisition for acquiring first position information indicating a relative position of the position corresponding to the position on the medium instructed by the instruction means on the plane including the captured image with respect to the partial image . and means, on said first sequence, The frame formed by arranging Ax pieces of the same size as the pattern image in the horizontal direction and Ay pieces in the vertical direction is moved to a frame position where the number of the divisions including the n unit images is equal to or more than a predetermined number. A second array generating means for generating a second array that is an array in the frame at the frame position and a movement amount in the horizontal direction and the vertical direction of the frame up to the frame position; Using the second array, second information acquisition means for acquiring second position information indicating the position of a portion corresponding to the second array of the print image on the medium; and the first Converting means for converting the position information into third position information indicating a relative position of the position on the medium instructed by the instruction means with respect to the print image, the second position information, and the movement amount. The portion of the printed image on the medium based on A first specifying means for specifying the position of the portion corresponding to the image, and the position of the portion corresponding to the partial image of the print image, based on said third position information, indicated by said indicating means And a second specifying unit that specifies a position on the medium.
The invention according to claim 2, wherein the first information obtaining means, the further obtains the angle information indicating the tilt angle with respect to the captured image of the partial image, and the converting means, using the angle information, the The position detection device according to claim 1, wherein the first position information is converted into the third position information.
According to a third aspect of the present invention, the partial image is a quadrilateral, and the first information acquisition unit includes first angle information indicating an inclination angle of the first side of the partial image with respect to the captured image. The second angle information indicating an inclination angle of the second side adjacent to the first side of the partial image with respect to the captured image is acquired as the angle information. This is a position detection device.
According to a fourth aspect of the present invention, the first information acquisition unit further acquires scale information indicating a scale in the captured image, and the conversion unit uses the scale information to obtain the first position information. The position detecting device according to claim 1, wherein the position detecting device is converted into the third position information.
According to a fifth aspect of the present invention, the partial image is a quadrilateral, and the first information acquisition unit includes a first scale indicating a scale in the captured image along the first side of the partial image. The information and second scale information indicating a scale in the captured image along a second side adjacent to the first side of the partial image are acquired as the scale information. 4. The position detection device according to 4.
The invention according to claim 6, wherein the first information obtaining means, according to claim 4 based on the image distance corresponding to the unit image on the captured image, and acquires the scale information It is a position detection apparatus of description.
According to the seventh aspect of the present invention, a pattern image printed on a medium and arranged with unit images at n (1 ≦ n <m) of m (m ≧ 3) is displayed in a horizontal direction on an Ax ( A function of acquiring a captured image obtained by capturing a print image in which Ax ≧ 2) and Ay (Ay ≧ 2) pattern images arranged in the vertical direction are periodically arranged; and from the captured image, A function for extracting a partial image constituting a part, a function for generating a first array for storing the presence or absence of the unit image in the partial image as an element , and an instruction means on a plane including the captured image A function of obtaining first position information indicating a relative position of the position corresponding to the position on the medium with respect to the partial image, and a section having substantially the same size as the pattern image on the first array. Ax in the horizontal direction, the vertical The second array, which is an array within the frame at the frame position, by moving the Ay arranged frames to the frame position where the number of sections including the n unit images is equal to or greater than a predetermined number; Corresponding to the second array of the print images on the medium , using the function of generating the horizontal and vertical movement amounts of the frame up to the frame position and the second array A function for obtaining second position information indicating the position of a part; and third position information indicating the relative position of the position on the medium instructed by the instruction means with respect to the print image. Based on the second position information and the movement amount, the function of specifying the position of the part corresponding to the partial image of the print image on the medium, and the print image the position of the portion corresponding to the partial image, said first Based positional information and the a program for realizing a function of specifying a position on the medium designated by the instruction means.
Invention of claim 8, wherein the further implemented to the computer a function of acquiring angular information indicating an inclination angle with respect to the captured image of the partial image, the function of the conversion, by using the angle information, the first 8. The program according to claim 7, wherein the position information of 1 is converted into the third position information.
The invention according to claim 9 further causes the computer to realize a function of acquiring scale information indicating a scale in the captured image, and the function of converting uses the scale information to calculate the first position information. The program according to claim 7, wherein the program is converted into the third position information.

According to the first aspect of the present invention, it is possible to specify the designated position on the medium even when there is a deviation between the designated position on the medium and the position obtained by capturing the print image. Have.
According to the second aspect of the present invention, even if there is a difference in angle between a captured image obtained by capturing a print image and a partial image extracted from the captured image in order to obtain a position on the medium, the indicated position on the medium Has the effect of being able to specify.
The invention of claim 3 has the effect that the indicated position on the medium can be specified even when the partial image extracted from the captured image is deformed, for example, as a parallelogram. .
The invention of claim 4 has the effect that the instructed position on the medium can be specified even if there is a difference in scale between the printed image and the captured image obtained by capturing it.
The invention according to claim 5 has an effect that the indicated position on the medium can be specified even when the partial image extracted from the captured image is deformed, for example, as a parallelogram. .
According to the sixth aspect of the present invention, when the designated position on the medium on which the print image in which the unit image is arranged in the predetermined area is specified, the print image is compared with the case where the configuration is not provided. And the difference between the scales of the captured image and the captured image can be obtained with high accuracy.
According to the seventh aspect of the present invention, it is possible to specify the instructed position on the medium even if there is a deviation between the instructed position on the medium and the position obtained by capturing the print image. Have.
According to the eighth aspect of the present invention, even if there is a difference in angle between a captured image obtained by capturing a print image and a partial image extracted from the captured image in order to obtain a position on the medium, the indicated position on the medium Has the effect of being able to specify.
The invention of claim 9 has the effect that the indicated position on the medium can be specified even if there is a difference in scale between the printed image and the captured image obtained by capturing it.

The best mode for carrying out the present invention (hereinafter referred to as “embodiment”) will be described in detail below with reference to the accompanying drawings.
First, the encoding method used in this embodiment will be described.
In the encoding method according to the present embodiment, mCn is represented by a pattern image (hereinafter referred to as “code pattern”) in which unit images are arranged at n (1 ≦ n <m) locations selected from m (m ≧ 3) locations. (= M! / {(Mn)! × n!}) Express information. That is, instead of associating one unit image with information, a plurality of unit images are associated with information. Assuming that one unit image is associated with information, there is a drawback in that erroneous information is expressed when the unit image is lost or noise is added. On the other hand, for example, if two unit images are associated with information, it can be easily understood that there is an error when the number of unit images is one or three. Furthermore, in the method of expressing 1 bit or 2 bits at most in one unit image, a synchronous pattern for controlling reading of the information pattern is expressed by a pattern visually similar to the information pattern for expressing the information. I can't. For this reason, the present embodiment employs the above encoding method. Hereinafter, such an encoding method is referred to as an mCn method.
Here, a unit image having any shape may be used. In the present embodiment, a dot image (hereinafter simply referred to as “dot”) is used as an example of a unit image, but an image having another shape such as a hatched pattern may be used.

FIG. 1 shows an example of a code pattern in the mCn system.
In the figure, a black area and a hatched area are areas where dots can be arranged, and a white area between them is an area where dots cannot be arranged. Of the areas where dots can be arranged, dots are arranged in black areas and dots are not arranged in hatched areas. In other words, FIG. 1 shows an example in which a total of 9 dots can be arranged in 3 × 3, and (a) shows that 2 dots are placed in 9 dots that can be arranged. An example of the code pattern in the 9C2 method to be arranged, (b) shows an example of the code pattern in the 9C3 method in which 3 dots are arranged in an area where nine dots can be arranged.

  However, the dots (black areas) arranged in FIG. 1 are only for information representation, and do not coincide with the dots (minimum squares in FIG. 1) that are the minimum units constituting the image. In this embodiment, when “dot” refers to the former dot and the latter dot is referred to as “pixel”, the dot has a size of 2 pixels × 2 pixels at 600 dpi. . Since the length of one side of one pixel at 600 dpi is 0.0423 mm, the length of one side of one dot is 84.6 μm (= 0.0423 mm × 2). The larger the dot, the more likely it will be noticeable, so it is preferable to make it as small as possible. However, if it is too small, printing with a printer becomes impossible. Therefore, the above value is adopted as the dot size, which is larger than 50 μm and smaller than 100 μm. However, the above value 84.6 μm is a numerical value to the last, and is about 100 μm in the actually printed toner image.

Next, FIG. 2 shows an example of all code patterns in the 9C2 system. Here, a space between dots is omitted. As shown in the figure, in the 9C2 system, 36 (= ₉ C ₂ ) code patterns are used. Also, a pattern value that is a number for uniquely identifying each code pattern is attached to all code patterns. In the figure, an example of assignment of the pattern value to each code pattern is also shown. However, the correspondence shown in the figure is merely an example, and any pattern value may be assigned to any code pattern.

FIG. 3 shows an example of all code patterns in the 9C3 system. Also here, the space between dots is omitted. As shown in the drawing, in the 9C3 system, 84 (= ₉ C ₃ ) code patterns are used. Also in this case, a pattern value which is a number for uniquely identifying each code pattern is attached to all code patterns. In the figure, an example of assignment of the pattern value to each code pattern is also shown. However, in this case as well, the correspondence shown in the figure is merely an example, and any pattern value may be assigned to any code pattern.

Here, the size of the area where the code pattern is arranged (hereinafter referred to as “pattern block”) is set so that 3 dots × 3 dots can be arranged. However, the size of the pattern block is not limited to this. That is, the size may be 2 dots × 2 dots, 4 dots × 4 dots, and the like.
Further, the shape of the pattern block may not be a square, but may be a rectangle as in the case of arranging 3 dots × 4 dots, for example. In this specification, a rectangle means a rectangle in which the lengths of two adjacent sides are not equal, that is, a rectangle other than a square.
Further, the number of dots to be arranged in an arbitrarily determined number of dot arrangement areas may be appropriately determined in consideration of the amount of information to be expressed and the allowable image density.

  Thus, in this embodiment, mCn types of code patterns are prepared by selecting n locations from m locations. Among these code patterns, a specific pattern is used as an information pattern, and the rest is used as a synchronization pattern. Here, the information pattern is a pattern representing information to be embedded in the medium. The synchronization pattern is a pattern used for extracting an information pattern embedded in the medium. For example, it is used for specifying the position of the information pattern or detecting the rotation of the image. Any medium may be used as long as an image can be printed. Since paper is a representative example, the medium will be described as paper below, but metal, plastic, fiber, or the like may be used.

Here, a synchronization pattern among the code patterns shown in FIG. 2 or FIG. 3 will be described. When these code patterns are used, since the shape of the pattern block is a square, it is necessary to recognize the rotation of the image in units of 90 degrees. Therefore, a set of synchronization patterns is constituted by four types of code patterns.
FIG. 4A shows an example of a synchronization pattern in the 9C2 system. Here, of the 36 types of code patterns, 32 types of code patterns are information patterns representing 5-bit information, and the remaining 4 types of code patterns constitute a set of synchronization patterns. For example, the synchronization pattern in which the code pattern of the pattern value “32” is upright, the synchronization pattern in which the code pattern of the pattern value “33” is rotated 90 degrees to the right, and the code pattern of the pattern value “34” is rotated 180 degrees to the right The synchronization pattern is a synchronization pattern obtained by rotating the code pattern of the pattern value “35” to the right by 270 degrees. However, the method of distributing the 36 types of code patterns to the information pattern and the synchronization pattern is not limited to this. For example, 16 types of code patterns may be used as information patterns expressing 4-bit information, and the remaining 20 types of code patterns may constitute five sets of synchronization patterns.

  FIG. 4B is an example of a synchronization pattern in the 9C3 system. Here, out of 84 kinds of code patterns, 64 kinds of code patterns are information patterns expressing 6-bit information, and the remaining 20 kinds of code patterns constitute five sets of synchronization patterns. The figure shows two of the five synchronization patterns. For example, in the first set, a synchronization pattern in which the code pattern of pattern value “64” is erected, a synchronization pattern in which the code pattern of pattern value “65” is rotated 90 degrees to the right, and a code pattern of pattern value “66” The synchronization pattern rotated 180 degrees to the right and the code pattern of pattern value “67” are the synchronization pattern rotated 270 degrees to the right. In the second set, a synchronization pattern in which the code pattern of pattern value “68” is erected, a synchronization pattern in which the code pattern of pattern value “69” is rotated 90 degrees to the right, and a code pattern of pattern value “70” The synchronization pattern rotated 180 degrees to the right and the code pattern of pattern value “71” are the synchronization pattern rotated 270 degrees to the right.

  Although not shown, when the shape of the pattern block is a rectangle, two types of code patterns may be prepared as synchronization patterns in order to detect image rotation. For example, when an area in which 3 dots in the vertical direction and 4 dots in the horizontal direction should be detected but an area in which the 4 dots in the vertical direction and 3 dots in the horizontal direction can be arranged is detected, the image is 90 degrees or 270 degrees at that time. It is because it turns out that it is rotating.

Next, a minimum unit of information expression (hereinafter referred to as “code block”) in which a synchronization pattern and an information pattern are arranged will be described.
FIG. 5 shows an example of the layout of the code block.
In the drawing, the layout of the code block is shown on the right side of each of (a) and (b). Here, a layout in which 25 pattern blocks of 5 × 5 are arranged is employed. Of the 25 pattern blocks, a synchronization pattern is arranged in one block at the upper left. In addition, an information pattern representing X coordinate information for specifying the horizontal coordinate on the paper surface is arranged in the right four blocks of the synchronous pattern, and the vertical coordinate on the paper surface is specified in the four blocks below the synchronous pattern. An information pattern representing coordinate information is arranged. Further, an information pattern representing the identification information of the document printed on the paper surface or the paper surface is arranged in 16 blocks surrounded by the information pattern representing the coordinate information.

  Further, on the left side of (a), it is shown that a code pattern in the 9C2 system is arranged in each pattern block. That is, 36 types of code patterns are divided into, for example, 4 types of synchronization patterns and 32 types of information patterns, and each pattern is arranged according to a layout. On the other hand, the left side of (b) shows that a code pattern in the 9C3 system is arranged in each pattern block. That is, 84 types of code patterns are divided into, for example, 20 types of synchronization patterns and 64 types of information patterns, and each pattern is arranged according to a layout.

  In the present embodiment, the coordinate information is expressed in M series in the vertical direction and the horizontal direction on the paper surface. Here, the M series is a series in which the partial sequence does not coincide with other partial sequences. For example, the eleventh order M-sequence is a bit string of 2047 bits. The same sequence as the partial sequence of 11 bits or more extracted from the 2047-bit bit string does not exist in the 2047-bit bit string other than itself. In the present embodiment, one code pattern is associated with 4 bits. That is, a bit string of 2047 bits is represented by a decimal number for every 4 bits, a code pattern is determined according to the assignment in FIG. 2 or FIG. Therefore, at the time of decoding, the position on the bit string is specified by specifying three consecutive code patterns and referring to the table storing the correspondence between the code patterns and the coordinates.

FIG. 6 shows an example of encoding coordinate information using an M sequence.
(A) shows a bit string “0111000101011010000110010...” As an example of the 11th order M-sequence. In the present embodiment, this is divided into 4 bits, and the first partial sequence “0111” is the code pattern of the pattern value “7”, and the second partial sequence “0001” is the pattern value “1”. As the code pattern, the third partial sequence “0101” is arranged on the paper surface as the code pattern of the pattern value “5”, and the fourth partial sequence “1010” is arranged as the code pattern of the pattern value “10”. .

  Further, when the code patterns are divided into 4 bits and assigned to the code patterns in this way, as shown in (b), all pattern strings are expressed in 4 cycles. That is, since the 11th order M-sequence is 2047 bits, if this sequence is cut out every 4 bits and expressed by a code pattern, 3 bits will be added at the end. The first 1 bit of the M series is added to these 3 bits to form 4 bits, which are represented by a code pattern. Further, a cut-out code pattern is represented every 4 bits from the second bit of the M sequence. Then, the next cycle starts from the third bit of the M sequence, and the next cycle starts from the fourth bit of the M sequence. Furthermore, the fifth cycle starts from the fifth bit, which coincides with the first cycle. Therefore, if four cycles of the M sequence are cut out every 4 bits, it is possible to use all 2047 code patterns. Since the M sequence is 11th order, the three consecutive code patterns do not match any other continuous code pattern at any position. Therefore, decoding can be performed by reading three code patterns. However, in the present embodiment, coordinate information is expressed by four code patterns in consideration of the occurrence of errors.

In addition, although some methods can be used for encoding the identification information, RS encoding is suitable in the present embodiment. This is because the RS code is a multi-value coding method, and the pattern value of the code pattern arranged in each pattern block may correspond to the multi-value of the RS code.
In this embodiment, the code pattern is used by, for example, printing the identification information on a paper image over a document image, reading a partial image on the paper surface with a pen-shaped scanner, and then identifying the document image from there. It is assumed that information is acquired. In this case, an error occurs due to dirt on the paper surface or the performance of the scanner, but this error is corrected by the RS code.

Here, the correction by the RS code and the amount of information that can be expressed when performing such correction will be specifically described.
In the present embodiment, as described above, a code pattern in which the number of dots in one pattern block is constant is employed. Thereby, if one dot disappears or one dot is added, the number of dots in the pattern block changes. Therefore, these are errors that can be recognized as errors. On the other hand, if the disappearance and addition of dots occur at the same time, they are misrecognized as other code patterns.
For example, out of 16 blocks in which an information pattern representing identification information is arranged, 10 blocks are arranged as information blocks representing identification information itself, and 6 blocks are used as correction blocks. In this case, up to 6 blocks that are known to be errors and up to 3 blocks that are not known to be errors are corrected. If this is realized by 32 types of information patterns in the 9C2 system, for example, 5 bits of information is expressed by 1 block, and therefore the identification information itself is expressed by 50 blocks of 10 bits. Further, for example, if it is realized by 64 types of information patterns in the 9C3 system, 6 bits of information is expressed by one block, and therefore the identification information itself is expressed by 60 bits by 10 blocks.

Next, a wide layout including the code block will be described.
FIG. 7 is a diagram showing an example of such a layout. In this layout, the code blocks in FIG. 5 are periodically arranged as basic units in the vertical and horizontal directions of the entire sheet.
Here, as the synchronization pattern, the same code pattern is arranged in the upper left pattern block in each code block. In the figure, the synchronization pattern is represented by “S”.
As the X coordinate information, the same sequence of code patterns is arranged in each pattern block in the same row as the synchronization pattern is arranged. As the Y coordinate information, the same sequence of code patterns is arranged in each pattern block in the same column as the synchronization pattern is arranged. In the figure, patterns representing X coordinate information are represented by “X01”, “X02”,..., And patterns representing Y coordinate information are represented by “Y01”, “Y02”,.
Furthermore, as the identification information, the same arrangement of code patterns is periodically arranged in the vertical direction and the horizontal direction. In the drawing, patterns representing identification information are represented by “I01”, “I02”,..., “I16”.
By adopting such a layout, for example, when the range indicated by a circle in the figure is read, the range including the entire code block in FIG. 5 is not read. However, identification information and coordinate information can be obtained by the processing described later.

The code image printed on the paper surface in such a layout is formed with K toner (infrared light absorbing toner containing carbon) or special toner using, for example, an electrophotographic method.
Here, as the special toner, an invisible toner having a maximum absorption rate of 7% or less in the visible light region (400 nm to 700 nm) and an absorption rate of 30% or more in the near infrared region (800 nm to 1000 nm) is exemplified. . Here, “visible” and “invisible” are not related to whether they can be recognized visually. “Visible” and “invisible” are distinguished depending on whether or not an image formed on a printed medium can be recognized by the presence or absence of color development due to absorption of a specific wavelength in the visible light region. Further, “invisible” includes those that have some color developability due to absorption of a specific wavelength in the visible light region but are difficult to recognize with human eyes.

When the pen device indicates a predetermined position on the image on which such a code image is printed, an image sensor provided in the pen device reads a part of the image. Then, a dot pattern is read from this image and a decoding process is performed. However, in this case, the decoded coordinate information represents the position of the image read by the image sensor, and does not strictly represent the position designated by the pen device.
The occurrence of this deviation will be specifically described below.
FIG. 8 is a schematic view of a pen device for this explanation.
As shown in the figure, a point where the pen center line passing through the center of the pen device intersects with the paper surface (pen tip center) and a point where the image sensor center line passing through the center of the image sensor intersects with the paper surface (imaging area center) There is a misalignment IP unique to the pen device between them.

Further, the deviation between the decoded coordinate information and the position designated by the pen device includes not only this fixed deviation but also a deviation that is dynamically determined according to the result of the image processing. I will explain.
FIG. 9 shows an area (hereinafter referred to as “reading area”) that the pen device reads for decoding on an area (hereinafter referred to as “imaging area”) captured by the image sensor of the pen device, and the center point of the area. FIG. 6 is a diagram showing a point (hereinafter referred to as “the center of a reading area”) and a point indicated by a pen point of a pen device (hereinafter referred to as “a center of a pen point”).

  The pen device reads an area of 17 dots × 17 dots necessary for decoding from the imaging area. This is a reading area, but a reading area appropriate for decoding does not necessarily exist near the center of the imaging area. There may be a case where an appropriate reading area is shifted from the center due to the influence of illumination or the like. Further, the reading area is not necessarily rectangular, and is often deformed into a parallelogram (for example, rhombus) due to the rotation and inclination of the pen device. Therefore, in the drawing, a parallelogram reading area is indicated by a thick broken line in a lower right portion of the imaging area.

In the figure, two coordinate systems are shown on the imaging region.
One is a coordinate system on the captured image. In this coordinate system, the X axis is parallel to the horizontal frame of the imaging region, the right direction is the positive X direction, the Y axis is parallel to the vertical frame of the imaging region, and the downward direction is the positive Y direction. In the figure, the X axis on the captured image is represented as “X _CCD ”, and the Y axis on the captured image is represented as “Y _CCD ”.
The other is a coordinate system on the code image. This is a coordinate system for representing each point on the paper surface corresponding to each point on the captured image. In this coordinate system, the X axis is parallel to one direction of the dot row, the right direction is the positive direction of X, the Y axis is parallel to another direction of the dot row, and the downward direction is Y. The positive direction. In the figure, the X axis on the code image is represented as “X _CODE ”, and the Y axis on the code image is represented as “Y _CODE ”.

Among these coordinate systems, first, in the coordinate system on the captured image, parameters necessary for calculating the position of the nib center are defined.
That is, assume that the center of the reading area is C, the nib center is P, and the relative coordinates of P with respect to C are (P _X , P _Y ). In addition, the inclination angle of the X axis on the code image is θ _X , and the inclination angle of the Y axis on the code image is θ _Y. Here, these inclination angles are based on the vertical line of the imaging region, that is, the negative direction of the Y axis on the captured image, θ _X represents clockwise rotation, and θ _Y represents counterclockwise rotation as the positive direction. Furthermore, the distance in the direction of the dot of the X-axis on the code image and d _X, the distance between the direction of the dots of the Y-axis on the code image and d _Y.

Next, a method for calculating the position of the nib center on the code image using these parameters will be described.
First, in FIG. 9, the coordinates on the code image of the perpendicular foot drawn from P to the X axis on the code image are (H _X , 0), and the code image of the vertical foot drawn on the Y axis on the code image Let the coordinates above be (0, H _Y ). Then, the following relationship is established among P _X , P _Y , H _X , H _Y , θ _X , and θ _Y.

Accordingly, H _X and H _Y are expressed as follows using P _X , P _Y , θ _X , and θ _Y.

That is, the X coordinate and the Y coordinate of the nib center P are represented by numerical values on the X axis and the Y axis on the code image. However, H _X and H _Y obtained here are based solely on the scale on the captured image. That is, the dot interval on the code image is 4 pixels in both the X-axis direction and the Y-axis direction as shown in FIG. 1, but it is not necessarily the same length on the captured image. Therefore, as described above, it is necessary to perform scale-related correction by setting the dot intervals on the captured image to be d _X and d _Y.

FIG. 10 shows a state of coordinate conversion from the coordinate system on the captured image to the coordinate system on the code image. The reading area is a parallelogram as shown in FIG. 9 on the captured image before coordinate conversion, but is rectangular on the code image after coordinate conversion.
Here, the point on the code image corresponding to the center of the read area on the captured image is C, the point on the code image corresponding to the pen tip center P is Q, and the relative coordinates of Q with respect to C are (Q _X , Q _Y ). Then, Q _X and Q _Y are obtained as follows by applying corrections related to the dot spacing to H _X and H _Y obtained above.

Thereafter, the coordinates of the center of the reading area are calculated based on the pattern image in the reading area, and the coordinates of the nib center on the paper surface are calculated by adding the above-mentioned Q _X and Q _Y to the coordinates.
Hereinafter, the calculation of the coordinates of the nib center on the paper surface will be described in more detail. In the present embodiment, any mCn method may be used. However, in the following description, for the sake of simplicity, it is assumed that the 9C2 method is used. Hereinafter, the pattern block is also simply referred to as “block”.

First, the image processing apparatus 20 that reads and processes a code image formed on a paper surface will be described.
FIG. 11 is a block diagram illustrating a configuration example of the image processing apparatus 20.
As illustrated, the image processing apparatus 20 includes an image reading unit 21, a dot array generation unit 22, a block detection unit 23, and a synchronization code detection unit 24. Also, the identification code detector 30, the identification code decoder 32, the X coordinate code detector 40, the X coordinate code decoder 42, the Y coordinate code detector 45, the Y coordinate code decoder 47, and the coordinate correction Unit 50 and an information output unit 55.

The image reading unit 21 reads a code image printed on a paper surface using an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).
The dot array generation unit 22 detects dots from the read code image, and generates a dot array with reference to the dot positions. Note that, as preprocessing for dot detection from the code image, processing for removing noise included in the read image is also performed. Here, the noise includes, for example, variations in image sensor sensitivity and noise generated by an electronic circuit. The type of noise removal processing should match the characteristics of the imaging system, but sharpening processing such as blurring processing and unsharp masking may be applied. In addition, dot detection is performed as follows. That is, first, a dot image portion and other background image portions are separated by binarization processing, and a dot position is detected from each binarized image position. At that time, since there are cases where a lot of noise components are included in the binarized image, it is necessary to combine filter processing for determining dots based on the area and shape of the binarized image. Thereafter, the dot array is generated by replacing the dot detected as an image with digital data on the two-dimensional array, for example, “1” for the position where the dot is and “0” for the position where there is no dot. Do. In the present embodiment, the dot array generation unit 22 is provided as an example of an image acquisition unit that acquires a captured image and as an example of an extraction unit that extracts a partial image. Furthermore, as an example of the first position information indicating the relative position of the designated position on the medium on the plane including the captured image, relative coordinates (P _X , P _Y ) with respect to the reading area center of the nib center are used. The dot array generation unit 22 is provided as an example of a first information acquisition unit that acquires the first position information.

  The block detection unit 23 detects a pattern block in the code block on the dot array. In other words, a frame with the same size and shape as the code block and a block frame having the same size and shape as the pattern block are moved as needed on the dot array, and the position where the number of dots in the block is equal is the correct frame position. A code array storing pattern values in each block is generated.

  The synchronization code detection unit 24 detects the synchronization code with reference to the type of each code pattern detected from the dot array. The synchronization code detector 24 also determines image rotation based on the detected synchronization code. For example, when a square code pattern is used, it may be rotated by 90 degrees. Therefore, the direction is detected depending on which of the four types of synchronization patterns the detected synchronization code corresponds to. In addition, when a rectangular code pattern is used, there is a possibility that the pattern is rotated by 180 degrees. Therefore, the direction is detected depending on which of the two types of synchronization patterns the detected synchronization code corresponds to. Furthermore, the synchronous code detection unit 24 rotates the code array by the rotation angle detected in this way, and sets the code array in the correct direction.

The identification code detection unit 30 detects the identification code from the code array whose angle is corrected with reference to the position of the synchronization code.
The identification code decoding unit 32 decodes the identification code using the same parameters (such as the number of blocks) used in the RS code encoding process described above, and outputs identification information.

The X coordinate code detection unit 40 detects the X coordinate code from the code array whose angle is corrected with reference to the position of the synchronization code.
The X coordinate code decoding unit 42 extracts the M series partial sequence from the detected X coordinate code, refers to the position of this partial sequence in the M sequence used for image generation, and corrects this position with the shift amount of the code block The obtained value is output as X coordinate information. In the present embodiment, X coordinate information is used as an example of second position information indicating the position of a print image on a medium, and X coordinate is used as an example of second information acquisition means for acquiring the second position information. An encoding / decoding unit 42 is provided.

The Y coordinate code detection unit 45 detects the Y coordinate code from the code array whose angle is corrected with reference to the position of the synchronization code.
The Y coordinate code decoding unit 47 extracts the M series partial sequence from the detected Y coordinate code, refers to the position of this partial sequence in the M sequence used for image generation, and corrects this position with the shift amount of the code block. The obtained value is output as Y coordinate information. In the present embodiment, Y coordinate information is used as an example of the second position information indicating the position of the print image on the medium, and the Y coordinate is used as an example of a second information acquisition unit that acquires the second position information. An encoding / decoding unit 47 is provided.

The coordinate correction unit 50 corrects the coordinate information made up of the X coordinate information output from the X coordinate code decoding unit 42 and the Y coordinate information output from the Y coordinate code decoding unit 47 to obtain the center of the pen tip on the paper surface. Get coordinate information. Here, the correction is performed by using the relative coordinates of the pen tip center acquired by the dot array generation unit 22 with respect to the reading area center, the inclination angle of the reading area on the captured image, and the dot interval on the captured image. . In the present embodiment, as an example of the third position information indicating the relative position of the instructed position on the medium, the relative coordinates (Q _X , Q _Y ) with respect to the reading area center of the nib center are used, As an example of conversion means for converting the position information into the third position information, a coordinate correction unit 50 is provided. In addition, a coordinate correction unit 50 is provided as an example of a specifying unit that specifies the instructed position on the medium.
The information output unit 55 outputs the identification information obtained from the identification code decoding unit 32 and the coordinate correction unit 50, and the X coordinate information and Y coordinate information of the nib center.

Next, the operation of the image processing apparatus 20 will be described. In the description of this operation, it is assumed that 9C2 code patterns are arranged in the layout of FIG.
First, the image reading unit 21 reads a code image printed on a paper surface with an imaging element.

Next, the dot array generation unit 22 cuts out a region suitable for dot detection from the read code image. This region may be cut out by known image processing. For example, the following method can be considered. That is, first, the imaging area is divided into a plurality of small areas. In this case, each small area is larger than one dot. Next, for each small region, the contrast between the dot portion and the portion other than the dot is evaluated. Then, the surrounding small areas having the next highest contrast are connected with the small area having the highest contrast as the center. This process is repeated, and the area is expanded until it is large enough to cover the size of 17 dots × 17 dots, and the finally obtained area is cut out from the imaging area.
When an area suitable for dot detection is cut out in this way, the dot array generation unit 22 detects a dot from the area.

The dot array generation unit 22 also obtains the inclination angle of the dot row.
FIG. 12A shows an example of a method for obtaining the inclination angle of the dot row.
First, as shown in the figure on the left side, several inclination angles of straight lines connecting adjacent pairs of dots are obtained. Then, the first axis angle, the second axis angle, and the third axis angle are selected from these inclination angles, and are set as shown in the right figure. Here, the first axis angle is an inclination angle with the highest frequency. The second axis angle is an inclination angle having the highest frequency in the vicinity of an angle offset by 90 degrees from the first axis angle. The third axis angle is the inclination angle that is most frequently in the vicinity of the symmetrical angle of the second axis angle with respect to the angle offset by 90 degrees from the first axis angle. For example, the first axis angle and the angle closest to the angle of the second axis angle and the third axis angle offset by 90 degrees from the first axis angle are set as the two inclination angles of the dot row.

FIG. 12B shows an example of a method for more accurately obtaining the inclination angle of the dot row.
First, as shown in the upper left figure, the dot coordinates are converted into relative coordinates from the synchronization origin. In the figure, the intersection of the vertical axis and the horizontal axis is the synchronization origin. For example, a dot closest to the center of gravity of a plurality of detected dots may be set as the synchronization origin. Next, as shown in the upper right figure, the dot coordinates are rotated by θ. Furthermore, as shown in the lower right diagram, a grid is applied to the coordinates of the rotated dots, and for example, the amount of deviation of the coordinates of the dots from the vertical grid is calculated. Then, as shown in the lower left figure, the variance is calculated and integrated for each dot in the same row, and θ that minimizes the variance is set as the inclination angle. Note that the inclination angle is preferably estimated independently for the X direction and the Y direction.

Thereafter, the dot array generation unit 22 generates a dot array in which “1” is set at the position where the dot is detected and “0” is set at the position where the dot is not detected.
FIG. 13 is a diagram illustrating conversion from a dot image to a dot array by the dot array generation unit 22. As shown in the figure, the dot array generation unit 22 converts 17 × 17 dots in the reading area into a 17 × 17 dot array. Each element of this array stores “1” if there is a dot at the corresponding position, and “0” if there is no dot at the corresponding position. At this time, the deviation from the center of the imaging area at the center of the 17 dot × 17 dot reading area cut out for the decoding process is known. Therefore, relative coordinates (P _X , P _Y ) with respect to the reading area center of the nib center are obtained from the misalignment information and information about the misalignment between the nib center and the imaging area center which are known in advance, and are stored in a memory (not shown). Remember.

Next, the block detection unit 23 detects a pattern block on the dot array corresponding to the reading area.
FIG. 14 is a diagram specifically showing processing when a block block is moved to detect a pattern block. Here, it is known that encoding has been performed by the 9C2 system, and a case of decoding by the 9C2 system is shown.
First, the block detection unit 23 acquires a dot array from the dot array generation unit 22. The size of the dot array acquired here is preset, and is (number of pattern blocks necessary for decoding × number of dots on one side of pattern block + number of dots on one side of pattern block−1) ² . However, since this dot arrangement corresponds to an arbitrarily selected area of the image, the position where the pattern block is separated is unknown. Therefore, first, the block frames are overlapped with the end of the dot array as the detection start position. In this example, since m = 9, block frames including a frame of 3 dots × 3 dots are overlapped. Next, the dots in each frame are counted. In this example, since n = 2, the position of the block frame when there are 2 dots in each frame indicates the pattern block delimiter position. However, since the number of dots varies at this position, it can be seen that it is not a pattern block break position. Therefore, the number of dots in the frame is counted by shifting the block frame. That is, the dots are counted at the start position, the position moved by one dot in the right direction, and the position moved by two dots in the right direction. In addition, with respect to each of these positions, dots are also counted at a position moved by one dot downward and a position moved by two dots downward. As a result, the number of dots in all the frames is “2” at the position moved by one dot to the right and moved by two dots downward. Therefore, it can be seen that the position of the block frame at this time is the pattern block delimiter position.

FIG. 15 shows the pattern block delimiter positions determined by the block detector 23.
Here, when a block frame is overlapped at a position indicated by a bold line, two “1” s are included in all the blocks, and this position is a pattern block separation position. The block detection may be started from any position, but here again, the upper left of the dot array is the starting point, and the amount of movement of the block frame from the starting point until the pattern block delimiter is found is determined in the X direction. MX dots and MY dots in the Y direction. In the figure, MX = 0 and MY = 1. Hereinafter, a 5 block × 5 block area (a part within a bold line frame in the figure) in which block frames overlap when a block delimiter position is found is referred to as a “decoding area”.

Further, the block detection unit 23 obtains a pattern value from the dot position (arrangement of “0” and “1”) in each pattern block. At this time, the correspondence between the dot position and the pattern value shown in FIG. 2 or 3 is referred to.
FIG. 16 is a diagram illustrating an example when pattern values are assigned to each pattern block.
For example, since the upper left block indicates that there are dots at the upper left and lower right in the block, the pattern value is “0” from the correspondence in FIG. In the right block, since it is shown that there are dots at the upper left and right center in the block, the pattern value is “12” from the correspondence in FIG.

Here, the detailed operation of the block detection unit 23 will be described.
FIG. 17 is a flowchart illustrating an operation example of the block detection unit 23.
First, the block detector 23 acquires a dot array from the dot array generator 22 (step 201). The size of this dot array is (number of blocks necessary for decoding × number of dots on one side of block + number of dots on one side of block−1) ² . In this embodiment, the number of blocks necessary for decoding is 5 × 5, and the number of dots on one side of the block is 3, so a 17 × 17 dot array is acquired.

  Next, a block frame is superimposed on the acquired dot array (step 202). Then, “0” is substituted into the counters I and J, and “0” is substituted into MaxBN (step 203). Here, I and J count the number of steps in which the block frame is moved from the initial position. The block frame is moved for each line of the image, and the number of lines moved at that time is counted by counters I and J. MaxBN records the maximum count value when counting the number of blocks in which the number of dots detected in the block is “2” while moving the block frame.

  Next, the block detector 23 moves the block frame I in the X direction and J in the Y direction (step 204). Since I and J are “0” in the initial state, the block frame does not move. Then, the number of dots included in each block of the block frame is counted, and the number of blocks in which the number of dots is “2” is counted. The counted number of blocks is stored in IB [I, J] (step 205). In I and J of IB [I, J], values of I and J indicating the movement amount of the block frame are recorded, respectively.

  Next, the block detection unit 23 compares IB [I, J] with MaxBN (step 206). Since MaxBN has an initial value “0”, in the first comparison, IB [I, J] is larger than MaxBN. In this case, the value of IB [I, J] is substituted for MaxBN, the value of I is substituted for MX, and the value of J is substituted for MY (step 207). When IB [I, J] is MaxBN or less, the values of MaxBN, MX, and MY are left as they are.

Thereafter, the block detection unit 23 determines whether I = 2 (step 208).
If I = 2 is not satisfied, “1” is added to I (step 209). Then, the processing in steps 204 and 205 is repeated to compare IB [I, J] with MaxBN (step 206).
If IB [I, J] is greater than MaxBN, which is the maximum value of IB [I, J] up to the previous time, the value of IB [I, J] is assigned to MaxBN, and the value of I at that time is set to MX. , J is substituted for MY (step 207). If MaxBN is larger than IB [I, J], it is determined whether I = 2 (step 208). When I = 2, it is next determined whether J = 2 or not (step 210). If J = 2 is not satisfied, “0” is substituted for I, and “1” is added to J (step 211). Such a procedure is repeated to detect a case where (I, J) is (0, 0) to (2, 2) and IB [I, J] is maximum.

When the processing up to I = 2 and J = 2 is completed, the block detection unit 23 compares the stored MaxBN with the threshold value TB (step 212). The threshold value TB is a threshold value for determining whether the number of blocks having a dot number of “2” is such that decoding is possible.
Here, when MaxBN is larger than the threshold value TB, the block frame is fixed at the positions of MX and MY, and the pattern value of each block is detected at that position. Then, the detected pattern value is recorded in the memory as a code array PA [X, Y] together with variables X and Y for identifying each block (step 213). At this time, if the pattern value cannot be converted, “−1” which is not used as a pattern value is recorded. Then, the block detection unit 23 outputs MX and MY and the code array PA [X, Y] to the synchronous code detection unit 24 (step 214).
On the other hand, if MaxBN is equal to or less than the threshold value TB, it is determined that decoding is impossible due to large noise in the image, and decoding is impossible (step 215).

Next, the synchronization code detection unit 24 searches for the pattern value of the synchronization pattern from the pattern values detected by the block detection unit 23. For example, in the present embodiment, the code pattern of the pattern values 32 to 35 is used as the synchronization pattern, so this number is searched. In the example of FIG. 16, the pattern value “35” is the synchronization pattern.
Referring to FIG. 4, the synchronization pattern having the pattern value “35” is a synchronization pattern obtained by rotating the synchronization pattern having the pattern value “32”, which is an upright synchronization pattern, to the right by 270 degrees. Therefore, in order to make the code array upright, the code array needs to be rotated 270 degrees to the left. In that case, it is necessary to change the pattern value of each element in the code array to the pattern value when rotated 270 degrees to the left.

  FIG. 18 shows a pattern value of a code pattern at an upright position, a pattern value of a code pattern obtained by rotating the code pattern at an upright position 90 degrees to the right, and a code pattern obtained by rotating the code pattern at an upright position 180 degrees to the right. Is a correspondence table between the pattern value of the erect position and the pattern value of the code pattern obtained by rotating the code pattern at the upright position 270 degrees to the right. Such a correspondence table may be stored in a memory (not shown), and the synchronization code detection unit 24 may refer to this correspondence table and change pattern values.

FIG. 19A specifically shows the pattern value change at this time. For example, a block with a pattern value “0” in the upper left turns into a block with a pattern value “1” in the upper right by rotating 270 degrees to the left. Further, the block with the pattern value “12” on the right is rotated to the left by 270 degrees to become the second block with the pattern value “13” from the top in the rightmost column.
When the code array is rotated, the values used for conversion from the decoding area to the reading area are also different. In other words, if the number of pixels from the upper left point of the decoding area to the upper left point of the reading area is RDX in the X-axis direction on the code coordinates and RDY in the Y-axis direction on the code image, it is necessary to use different values as these values. There is. Therefore, as shown in FIG. 19B, the synchronization code detection unit 24 stores the values of RDX and RDY in a memory (not shown) in addition to the rotation angle for each synchronization pattern.

Here, the detailed operation of the synchronization code detector 24 will be described.
FIG. 20 is a flowchart illustrating an operation example of the synchronization code detection unit 24.
First, the synchronization code detection unit 24 acquires MX and MY and the code array PA [X, Y] from the block detection unit 23 (step 251).
Next, the synchronous code detection unit 24 substitutes “1” for K and L (step 252). K is a counter indicating the number of blocks in the X direction, and L is a counter indicating the number of blocks in the Y direction.

Next, the synchronization code detector 24 determines whether the pattern value of PA [K, L] is “32” (step 253).
If the pattern value of PA [K, L] is “32”, it is determined that the code array PA [X, Y] does not need to be rotated, and K is substituted for the X coordinate SX of the block having the synchronization code, and Y Substitute L for coordinates SY. Further, MX is substituted for the movement amount ShiftX in the X direction of the block frame, and MY is substituted for the movement amount ShiftY in the Y direction (step 254).

Next, the synchronization code detector 24 determines whether the pattern value of PA [K, L] is “33” (step 255).
If the pattern value of PA [K, L] is “33”, the code array PA [X, Y] is rotated 90 degrees to the left (step 256). As shown in FIG. 4A, the code pattern of the pattern value “33” is an image obtained by rotating the code pattern of the pattern value “32” by 90 degrees in the right direction. Is upright. At this time, all pattern values in the code array PA [X, Y] are converted into pattern values when rotated 90 degrees to the left.
In accordance with this rotation, L is substituted for the X coordinate SX of the block having the synchronization code, and 6-K is substituted for the Y coordinate SY. Also, MY is substituted for the movement amount ShiftX in the X direction of the block frame, and 2-MX is substituted for the movement amount ShiftY in the Y direction (step 257).

Next, the synchronization code detection unit 24 determines whether or not the pattern value of PA [K, L] is “34” (step 258).
If the pattern value of PA [K, L] is “34”, the code array PA [X, Y] is rotated 180 degrees to the left (step 259). As shown in FIG. 4A, since the code pattern of the pattern value “34” is an image obtained by rotating the code pattern of the pattern value “32” by 180 degrees, the code pattern of the pattern value “34” is rotated by 180 degrees. Let the image stand upright. At this time, all the pattern values in the code array PA [X, Y] are converted into pattern values when rotated 180 degrees.
In accordance with this rotation, 6-K is substituted for the X coordinate SX of the block having the synchronization code, and 6-L is substituted for the Y coordinate SY. Further, 2-MX is substituted for the movement amount ShiftX in the X direction of the block frame, and 2-MY is substituted for the movement amount ShiftY in the Y direction (step 260).

Next, the synchronization code detection unit 24 determines whether the pattern value of PA [K, L] is “35” (step 261).
If the pattern value of PA [K, L] is “35”, the code array PA [X, Y] is rotated 270 degrees to the left (step 262). As shown in FIG. 4A, the code pattern with the pattern value “35” is an image obtained by rotating the code pattern with the pattern value “32” to the right by 270 degrees, so that the image is rotated by 270 degrees in the reverse direction. Erect. At this time, all the pattern values in the code array PA [X, Y] are converted into pattern values when rotated 270 degrees in the left direction.
In accordance with this rotation, 6-L is substituted for the X coordinate SX of the block having the synchronization code, and K is substituted for the Y coordinate SY. Further, 2-MY is substituted for the movement amount ShiftX in the X direction of the block frame, and MX is substituted for the movement amount ShiftY in the Y direction (step 263).

When values are assigned to SX, SY, ShiftX, and ShiftY in steps 254, 257, 260, and 263, the synchronization code detection unit 24 converts PA [X, Y] and these values into the identification code detection unit 30. The X coordinate code detection unit 40 and the Y coordinate code detection unit 45 are output (step 264).
If PA [K, L] is not any of the pattern values “32” to “35”, the synchronization code detector 24 determines whether K = 5 (step 265). If K = 5 is not satisfied, “1” is added to K (step 266), and the process returns to step 253. If K = 5, it is determined whether L = 5 (step 267). If L = 5 is not satisfied, “1” is substituted for K, “1” is added to L (step 268), and the process returns to step 253. That is, the processes in steps 253 to 264 are repeated while changing the values of K and L until the blocks having pattern values “32” to “35” are detected. In addition, even if K = 5 and L = 5, if a block having pattern values “32” to “35” cannot be detected, a determination signal indicating that decoding is impossible is output (step 269).

Thereafter, the identification code detection unit 30 and the identification code decoding unit 32 acquire identification information for identifying the paper surface or the image on the paper surface from the reading area.
FIG. 21 is a flowchart showing an operation example of the identification code detection unit 30 and the identification code decoding unit 32.
First, the identification code detection unit 30 acquires the code arrays PA [X, Y], SX, and SY from the synchronization code detection unit 24 (step 301).
Next, the identification code detection unit 30 initializes all elements of the identification code array IA [X, Y] with “−1” (step 302). Note that “−1” is a number that is not used as a pattern value. Then, “1” is substituted into counters IX and IY for identifying each block in the code block (step 303). Here, IX is a counter indicating the number of blocks in the X direction, and IY is a counter indicating the number of blocks in the Y direction.

The identification code detection unit 30 determines whether IY-SY is divisible by “5” (step 304). That is, it is determined whether or not the synchronization code is arranged in the row specified by IY.
Here, when IY-SY is divisible by “5”, that is, when a synchronization code is arranged in this row, since the identification code is not extracted, “1” is added to IY (step 305). ), Go to step 304.
On the other hand, if IY-SY is not divisible by “5”, that is, if no synchronization code is arranged in this row, it is determined whether IX-SX is divisible by “5” (step 306). That is, it is determined whether or not the synchronization code is arranged in the column specified by IX.

Here, when IX-SX is divisible by “5”, that is, when a synchronization code is arranged in this column, “1” is added to IX because the identification code is not extracted (step 307). ), Go to Step 306.
On the other hand, when IX-SX is not divisible by “5”, that is, when a synchronization code is not arranged in this column, the identification code detection unit 30 performs IA [(IX-SX) mod5, (IY-SY) mod [5] is substituted for PA [IX, IY] (step 308).

Then, it is determined whether or not IX = 5 (step 309).
If IX = 5 is not satisfied, “1” is added to IX (step 307), and the processing of steps 306 to 308 is repeated until IX = 5. When IX = 5, it is next determined whether IY = 5 (step 310). If IY = 5 is not satisfied, “1” is substituted for IX (step 311), “1” is added to IY (step 305), and the processing of steps 304 to 309 is repeated until IY = 5. . When IY = 5, the process proceeds to the identification code decoding unit 32.

That is, the identification code decoding unit 32 determines whether or not IA [X, Y] can be decoded (step 312).
If it is determined that IA [X, Y] can be decoded, the identification code decoding unit 32 obtains identification information from IA [X, Y] (step 313). If it is determined that IA [X, Y] cannot be decoded, N / A is substituted for identification information (step 314).

In addition, the X coordinate code detection unit 40 and the X coordinate code decoding unit 42 acquire the position of the reading area in the X direction.
FIG. 22 is a flowchart illustrating an operation example of the X coordinate code detection unit 40 and the X coordinate code decoding unit 42.
First, the X coordinate code detection unit 40 acquires the code arrays PA [X, Y], SX, SY, ShiftX, and ShiftY from the synchronization code detection unit 24 (step 401).
Next, the X coordinate code detection unit 40 initializes all elements of the X coordinate code array XA [X] with “−1” (step 402). Note that “−1” is a number that is not used as a pattern value. Then, “1” is substituted into counters IX and IY for identifying each block in the code block. Here, IX is a counter indicating the number of blocks in the X direction, and IY is a counter indicating the number of blocks in the Y direction. Furthermore, the X coordinate code detection unit 40 substitutes “1” into a counter KX for identifying each element in the X coordinate code array (step 403).

Further, the X coordinate code detection unit 40 determines whether IY-SY is divisible by “5” (step 404). That is, it is determined whether or not the synchronization code is arranged in the row specified by IY.
Here, if IY-SY is not divisible by “5”, that is, if no synchronous code is arranged in this row, the X coordinate code is not extracted, so “1” is added to IY ( Step 405), the process proceeds to Step 404.
On the other hand, when IY-SY is divisible by “5”, that is, when a synchronous code is arranged in this row, the X coordinate code detection unit 40 determines whether IX-SX is divisible by “5”. (Step 406). That is, it is determined whether or not the synchronization code is arranged in the column specified by IX.

Here, when IX-SX is divisible by “5”, that is, when the synchronization code is arranged in this column, “1” is added to IX because the X coordinate code is not extracted (step 1). 407), go to step 406.
On the other hand, if IX-SX is not divisible by “5”, that is, if no synchronization code is arranged in this column, the X coordinate code detection unit 40 substitutes PA [IX, IY] for XA [KX]. (Step 408).

Then, it is determined whether or not IX = 5 (step 409).
If IX = 5 is not satisfied, “1” is added to KX (step 410), “1” is added to IX (step 407), and the processing of steps 406 to 408 becomes IX = 5. Repeat until When IX = 5, the process proceeds to the process of the X coordinate code decoding unit 42.

That is, the X coordinate code decoding unit 42 determines whether or not XA [X] can be decoded (step 411).
If it is determined that XA [X] can be decoded, the X coordinate code decoding unit 42 decodes X coordinate information from XA [X] and ShiftX (step 412). If it is determined that XA [X] cannot be decoded, N / A is substituted into the X coordinate information (step 413).

  Although only the operations of the X coordinate code detection unit 40 and the X coordinate code decoding unit 42 have been described here, the Y coordinate code detection unit 45 and the Y coordinate code decoding unit 47 perform the same operation.

Finally, the coordinate correction unit 50 obtains the coordinates of the nib center on the paper surface from the X coordinate information acquired by the X coordinate code decoding unit 42 and the Y coordinate information acquired by the Y coordinate code decoding unit 47.
FIG. 23 is a flowchart showing an operation example of the coordinate correction unit 50.
First, the coordinate correction unit 50 takes the relative coordinates (P _X , P _Y ) of the center of the pen tip with respect to the center of the reading area on the captured image, the inclination angle (θ _X , θ _Y ) of the reading area with respect to the imaging area, and the imaging. The dot spacing (d _X , d _Y ) on the image is acquired (step 501). Here, (P _X , P _Y ), (θ _X , θ _Y ), and (d _X , d _Y ) are stored in the memory when the dot array generation unit 22 detects the dot, read out. Then, the relative coordinates (Q _X , Q _Y ) of the nib center with respect to the reading area center on the code image are calculated (step 502). Here, (Q _X , Q _Y ) is calculated by the following equation as described above.

Next, the coordinate correction unit 50 acquires the number of pixels (RDX, RDY) of the movement amount of the block frame (step 503). Here, (RDX, RDY) is stored in the memory when the synchronous code detection unit 24 detects the rotation of the code image, so that it is read out.
Also, the coordinate correction unit 50 acquires the position BX of the leftmost pattern block of the code block from the X coordinate code decoding unit 42 and acquires the position BY of the pattern block at the upper end of the code block from the Y coordinate code decoding unit 47. (Step 504).

Thereafter, the coordinate correction unit 50 obtains the pixel position (DSX, DSY) of the decoding area start point (the upper left point of the decoding area) (step 505). The leftmost pixel position in the leftmost pattern block of the code block is DSX, and the uppermost pixel position in the uppermost pattern block of the code block is DSY. And since one block is 12 pixels, (DSX, DSY) is
DSX = (BX−1) × 12 + 1
DSY = (BY-1) × 12 + 1
Is required.

By the process so far, the upper left pixel position of the decoding area is obtained, but as described above, since the decoding area and the reading area are not the same position, in order to obtain the position of the reading area center, correction is performed. is required. Therefore, the coordinate correction unit 50 corrects the pixel position (DSX, DSY) of the decoding area start point to the pixel position (RSX, RSY) of the reading area start point (upper left point of the reading area) (step 506). In this correction, (RDX, RDY) read from the memory in step 503 is used. That is, the upper left coordinates (RSX, RSY) of the reading area are
RSX = DSX + RDX
RSY = DSY + RDY
Is required.

Further, the coordinate correction unit 50 obtains the pixel position (CRX, CRY) at the center of the reading area (step 507). Here, the width and height of the reading area are predetermined design values. Therefore, if these are DAW and DAH, respectively, the reading area center (CRX, CRY) is
CRX = RSX + DAW / 2
CRY = RSY + DAH / 2
Is required.

Finally, the pixel position (PPX, PPY) at the nib center is obtained (step 508). In step 502, the relative coordinates (Q _X , Q _Y ) of the nib center with respect to the reading area center are obtained, and this is added to the pixel position of the reading area center obtained in step 507. That is, (PPX, PPY) is
PPX = CRX + Q _X
PPY = CRY + Q _Y
Is required.
This is the end of the description of the operation of the present embodiment.

In the above description, the relative coordinates (P _X , P _Y ) of the pen tip center with respect to the reading area center on the captured image, the inclination angle (θ _X , θ _Y ) of the reading area with respect to the imaging area, The coordinates of the center of the pen tip on the paper surface were obtained using the dot spacing (d _X , d _Y ). However, not all of these parameters must be used. For example, when the reading area is guaranteed to be rectangular, the inclination angle θ in one direction may be used instead of the inclination angles (θ _X , θ _Y ) in the two directions. Instead of the dot interval (d _X , d _Y ), the dot interval d in one direction may be used. In addition, when it is ensured that there is no difference in angle between the imaging area and the reading area, the inclination angles (θ _X , θ _Y ) may not be used. Furthermore, when it is guaranteed that there is no difference in scale between the captured image and the code image, the dot spacing (d _X , d _Y ) may not be used.

Next, a specific hardware configuration of the image processing apparatus 20 in the present embodiment will be described.
First, the pen device 60 that implements the image processing apparatus 20 will be described.
FIG. 24 is a view showing the mechanism of the pen device 60.
As illustrated, the pen device 60 includes a control circuit 61 that controls the operation of the entire pen. In addition, the control circuit 61 includes an image processing unit 61a that processes a code image detected from an input image, and a data processing unit 61b that extracts identification information and position information from the processing result.
The control circuit 61 is connected to a pressure sensor 62 that detects the writing operation by the pen device 60 by the pressure applied to the pen tip 69. Further, an infrared LED 63 that irradiates infrared light onto the medium and an infrared CMOS 64 that inputs an image are also connected. Further, an information memory 65 for storing identification information and position information, a communication circuit 66 for communicating with an external device, a battery 67 for driving the pen, and pen identification information (pen ID) are stored. A pen ID memory 68 is also connected.

  The image reading unit 21 shown in FIG. 11 is realized by, for example, the infrared CMOS 64 in FIG. The dot array generation unit 22 is realized by, for example, the image processing unit 61a in FIG. Furthermore, the block detection unit 23, the synchronization code detection unit 24, the identification code detection unit 30, the identification code decoding unit 32, the X coordinate code detection unit 40, the X coordinate code decoding unit 42, and the Y coordinate code detection unit 45 shown in FIG. The Y coordinate code decoding unit 47, the coordinate correction unit 50, and the information output unit 55 are realized by, for example, the data processing unit 61b in FIG.

  In the present embodiment, the pen tip 69 of the pen device 60 is provided as an example of the instruction unit that indicates the position on the medium. However, the present invention is not limited to this. Instead of writing on the medium, a structure for simply pointing to a predetermined position on the medium may be regarded as the instruction means.

The processing realized by the image processing unit 61a or the data processing unit 61b in FIG. 24 may be realized by a general-purpose computer, for example. Accordingly, assuming that such processing is realized by the computer 90, the hardware configuration of the computer 90 will be described.
FIG. 25 is a diagram illustrating a hardware configuration of the computer 90.
As shown in the figure, the computer 90 includes a CPU (Central Processing Unit) 91 as a calculation means, a main memory 92 as a storage means, and a magnetic disk device (HDD: Hard Disk Drive) 93. Here, the CPU 91 executes various types of software such as an OS (Operating System) and applications to realize the above-described functions. The main memory 92 is a storage area for storing various software and data used for execution thereof, and the magnetic disk device 93 is a storage area for storing input data for various software, output data from various software, and the like. .
Further, the computer 90 includes a communication I / F 94 for performing communication with the outside, a display mechanism 95 including a video memory and a display, and an input device 96 such as a keyboard and a mouse.

  The program for realizing the present embodiment can be provided not only by communication means but also by storing it in a recording medium such as a CD-ROM.

It is the figure which showed an example of the code pattern in 9C2 system and 9C3 system. It is the figure which showed an example of all the code patterns in 9C2 system. It is the figure which showed an example of all the code patterns in 9C3 system. It is the figure which showed the example of the synchronous pattern in 9C2 system and 9C3 system. It is the figure which showed the example of the basic layout of a code block. It is a figure for demonstrating the expression of the coordinate by M series. It is the figure which showed the example of the extensive layout of a code block. It is a figure for demonstrating the shift | offset | difference of a nib center and an imaging area center. It is the figure which showed positions, such as a nib center and a reading area center, on a captured image. It is a figure for demonstrating the coordinate transformation from the coordinate system on a captured image to the coordinate system on a code | cord | chord image. It is the block diagram which showed the function structure of the image processing apparatus in this Embodiment. It is a figure for demonstrating calculation of the inclination angle of a row | line | column of a dot. It is a figure for demonstrating the process at the time of producing | generating a dot array. It is a figure for demonstrating the process at the time of detecting a block on a dot arrangement | sequence. It is a figure for demonstrating the state after detecting a block on a dot arrangement | sequence. It is a figure for demonstrating the process at the time of converting from a dot arrangement to a code arrangement. It is the flowchart which showed the operation example of the block detection part in this Embodiment. It is the figure which showed the correspondence of a rotation angle and a pattern number. It is a figure for demonstrating the memory | storage of the information accompanying rotation of a code arrangement | sequence and rotation. It is the flowchart which showed the operation example of the synchronous code detection part in this Embodiment. It is the flowchart which showed the operation example of the identification code | cord | chord detection part etc. in this Embodiment. It is the flowchart which showed the operation example of the X coordinate code | symbol detection part etc. in this Embodiment. It is the flowchart which showed the operation example of the coordinate correction | amendment part in this Embodiment. It is the figure which showed the mechanism of the pen device which can implement | achieve the image processing apparatus in this Embodiment. It is a hardware block diagram of the computer which can apply this Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 20 ... Image processing apparatus, 21 ... Image reading part, 22 ... Dot array production | generation part, 23 ... Block detection part, 24 ... Synchronization code detection part, 30 ... Identification code detection part, 32 ... Identification code decoding part, 40 ... X coordinate Code detection unit, 42 ... X coordinate code decoding unit, 45 ... Y coordinate code detection unit, 47 ... Y coordinate code decoding unit, 50 ... Coordinate correction unit, 55 ... Information output unit

Claims (9)

Indicating means for indicating a position on the medium;
Ax (Ax ≧ 2) pattern images in the horizontal direction and Ay (Ay in the vertical direction) are printed on the medium, and unit images are arranged at n (1 ≦ n <m) of m (m ≧ 3). Ay ≧ 2) an image acquisition means for acquiring a captured image obtained by capturing a print image in which a group of pattern images arranged periodically is arranged ;
Extracting means for extracting a partial image constituting a portion of the captured image from the captured image;
First array generation means for generating a first array for storing the presence or absence of the unit image in the partial image as an element;
First information acquisition means for acquiring first position information indicating a relative position with respect to the partial image at a position corresponding to a position on the medium instructed by the instruction means on a plane including the captured image;
On the first array, a frame including n pieces of unit images having a frame formed by arranging Ax pieces of the same size as the pattern image in the horizontal direction and Ay pieces in the vertical direction is predetermined. By moving to a frame position that is greater than or equal to the number, a second array that is an array within the frame at the frame position and the amount of movement of the frame in the horizontal direction and the vertical direction up to the frame position are generated. Second array generating means for
Second information acquisition means for acquiring second position information indicating a position of a portion corresponding to the second array of the print image on the medium using the second array ;
Conversion means for converting the first position information into third position information indicating a relative position of the position on the medium instructed by the instruction means with respect to the print image;
First specifying means for specifying a position of a portion corresponding to the partial image of the print image on the medium based on the second position information and the movement amount;
A second specifying unit that specifies a position on the medium instructed by the instruction unit based on a position of a portion corresponding to the partial image of the print image and the third position information; A position detecting device characterized by that. It said first information acquisition means further acquires angular information indicating an inclination angle with respect to the captured image of the partial image,
The position detection apparatus according to claim 1, wherein the conversion unit converts the first position information into the third position information using the angle information. The partial image is a quadrilateral,
The first information acquisition unit includes first angle information indicating an inclination angle of the first side of the partial image with respect to the captured image, and a second side adjacent to the first side of the partial image. The position detection apparatus according to claim 2, wherein second angle information indicating an inclination angle with respect to the captured image is acquired as the angle information. The first information acquisition means further acquires scale information indicating a scale in the captured image,
The position detecting apparatus according to claim 1, wherein the converting unit converts the first position information into the third position information using the scale information. The partial image is a quadrilateral,
The first information acquisition means includes first scale information indicating a scale in the captured image along the first side of the partial image, and a second side adjacent to the first side of the partial image. 5. The position detection apparatus according to claim 4, wherein second scale information indicating a scale in the captured image along the line is acquired as the scale information. The position detecting device according to claim 4, wherein the first information acquisition unit acquires the scale information based on an interval between images corresponding to the unit images on the captured image. On the computer,
Ax (Ax ≧ 2) pattern images printed in the medium and having unit images arranged at n (1 ≦ n <m) of m (m ≧ 3) locations in the horizontal direction and Ay (Ay in the vertical direction) ≧ 2) a function of acquiring a captured image obtained by capturing a print image in which a group of pattern images arranged periodically is arranged ;
A function of extracting a partial image constituting a portion of the captured image from the captured image;
A function of generating a first array that stores the presence or absence of the unit image in the partial image as an element;
A function of acquiring first position information indicating a relative position of the position corresponding to the position on the medium instructed by the instruction unit with respect to the partial image on a plane including the captured image;
On the first array, a frame including n pieces of unit images having a frame formed by arranging Ax pieces of the same size as the pattern image in the horizontal direction and Ay pieces in the vertical direction is predetermined. By moving to a frame position that is greater than or equal to the number, a second array that is an array within the frame at the frame position and the amount of movement of the frame in the horizontal direction and the vertical direction up to the frame position are generated. Function to
A function of acquiring second position information indicating a position of a portion corresponding to the second array of the print image on the medium using the second array ;
A function of converting the first position information into third position information indicating a relative position of the position on the medium instructed by the instruction unit with respect to the print image;
A function of specifying a position of a portion corresponding to the partial image of the print image on the medium based on the second position information and the movement amount;
A program for realizing a function of specifying a position on the medium instructed by the instruction unit based on a position of a part corresponding to the partial image of the print image and the third position information. Further to realize said computer a function of acquiring angular information indicating an inclination angle with respect to the captured image of the partial image,
8. The program according to claim 7, wherein the converting function converts the first position information into the third position information using the angle information. The computer further realizes a function of acquiring scale information indicating a scale in the captured image,
8. The program according to claim 7, wherein the converting function converts the first position information into the third position information by using the scale information.

Images