JP6857867B2 - Handwriting input sheet - Google Patents

Description

The present invention relates to a handwritten input / output system and a handwritten input sheet for inputting handwritten information such as characters and figures into a computer system.

The present invention relates to an information input system and an information input assist sheet for assisting input to a computer system.

Computers are becoming used in every aspect of our lives. In addition to conventional applications such as creating documents and performing calculations, computer functions and applications such as drawing pictures and drawing buildings and machines are increasing dramatically.

Conventionally, in performing these actions, it has been required to perform operations such as character input and drawing using a keyboard, mouse, tablet, CAD system, or the like.

However, there are many complicated operations for inputting characters with a keyboard and drawing with a tablet or CAD system, and it takes time to learn. In particular, there is a problem that it is difficult for a person who has difficulty in operating the device, such as an elderly person. Furthermore, the "digital divide", which creates a disparity in information and opportunities between those who can operate the keyboard and mouse and can master the computer, and those who cannot operate the computer and cannot master the computer, has become a problem. ing.

As a solution to such a problem, a system for inputting handwritten information into a computer has been proposed. Such a system is composed of a base and a drawing device, and when the drawing device is moved with respect to the base, graphical information (handwritten information) is generated. The base is provided with a position coding pattern which is a fine dot, and position information on the base is registered in this position coding pattern. When drawing on the base using a drawing device, the drawing device reads the position coding pattern. The computer system records the position information on the base from the drawing device as a graphical input. This makes it possible to input the drawn contents to the computer system, output them to the display, or save them in the memory (for example, Patent Document 1).

Special Table 2003-528387

However, as the position coding pattern used in the system of Patent Document 1, only position information is registered. Therefore, the computer system can only input the graphical information drawn by the drawing device, and for drawing, for example, in order to input and record various information related to the graphical information such as the page number of the booklet. An area different from the area of No. 1 must be provided separately. Therefore, the handwriting input / output system becomes complicated, and the user also has a problem that the operation in a plurality of areas is required. In addition, a reference table for inputting and recording various information related to the graphical information must be provided for all the position information corresponding to the graphical information drawn by the drawing device. In particular, when the drawing area is not a rectangle but an area surrounded by polygons or curves, a reference table for all the position information in the area is required.

The present invention has been made in view of such a problem, and it is a technical subject to provide a handwritten input / output system capable of inputting characters, figures and the like by handwriting easily and inexpensively. Another technical issue is to provide a handwritten input / output system capable of collectively inputting not only character and graphic information but also information related to them.

<Handwriting input / output system>
The handwritten information processing system according to the present invention includes a medium having a writing area in which coordinate information and a dot pattern in which code information is repeatedly defined are formed in at least a part or a plurality of areas, and the dots in the writing area. By storing the image data of the dot pattern imaged and transmitted by the imaging means and the imaging means that images and transmits the pattern, and code-analyzing the image data, the coordinate information and the code information are used. It is recognized as a dot pattern analysis means for obtaining the locus information, and a locus recognition means for recognizing the locus information of characters and / or figures traced on the writing area surface based on the analyzed change of the coordinate information. It is characterized by including a processing command based on the recognition information and a processing command means for transmitting the trajectory information to the information processing means.

According to the above features, in the handwritten input / output system according to the present invention, the dot pattern formed on the medium defines not only the coordinate information but also the code information, so that the processing instruction means informs the information processing unit of the locus information. It is possible to collectively transmit processing instructions based on the locus information.

It is preferable that the medium is further formed with text, graphics, and photographs related to the writing area in at least a part or a plurality of areas.

According to the above characteristics, the handwritten input / output system according to the present invention can easily know the position of the writing area and what kind of matter should be written in the writing area.

The medium is formed by newly superimposing the dot pattern together with at least the locus information drawn on the writing area surface, the editing result information of the locus information, and / or the printing information on the medium surface. Is preferable.

According to the above features, the handwritten input / output system according to the present invention can superimpose print a medium once handwritten and input together with a new dot pattern, and further write / edit on the medium.

The medium is a specific information and / or operation instruction (program start, command input, data input / output / search, search target) that specifies the write area with respect to a predetermined medium surface and / or the write area surface. It is preferable that the dot pattern that repeatedly forms the code information that defines (such as the designation of) is formed by superimposing an icon that means the code information.

According to the above characteristics, in the handwritten input / output system according to the present invention, the user can easily see what the code information means (what processing instruction is transmitted) by looking at the icon. You can know.

The icon is preferably formed on a stickable sticker, a sticky note, or other medium surface so as to be superimposed on the dot pattern.

According to the above features, in the handwriting input / output system according to the present invention, the icons can be freely laid out on the medium surface, so that the degree of freedom in layout is increased.

The medium forms a dot pattern in which the medium provided with the writing area is used as the first medium and code information for specifying the first medium surface and / or the writing area surface is repeatedly defined. By reading the code information formed on the second medium with the imaging means, the print information on the first medium surface and / or the locus information drawn on the writing area surface is provided. , And / or it is preferable to output information related to the locus information.

According to the above features, the handwritten input / output system according to the present invention prints code information for identifying a medium surface on another medium (such as a notebook) and touches it with an imaging means to input handwritten information on the medium surface. You can search and output the information you have created. Moreover, the space of the writing area of the medium can be widened.

The second medium preferably has a stickable sticker, a sticky note, or the like, or a card-like form.

According to the above features, the handwritten input / output system according to the present invention further increases the degree of freedom in the layout of the writing area, and searches and outputs the information handwritten and input in the writing area according to the convenience and preference of the user. It can be executed freely.

The medium is a transparent medium in which a dot pattern in which coordinate information and / or code information is repeatedly defined is formed, that is, a grid, which is used by the handwriting input / output system so as to be superimposed on a monitor surface on which the writing area is displayed. It is preferably a sheet.

According to the above features, the handwriting input / output system according to the present invention can provide a touch panel with good maneuverability.

The medium is a transparent medium in which a dot pattern in which coordinate information and / or code information is repeatedly defined is formed, that is, a grid sheet, which is used by superimposing the writing area by the handwriting input / output system on a printed medium. It is preferable to have.

According to the above features, the handwriting input / output system according to the present invention can perform handwriting input by covering the medium surface with the grid sheet without overlapping the grid sheet on the monitor surface.

The image pickup means includes an irradiation means for irradiating the medium surface with light having a predetermined wavelength through an image pickup port, an image pickup device for capturing the reflected light of the predetermined wavelength, and transmitting the reflected light as image data to the dot pattern analysis means. The dot pattern analysis means causes the irradiation means to emit light of a predetermined intensity at a first predetermined time, and the dot pattern analysis means captures the image data by the image pickup device and stores the transmitted image data. Further, the light of the predetermined intensity is used for the analysis time or longer required for the dot pattern analysis means to code-analyze the image data so that the image data can be fixed to the image pickup device as an afterimage. It is preferable that the light control means for controlling the irradiation means is further provided so that the light is emitted again after the elapse of the predetermined time of 2 and the first predetermined time and the second predetermined time are continuous.

According to the above-mentioned features, the handwriting input / output system according to the present invention can print a dot pattern on an image pickup device without blurring with respect to the movement of the image pickup means on the medium surface when writing. By analyzing the afterimage, handwriting input can be performed even if the frame rate (imaging speed) of the imaging means is low.

After the code analysis of the image data by the dot pattern analysis means is completed, the light control means immediately instructs the irradiation means to emit light without waiting for the second predetermined time, and the irradiation means is the first. It is preferable that the dot pattern analysis means stores the image data newly captured and transmitted by the imaging means and performs code analysis by emitting the light of the predetermined intensity for the predetermined time.

According to the above features, the handwritten input / output system according to the present invention immediately determines the next code when the code analysis is completed earlier than the second predetermined time due to an ideal dot pattern imaging / analysis or an error. By moving to the analysis, the imaging interval is shortened and smooth trajectory information can be acquired.

The optical control means immediately instructs the irradiation means to emit light without waiting for the second predetermined time during the analysis time of the dot pattern by the dot pattern analysis means, and the irradiation means is the first. The dot pattern analysis means emits light of the predetermined intensity for a predetermined time, and during the analysis time, the dot pattern analysis means stores image data newly captured and transmitted by the imaging means, and the analysis time is changed. It is preferable to perform code analysis of the image data immediately after completion.

According to the above features, in the handwritten input / output system according to the present invention, the dot pattern analysis means stores the next image data during the code analysis and immediately moves to the code analysis of the next image data after the code analysis is completed. As a result, the imaging interval is further shortened, and smooth trajectory information can be acquired.

The imaging means further includes an instruction protrusion, and when tracing a character and / or a figure traced by the instruction protrusion on the writing area surface, the dot pattern on a locus in the vicinity of the instruction protrusion The dot pattern analysis means stores the image data of the dot pattern transmitted by the imaging means, code-analyzes the dot pattern, and obtains the coordinate information and the locus information including the code information. Is preferable.

According to the above features, the handwriting input / output system according to the present invention makes it easy to visually recognize the position traced by the imaging means on the writing area surface.

The imaging means further includes at least one pen tip, and when tracing a character and / or a figure drawn by the pen tip on the writing area surface, the imaging means captures the dot pattern on a locus in the vicinity of the pen tip. The dot pattern analysis means stores the image data of the dot pattern imaged and transmitted by the imaging means, code-analyzes the dot pattern, and obtains locus information including the coordinate information and the code information. It is preferable to obtain it.

According to the above features, the handwriting input / output system according to the present invention can leave a locus in the vicinity of the imaging means on the writing area surface.

The instruction protrusion is further provided with at least one pen tip, a storage space for storing the pen tip in the instruction protrusion, and a button for controlling storage / removal of the pen tip. It is preferably a protrusion.

According to the above features, the handwritten input / output system according to the present invention can be used properly depending on whether it is desired to leave a locus in the vicinity of the imaging means drawn on the writing area surface or not.

The imaging means further includes an erasing means for erasing characters and / or figures drawn on the writing area surface on the instruction protrusion, and at the same time erasing the area traced by the erasing means on the writing area surface. The dot pattern on the locus in the vicinity of the erasing means is imaged, and the processing command means transmits the processing command for erasing the locus information of the character and / or the graphic and the locus information to the information processing means. It is preferable to do so.

According to the above features, the handwritten input / output system according to the present invention can not only correspond the writing information input on the writing area surface and the information processing means, but also correspond to the erasing of the writing information.

The locus recognition means corrects the coordinate information obtained by code analysis by the dot pattern analysis means so as to trace or approximate the locus of characters and / or figures drawn by tracing the writing area surface. It is preferable to do so.

According to the above characteristics, the handwritten input / output system according to the present invention matches the actual locus by correcting the locus information even when the locus information recognized by the locus recognition means and the locus on the medium are different. Alternatively, it can be approximated to the extent that it does not interfere with practical use.

The locus recognition means further includes a time measuring means, in which the time measuring means measures the time when the imaging means touches the writing area surface and the time when the imaging means is separated from the writing area surface, and the imaging means is drawn by touching the writing area surface. It is preferable to record the order of the loci and the time of contact and transmit them to the information processing means.

According to the above characteristics, the handwriting input / output system according to the present invention grasps the time required for handwriting input by the information processing means and compares it with the normal time to determine the character, age, gender, etc. of the writer. You can do various research.

It is preferable that the timekeeping means records the time when the imaging means touches the writing region surface and the dot pattern analysis means obtains the locus information, and transmits the time to the information processing means.

According to the above characteristics, the handwriting input / output system according to the present invention can analyze the splash and difficulty of writing by grasping the moving speed of the pen, and can also infer the psychological state and health state of the writer.

The processing instruction means is based on the code information and / or the recognition information, and is a specific information and / or an operation instruction (program start, command input, data input / output / search, search target) for specifying the writing area. It is preferable that the designation etc.) can be uniquely input.

According to the above features, in the handwritten input / output system according to the present invention, the processing instruction means can give a unique instruction to the information processing means based on the code information and / or the recognition information. This is the most important function in the present invention, and it is possible to easily identify which writing area is always written or what operation instruction is given to which writing area.

It is preferable that the locus recognition means further recognizes the locus information with high accuracy by referring to a pre-classified character and / or graphic information pattern table regarding the contents to be written in the writing area based on the code information.

According to the above features, the handwritten input / output system according to the present invention refers to a character and / or graphic information pattern table preclassified based on the code information when recognizing the locus information, so that the locus information is high. It can be recognized with accuracy.

After recognizing the locus information, the locus recognizing means obtains semantic information of the character and / or the figure, or a phrase composed of one or a plurality of the characters, based on the change of the coordinate information further analyzed. , The predefined semantic information code table (dictionary) corresponding to the code information is referred to, converted into the semantic information code, and the processing command means is the semantic information code and the processing command based on the semantic information. And are preferably transmitted to the information processing means.

According to the above features, the handwritten input / output system according to the present invention recognizes the locus information with high accuracy by referring to the semantic information code table at the time of locus recognition, and further converts the locus information into a semantic information code. A processing instruction can be transmitted as the meaning.

It is preferable that the processing instruction means instructs the information processing means to execute a first operation for inputting the recognition result of the locus information by the locus recognition means as a document and / or a graphic.

According to the above features, the handwritten input / output system according to the present invention can write a document or draw a picture by handwriting input.

It is preferable that the processing instruction means instructs the information processing means to execute a second operation in which the recognition result of the locus information by the locus recognition means is input as a comment.

According to the above features, the handwritten input / output system according to the present invention can input a comment document in addition to a document or a picture input by handwriting input.

The processing command means uses the recognition result of the locus information by the locus recognition means as an editing symbol for editing a document and / or a figure and / or a comment, and based on the editing symbol, the document and / or the figure and / Alternatively, it is preferable to instruct the information processing means to execute a third operation for editing the comment.

According to the above features, the handwritten input / output system according to the present invention can edit a document or an image (including one not based on the handwritten input / output system) by handwriting input.

The handwriting input / output system according to the present invention measures at least the inclination and the tilting direction formed by tilting the imaging optical axis of the imaging means in contact with the medium surface with respect to the vertical line of the medium surface. The tilt and / or direction by tilting the imaging means, that is, the grid tilt operation, with reference to the angle measuring means and the tilt and the direction measured when the imaging means comes into contact with the medium surface. It is preferable that the processing instruction means further includes an angle change recognizing means for analyzing a predetermined change, and the processing instruction means transmits a predetermined processing instruction to the information processing means based on the result of the analysis.

According to the above features, the handwriting input / output system according to the present invention can give an operation instruction by a grid tilt operation of the imaging means.

In the medium, the dot pattern arranged at a predetermined angle with respect to the direction of the writing area is formed in the writing area, and in the angle measuring means and the angle change recognizing means, the imaging means is the writing area. It is preferable to analyze a predetermined change in the tilting direction and / or the inclination with respect to the direction of the writing region based on the direction of the dot pattern read when the surface is contacted.

According to the above features, the handwritten input / output system according to the present invention can recognize the inclination and the change with reference to the direction of the dot pattern in addition to the change in the angle.

When the processing instruction means recognizes that the image pickup means is tilted in the first predetermined direction with respect to the upward direction of the image pickup means or the writing area by the grid tilt operation, the angle change recognition means recognizes that the image pickup means is tilted in the first predetermined direction. It is preferable to instruct the information processing means to execute the first operation of inputting the recognition result of the locus information by the locus recognition means as a document and / or a graphic.

According to the above features, the handwritten input / output system according to the present invention can be switched to the first operation by the grid tilt operation.

When the processing instruction means recognizes that the image pickup means is tilted in a second predetermined direction with respect to the upward direction of the image pickup means or the writing area by the grid tilt operation, the angle change recognition means recognizes that the image pickup means is tilted in a second predetermined direction. It is preferable to instruct the information processing means to execute a second operation for inputting the recognition result of the locus information by the locus recognition means as a comment.

According to the above features, the handwriting input / output system according to the present invention can be switched to the second operation by the grid tilt operation.

When the processing command means recognizes that the image pickup means is tilted in a third predetermined direction with respect to the upward direction of the image pickup means or the writing area by the grid tilt operation, the angle change recognition means recognizes that the image pickup means is tilted in a third predetermined direction. , The recognition result of the locus information by the locus recognition means is used as an edit symbol for editing the document and / or the comment, and the execution of the third operation for editing the document and / or the figure and / or the comment based on the edit symbol. Is preferably ordered to the information processing means.

According to the above features, the handwriting input / output system according to the present invention can be switched to the third operation by the grid tilt operation.

When the angle change recognizing means recognizes that the processing command means is tilted in a fourth predetermined direction with respect to the upward direction of the imaging means or the writing area by the grid tilt operation. , It is preferable to instruct the information processing means to end the third operation.

According to the above characteristics, the handwritten input / output system according to the present invention terminates the third operation by the grid tilt operation, switches to the original first or second operation, and re-executes the third operation. Can be done.

When the information processing means determines whether or not to execute a predetermined process, the processing instruction means touches the medium surface only once within a predetermined time, and the dot pattern analysis means determines. When it is recognized, it is assumed that the information processing means is instructed to decide not to execute the predetermined process, the imaging means touches the medium surface once, and the medium surface is further touched within a predetermined time. When the dot pattern analysis means recognizes, it is preferable to instruct the information processing means to determine the execution of the predetermined process.

According to the above-mentioned features, the handwriting input / output system according to the present invention can instruct the determination / rejection of processing only by the operation of touching the surface of the medium by the imaging means.

In the handwriting input / output system, the imaging optical axis of the imaging means is set up substantially vertically with respect to the writing region surface, and the direction of the dot pattern read when the image pickup means comes into contact with the writing region surface and the imaging are performed. The operation of rotating the image pickup means, that is, the grid turn, with reference to the rotation angle reading means for reading the rotation angle formed by the angle of the means and the rotation angle measured when the image pickup means comes into contact with the medium surface. The processing instruction means further includes a rotation angle change recognition means for analyzing a predetermined change in the rotation angle by rotating the imaging optical axis in a predetermined rotation direction due to the operation, and the processing instruction means is the result of the analysis. It is preferable to transmit a predetermined processing instruction to the information processing means based on the above.

According to the above-mentioned features, the handwriting input / output system according to the present invention can give an instruction by a grid turn operation.

The information processing means that when the rotation angle change recognizing means recognizes that the imaging means has rotated in the first predetermined direction, the processing command means saves the change result of the file contents in the running application. When the means is instructed and the rotation angle change recognition means recognizes that the imaging means has rotated in the second predetermined direction opposite to the first predetermined direction, the file contents in the running application. It is preferable to instruct the information processing means to cancel the change of.

According to the above features, the handwritten input / output system according to the present invention can instruct to save the result of changing the file contents by the grid turn operation and to cancel the change.

The processing instruction means recognizes a change in rotation angle as a result of the imaging means rotating in a first predetermined direction and further rotating in a second predetermined direction opposite to the first predetermined direction within a predetermined time. When the means recognizes, the information processing means is instructed to start the handwriting input application, the imaging means rotates in the second predetermined direction, and the first predetermined direction is further within a predetermined time. When the rotation angle change recognizing means recognizes that the rotation angle has been rotated, it is preferable to terminate the handwriting input application.

According to the above features, the handwriting input / output system according to the present invention can instruct the start / end of the handwriting input / output application by the grid turn operation.

In the handwritten information processing system, after the writing area is imaged by the imaging means and the coordinate information and the code information are obtained by the dot pattern analysis means, the image pickup means traces a character in the writing area by a predetermined operation. , Or a voice recording means that records voice instead of drawing, and a pre-defined voice recognition information table (dictionary) corresponding to the code information, recognizes the recorded voice, converts it into character information, and processes information processing means. It is preferable that the voice recognition means for transmitting the information is further provided.

According to the above features, the handwritten input / output system according to the present invention can perform voice input instead of handwritten input.

It is preferable that the predetermined operation is performed by tracing or drawing a symbol meaning voice input in the writing area with the imaging means.

According to the above features, the handwriting input / output system according to the present invention can easily switch to voice input as an extension of a series of operations during handwriting input, and records the history of starting voice input in a writing area or a computer. Can be left on.

It is preferable that the predetermined operation is performed by touching and bouncing the imaging means on the writing area.

According to the above features, the handwriting input / output system according to the present invention can easily switch to voice input with a simpler operation, and uses voice input without leaving a history of starting voice input in the writing area. it can.

It is preferable that the predetermined operation is performed by recording a predetermined word in the voice recording means and recognizing the predetermined word by the voice recognition means.

According to the above-mentioned features, the handwriting input / output system according to the present invention can more intuitively start the voice input operation without memorizing troublesome operations.

The medium has voice guide instruction information defined in advance in the code information in the medium, and the handwritten input / output system further includes a voice output means for explaining the contents to be written in the writing area by voice. Is preferable.
<Handwriting input sheet>
The handwritten input sheet according to the present invention is characterized by comprising a medium including a writing area in which coordinate information and a dot pattern in which code information is repeatedly defined are formed in at least a part or a plurality of areas.

According to the above-mentioned features, in the handwriting input sheet according to the present invention, the dot pattern formed on the medium defines not only the coordinate information but also the code information, so that the coordinate information and the unique information or the command are collectively integrated in the handwriting input. Can be entered.

The medium is preferably formed by further superimposing text, graphics, and photographs related to the writing area.

According to the above-mentioned features, the handwriting input sheet according to the present invention allows the user to easily know the content of the writing area in the handwriting input.

The medium defines specific information and / or operation instructions (program startup, command input, data input / output / search, search target designation, etc.) that specify a predetermined medium surface and / or the writing area surface. It is preferable that an icon meaning the code information is further superimposed on the dot pattern that repeatedly forms the code information.

According to the above features, the handwritten input sheet according to the present invention allows the user to further know the content of unique information or instructions input based on the code information in the handwritten input.

The icon is preferably formed together with the dot pattern on a stickable sticker, sticky note, or other medium surface.

According to the above features, the handwritten input sheet according to the present invention can increase the degree of freedom in the layout of the medium, and can search and output the information handwritten and input in the writing area according to the convenience and preference of the user. It can be executed freely.

The handwritten input sheet preferably further includes Braille in and / or around the writing area.

According to the above characteristics, the handwriting input sheet according to the present invention contributes to maintaining the same input efficiency as that of a healthy person when a visually impaired person performs handwriting input.

It is preferable that the handwriting input sheet is further provided with a guide of a narrow convex portion along the outer circumference of the writing area so that the imaging means does not protrude when writing to the writing area.

According to the above characteristics, the handwriting input sheet according to the present invention is provided with a guide so that the writing area can be accurately traced and drawn by the imaging means, and the visually impaired person is similar to a healthy person in performing handwriting input. Contributes to maintaining input efficiency.
<Information input system>
The information input system according to the present invention has an infrared reflecting layer having a characteristic of reflecting infrared rays from one surface side and transmitting visible light, and a characteristic of being provided on one surface side of the infrared reflecting layer to diffusely reflect infrared rays. A dot pattern layer in which dots of a dot pattern in which coordinate information and / or code information are repeatedly defined are arranged in at least a part or a plurality of regions made of the material of the dot pattern layer, and an infrared reflective layer of the dot pattern layer. An information input auxiliary sheet, that is, a grid sheet, which is provided on the opposite side and is laminated with a protective layer having a property of transmitting infrared rays and visible light, and at least a part of a medium surface, the dots of the dot pattern emit infrared rays. An imaging means formed of a material having a property of absorbing, an imaging means for imaging and transmitting the dot pattern, and image data of the dot pattern imaged and transmitted by the imaging means on the dot pattern on the medium surface. Then, in the dot portion where the brightness is imaged lower than the periphery, or in the image data of the dot pattern imaged and transmitted by the imaging means for the dot pattern of the grid sheet, the brightness is imaged higher than the periphery. The dot portion is determined by a predetermined determination method, the image data of the dot pattern is stored in the storage element, and the code analysis is performed, so that the coordinate information and / or the code information is input from the dot pattern analysis means. It is characterized by becoming.

According to the above features, the information input system according to the present invention can avoid the eyeball phenomenon without providing the diffuse reflection layer on the grid sheet.

In the predetermined determination method, the captured images stored in the storage element composed of pixels of m rows and n columns are separated from each other by a predetermined interval (k) in the right direction starting from the left end for each row (i). The difference (ΔBL ＝ BL (i, j + k) −BL (i, j)) of the brightness (BL (i, j)) of is obtained in order, and whether the difference exceeds a predetermined positive / negative threshold. In the method of determining whether the brightness is lower than or equal to, and the determination of the dot portion whose brightness is lower than that of the periphery, when the difference in brightness falls below the negative threshold, the pixel on the right side is determined as a dot, and that pixel is determined. It is determined that the dot started from, and when it exceeds the positive threshold, it is also determined that the pixel on the right side is not a dot, and it is determined that the dot ends up to the pixel in front of it. In the determination of the dot portion to be imaged, when the difference in brightness exceeds the positive threshold, the pixel on the right side is determined as a dot, it is determined that the dot starts from that pixel, and the same applies when the difference falls below the negative threshold. It is a determination method including a method of determining that the pixel on the right side is not a dot and determining that the dot is completed up to the pixel in front of the pixel, and the dot pattern analysis means is the medium according to the predetermined determination method. It is preferable to determine whether it is a surface or the dot pattern provided on the grid sheet, and perform code analysis by the predetermined determination method.

The predetermined determination method uses a determination method of a dot portion in which the brightness of a predetermined region of the captured image is measured, and if the brightness exceeds a predetermined threshold value, the brightness is lower than that of the periphery, and the predetermined determination method is used. If it is below the threshold value, it is preferable to use a method for determining a dot portion whose brightness is higher than that of the periphery.
<Information input assistance sheet>
The information input auxiliary sheet according to the present invention has an infrared reflecting layer having a characteristic of reflecting infrared rays from one surface side and transmitting visible light, and a characteristic of being provided on one surface side of the infrared reflecting layer and diffusely reflecting infrared rays. A dot pattern layer in which dots of a dot pattern in which coordinate information and / or code information are repeatedly defined are arranged in at least a part or a plurality of regions, and an infrared reflective layer of the dot pattern layer. It is characterized in that a protective layer provided on the opposite side of the surface and having a property of transmitting infrared rays and visible light is laminated.

According to the above characteristics, the information input auxiliary sheet according to the present invention can avoid the eyeball phenomenon without providing the diffuse reflection layer.

The input auxiliary sheet according to the present invention is provided on both sides of an infrared reflecting layer having a property of reflecting infrared rays incident on each surface and transmitting visible light, and a property of absorbing infrared rays. A dot pattern layer in which dots of the dot pattern are arranged and a protective layer provided on the outer side of each of the dot pattern layers and having a property of transmitting infrared rays and visible light incident from the respective directions. And / or are laminated, and the coordinate information and / or the code information patterned in the dot pattern arranged in each of the dot pattern layers is the same as or different from each other.

According to the above characteristics, the information input auxiliary sheet according to the present invention can avoid the eyeball phenomenon without providing the diffuse reflection layer.

In the handwritten input / output system and the handwritten input sheet according to the present invention, the writing area preferably has an arbitrary shape.

Conventionally, when trying to specify the writing area using only the coordinate information, the coordinates indicating the boundary of the writing area are defined in advance in the table or function, and the input coordinate values fall into the category of the table or function. I had to judge sequentially. For this reason, the analysis becomes complicated, and when the writing area is specified using only the coordinate information, the writing area needs to be substantially rectangular in order to maintain the analysis efficiency such as the memory used and the calculation speed. In the present invention, by specifying the writing area using the code information, the same analysis efficiency can be exhibited regardless of the shape of the writing area.

In the handwritten input / output system and the handwritten input sheet according to the present invention, the medium is provided with a plurality of reference points so that the size and orientation of the block can be formed in the area of the block in which the information dots are arranged, and the reference points are provided. A plurality of virtual reference points defined from are arranged, and the coordinate information or the information dot in which the coordinate information and the code information are defined by the distance and the direction from the virtual reference point is arranged, and the plurality of virtual reference points are arranged. Among the combinations of the arrangement directions of the information dots in the above, the combination of the arrangement directions of the predetermined information dots forms a dot pattern in which the coordinate information or the coordinate information and the code information are repeatedly defined in at least a part or a plurality of areas. It is preferable that the dot pattern analysis means is provided with the written area, determines that it is the written area by the combination of the arrangement directions of the predetermined information dots, and processes the handwritten input.

According to the above characteristics, when handwriting input is performed, a predetermined information dot is not required to operate a button separately provided in the handwriting input / output system or to read a dot pattern in which a handwriting input operation instruction is defined by an imaging means. It is determined that it is a writing area only by code-analyzing the combination of the arrangement directions of If it is not a write area, the code information and / or the information associated with the coordinate information is output and / or the instruction processing is executed.

In the handwritten input / output system and the handwritten input sheet according to the present invention, the combination of the arrangement directions of the predetermined information dots is such that the information dots are on the vertical / horizontal direction lines and diagonally from the virtual reference points at the plurality of virtual reference points. It is preferable that the combination of the arrangement directions of the predetermined information dots among the combinations of the arrangement directions arranged on any of the direction lines.

According to the above characteristics, since the arrangement direction of the information dots is limited to the up, down, left, right, and diagonal directions, the dot pattern analysis means can accurately recognize the combination of the arrangement directions of the predetermined information dots, and is it a writing area? Whether or not it is determined, and if it is a writing area, it is processed as a handwritten input. If it is not a write area, the code information and / or the information associated with the coordinate information is output and / or the instruction processing is executed.

A write flag indicating that the code information is in the write area is defined in the code information, and it is preferable that the dot pattern analysis means determines that the code information is in the write area and processes the handwritten input.

According to the above characteristics, when the imaging means reads the dot pattern formed on the medium surface, the dot pattern analysis means immediately determines by the write flag whether or not the area is the write area, and in the write area, If there is, it is processed as handwritten input. If it is not a write area, the code information and / or the information associated with the coordinate information is output and / or the instruction processing is executed.

The code information defines specific information that specifies the writing area and / or an operation instruction that means a writing process, and the dot pattern analysis means determines that the specific information and / or the operation instruction. It is preferable to process handwritten input / output.

According to the above characteristics, when the imaging means reads the dot pattern formed on the medium surface, it is possible to instantly identify in which writing area the dot pattern is written. When the content handwritten in this writing area belongs to a predetermined category, the recognition rate is remarkably improved by collating with the dictionary of that category. In addition, it is possible to determine how to process the written contents and perform appropriate processing.

The locus information includes coordinate information of the image pickup center of the image pickup means, coordinate information of a position where the instruction protrusion or the pen tip provided in the vicinity of the image pickup means touches the writing area surface, and code information. , It is preferable that the dot pattern analysis means obtains the calculation by interpolation based on the coordinate information defined in the dot pattern.

According to the above features, the coordinate information defined in the dot pattern is necessary to accurately reproduce the locus information because the increment of each block in which one set of XY coordinate values is defined is one unit. A predetermined accuracy of less than one unit can be obtained by interpolation calculation based on the coordinate information. For example, when the size of one block is 2 mm, in order to accurately reproduce the character, the XY coordinate value is obtained with an accuracy of about 0.1 to 0.2 mm, which is 1/10 to 1/20 of that, and the locus information. You just have to recognize.

The present invention provides a handwritten input / output system capable of giving a user a variety of new added value and convenience, which is completely different from the conventional technology by having the above-mentioned various features. Can be done.

It is an external view which shows the use state of the handwriting input / output system which concerns on this invention. It is a block diagram which shows the structure of the 1st Example of the handwriting input / output system which concerns on this invention. It is a block diagram which shows the structure of the 2nd Example of the handwriting input / output system which concerns on this invention. It is a block diagram which shows the structure of the 3rd Example of the handwriting input / output system which concerns on this invention. It is a block diagram which shows the structure of the 4th Example of the handwriting input / output system which concerns on this invention. It is a block diagram which shows the structure of the 5th Example of the handwriting input / output system which concerns on this invention. It is a block diagram which shows the structure of the 6th Example of the handwriting input / output system which concerns on this invention. It is a figure (1) explaining the dot pattern used in the handwriting input / output system which concerns on this invention. It is a figure (2) explaining the dot pattern used in the handwriting input / output system which concerns on this invention. It is a figure (3) explaining the dot pattern used in the handwriting input / output system which concerns on this invention. It is a figure (4) explaining the dot pattern used in the handwriting input / output system which concerns on this invention. It is a figure (5) explaining the dot pattern used in the handwriting input / output system which concerns on this invention. It is a figure (6) explaining the dot pattern used in the handwriting input / output system which concerns on this invention. It is a figure explaining the direction dot used in the handwriting input / output system which concerns on this invention. It is a figure (7) explaining the dot pattern used in the handwriting input / output system which concerns on this invention. It is an external view which shows the whole structure of the image pickup part used for the handwriting input / output system which concerns on this invention. It is a figure (1) explaining the function of the imaging unit used in the handwriting input / output system which concerns on this invention. It is a figure (2) explaining the function of the imaging unit used in the handwriting input / output system which concerns on this invention. It is a figure (3) explaining the function of the imaging unit used in the handwriting input / output system which concerns on this invention. It is a figure (4) explaining the function of the imaging unit used in the handwriting input / output system which concerns on this invention. It is a figure (5) explaining the function of the imaging unit used in the handwriting input / output system which concerns on this invention. It is a figure (6) explaining the function of the imaging unit used in the handwriting input / output system which concerns on this invention. It is a figure (7) explaining the function of the imaging unit used in the handwriting input / output system which concerns on this invention. It is a figure (8) explaining the function of the imaging unit used in the handwriting input / output system which concerns on this invention. It is a figure (9) explaining the function of the imaging unit used in the handwriting input / output system which concerns on this invention. It is a figure (10) explaining the function of the imaging unit used in the handwriting input / output system which concerns on this invention. It is a figure (11) explaining the function of the imaging unit used in the handwriting input / output system which concerns on this invention. It is a figure (12) explaining the function of the imaging unit used in the handwriting input / output system which concerns on this invention. It is a figure (1) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (2) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (3) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (4) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (5) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (6) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (7) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (8) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (9) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (10) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (11) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (12) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (13) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (14) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. FIG. 1 is a diagram (1) illustrating a method of calculating the direction and angle of inclination of the imaging unit used in the handwriting input / output system according to the present invention. FIG. 2 is a diagram (2) illustrating a method of calculating the direction and angle of inclination of the imaging unit used in the handwriting input / output system according to the present invention. FIG. 3 is a diagram (3) illustrating a method of calculating the direction and angle of inclination of the imaging unit used in the handwriting input / output system according to the present invention. FIG. 4 is a diagram (4) illustrating a method of calculating the direction and angle of inclination of the imaging unit used in the handwriting input / output system according to the present invention. FIG. 5 is a diagram (5) illustrating a method of calculating the direction and angle of inclination of the imaging unit used in the handwriting input / output system according to the present invention. It is a figure explaining the method of discriminating the touch operation and the bounce operation of the image pickup unit used in the handwriting input / output system according to the present invention. It is a figure explaining the 1st operation in the handwriting input / output system which concerns on this invention. It is a figure (1) explaining the 2nd operation in the handwriting input / output system which concerns on this invention. It is a figure (2) explaining the 2nd operation in the handwriting input / output system which concerns on this invention. It is a figure (3) explaining the 2nd operation in the handwriting input / output system which concerns on this invention. It is a figure (1) explaining the 3rd operation in the handwriting input / output system which concerns on this invention. It is a figure (2) explaining the 3rd operation in the handwriting input / output system which concerns on this invention. It is a figure (3) explaining the 3rd operation in the handwriting input / output system which concerns on this invention. It is a figure (4) explaining the 3rd operation in the handwriting input / output system which concerns on this invention. It is a figure (5) explaining the 3rd operation in the handwriting input / output system which concerns on this invention. It is a figure (6) explaining the 3rd operation in the handwriting input / output system which concerns on this invention. It is a figure explaining the method of saving the file change result by the rotation operation of the image pickup unit in the handwriting input / output system which concerns on this invention. It is a figure explaining the method of canceling the file content change by the rotation operation of the image pickup unit in the handwriting input / output system which concerns on this invention. It is a figure explaining the method of starting an application by the rotation operation of the image pickup part in the handwriting input / output system which concerns on this invention. It is a figure explaining the termination method of the application by the rotation operation of the image pickup part in the handwriting input / output system which concerns on this invention. It is a figure (1) explaining the operation switching method by the tilting operation of the image pickup unit in four directions in the handwriting input / output system which concerns on this invention. FIG. 2 is a diagram (2) for explaining a method of switching operations by tilting the imaging unit in four directions in the handwriting input / output system according to the present invention. It is a figure explaining the touch operation of the image pickup part in the handwriting input / output system which concerns on invention. It is a block diagram explaining the structure of the audio recording unit and the audio output unit in the handwriting input / output system according to the present invention. It is a figure explaining the bouncing operation of the image pickup part in the handwriting input / output system which concerns on this invention. It is a figure (1) explaining the grid sheet which is an Example of the medium used for the handwriting input / output system which concerns on this invention. It is a figure (2) explaining the grid sheet which is an Example of the medium used for the handwriting input / output system which concerns on this invention. It is a figure (3) explaining the grid sheet which is an Example of the medium used for the handwriting input / output system which concerns on this invention. It is a figure (4) explaining the grid sheet which is an Example of the medium used for the handwriting input / output system which concerns on this invention. It is a figure (5) explaining the grid sheet which is an Example of the medium used for the handwriting input / output system which concerns on this invention. It is a figure (1) explaining the grid sheet used for the information input system which concerns on this invention. It is a figure (2) explaining the grid sheet used for the information input system which concerns on this invention. It is a figure (3) explaining the grid sheet used for the information input system which concerns on this invention. It is a figure (4) explaining the grid sheet used for the information input system which concerns on this invention. It is a figure (5) explaining the grid sheet used for the information input system which concerns on this invention. It is a figure (15) explaining the function of the medium which formed the writing area used for the handwriting input / output system which concerns on this invention. It is a figure (7) explaining the dot pattern used in the handwriting input / output system which concerns on this invention. It is a figure (8) explaining the dot pattern used in the handwriting input / output system which concerns on this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Explanation of the overall system configuration>
FIG. 1 is an external view showing an example of a usage state of the handwriting input / output system 1 according to the present invention. According to the figure, when writing is performed on the medium 2 using the scanner 3, the document as written is displayed on the display 6. This is realized by the scanner 3 capturing the dot pattern on the written trajectory and analyzing it by the computer 4 (accessing the server 5 as needed). However, the computer 4 may be built in the scanner 3.

FIG. 2 is a block diagram showing a configuration of a first embodiment of the handwritten input / output system 1 according to the present invention.

The handwritten input / output system 1 has at least a medium 2 (handwritten input sheet), an imaging unit 7 (imaging means), a dot pattern analysis unit 8 (dot pattern analysis means), a locus recognition unit 9 (trajectory recognition means), and a processing instruction unit. By providing 10 (processing instruction means), various instructions are given to the information processing unit 11, and various servers 5 are accessed as needed.

FIG. 3 is a block diagram showing a configuration of a second embodiment of the handwritten input / output system 1 according to the present invention. In the second embodiment, in addition to the configuration provided in the first embodiment, the image sensor 12, the irradiation unit 13, and the optical control unit 14 are further provided.

Here, as shown by the broken line in the figure, the image sensor 12 and the irradiation unit 13 are integrated with the image pickup unit 7, and the optical control unit 14 is integrated with the dot pattern analysis unit 8.

FIG. 4 is a block diagram showing a configuration of a third embodiment of the handwriting input / output system 1 according to the present invention. In the third embodiment, in addition to the configuration provided in the first embodiment, the timing unit 15 is further provided.

Here, as shown by the broken line in the figure, the time measuring unit 15 is integrated with the locus recognition unit 9.

FIG. 5 is a block diagram showing a configuration of a fourth embodiment of the handwriting input / output system 1 according to the present invention. In the fourth embodiment, in addition to the configuration provided in the first embodiment, the angle measuring unit 16 and the angle change recognition unit 17 are further provided.

Here, as shown by the broken line in the figure, the angle measuring unit 16 and the angle change recognition unit 17 are integrated with the dot pattern analysis unit 8.

FIG. 6 is a block diagram showing a configuration of a fifth embodiment of the handwriting input / output system 1 according to the present invention. In the fifth embodiment, in addition to the configuration provided in the first embodiment, the rotation angle reading unit 18 and the rotation angle change recognition unit 19 are further provided.

Here, as shown by the broken line in the figure, the rotation angle reading unit 18 and the rotation angle change recognition unit 19 are integrated with the dot pattern analysis unit 8.

FIG. 7 is a block diagram showing a configuration of a sixth embodiment of the handwriting input / output system 1 according to the present invention. In the first embodiment, in addition to the configuration provided in the first embodiment, the voice recording unit 20, the voice recognition unit 21, and the voice output unit 22 are further provided.

<Dot pattern>
Hereinafter, the dot pattern used in the handwriting input / output system 1 according to the present invention will be described with reference to FIGS. 8 to 15.

<Explanation of dot pattern GRID1>
8 to 15 are explanatory views showing GRID 1 which is an example of a dot pattern formed on the medium 2 in the handwriting input / output system 1 according to the present invention.

In these figures, the grid lines in the vertical and horizontal directions are added for convenience of explanation and do not exist on the actual printed surface. The key dots 102, information dots 103, reference grid point dots 104, and the like constituting the dot pattern 101 are invisible ink or carbon ink that absorbs the infrared light when the scanner 3 which is an imaging means has an infrared irradiation means. It is desirable that it is printed with.

FIG. 8 is an enlarged view showing an example of bit display of the information dots of the dot pattern 101 and the data defined therein. 9 (a) and 9 (b) are explanatory views showing an information dot 103 in which the key dot 102 is arranged at the center.

The information input / output method using the dot pattern includes generation of the dot pattern 101, recognition of the dot pattern 101, and means for outputting information and a program from the dot pattern 101. That is, the dot pattern 101 is captured as image data by the imaging unit 7 built in the scanner 3, the reference grid point dots 104 are first extracted, and then the dots are not struck at the positions where the reference grid point dots 104 are originally located. The key dot 102 is extracted by, and then the information dot 103 is digitized to extract the information area to quantify the information, and the information and the program are output from the dot pattern 101 from the numerical information. .. For example, information or a program such as voice is output from the dot pattern 101 to an information output device, a personal computer, a PDA, a mobile phone, or the like.

In the generation of the dot pattern 101, fine dots, that is, key dots 102, information dots 103, and reference grid point dots 104 are arranged according to a predetermined rule in order to recognize numerical information by a dot code generation algorithm. As shown in FIG. 7, in the block of the dot pattern 101 representing information, 5 × 5 reference grid point dots 104 are arranged with reference to the key dot 102, and the center surrounded by the four reference grid point dots 104. The information dot 103 is arranged around the virtual grid point 5. Arbitrary numerical information is defined in this block. In the illustrated example of FIG. 8, four blocks (inside the thick line frame) of the dot pattern 101 are arranged in parallel. It should be noted that the dot pattern 101 is not limited to the four blocks, and may be arranged repeatedly in the vertical and horizontal directions as many times as necessary.

When the image pickup unit 7 captures the dot pattern 101 as image data, the reference grid point dots 104 are distorted by the lens of the image pickup unit 7, imaged from an angle, expansion and contraction of the paper surface, curvature of the medium surface, and distortion during printing. Can be corrected. Specifically, a correction function (X n, Y _{n ) = f (X n'} , Y n') for converting the distorted four reference grid point dots 104 into the original square is obtained, and the information is obtained by the same function. The dot 103 is corrected to obtain the correct vector of the information dot 103.

When the reference grid point dots 104 are arranged in the dot pattern 101, the image data captured by the image pickup unit 7 corrects the distortion caused by the image pickup unit 7, so that a lens having a high distortion rate is attached. It can be accurately recognized even when the image data of the dot pattern 101 is captured by a popular camera. Further, even if the imaging unit 7 is tilted with respect to the surface of the dot pattern 101 and read, the dot pattern 101 can be accurately recognized.

As shown in FIG. 8, the key dot 102 is a dot in which four reference grid point dots 104 at the corners of the four corners of the block are arranged so as to be shifted in a certain direction. The key dot 102 is a representative point of the dot pattern 101 for one block representing the information dot 103. For example, the reference grid point dots 104 at the four corners of the block of the dot pattern 101 are 20% of the distance between the reference grid point dots 104 adjacent to the upper side (0 when the distance between the reference grid point dots is 0.5 mm). It is shifted by about 1 mm). However, this numerical value is not limited to this, and can be changed according to the size of the block of the dot pattern 101.

The information dot 103 is a dot that recognizes various types of information. The information dot 103 is arranged around the key dot 102 as a representative point, and the center surrounded by the four reference grid point dots 104 is set as the virtual grid point 5, which is represented by a vector as a starting point. It is placed at the end point. For example, the information dot 103 is surrounded by the reference grid point dots 104, and as shown in FIG. 9A, 20% of the distance between the virtual grid points 5 and the adjacent reference grid point dots 104 (reference grid points). Dots separated by about 0.1 mm when the dot spacing is 0.5 mm have the direction and length expressed by the vector, so they are rotated by 45 degrees clockwise and arranged in 8 directions. Represent a bit. Therefore, 3 bits × 16 = 201 bits can be expressed by the dot pattern 101 of one block.

FIG. 9B is a method of defining the information dot 103 having 2 bits for each grid in the dot pattern of FIG. 3, and the information of each 2 bits is shifted by shifting the dots in the + direction and the × direction. It is defined. As a result, 201-bit information can be originally defined, but data can be given in units of 32 bits by dividing the information according to the application. + The direction and × direction of combinations can be realized up to ^{2 16} (approximately 65000) dot pattern formats street.

Not limited to this, it is possible to express 4 bits by arranging them in 16 directions, and it goes without saying that various changes can be made.
When the scanner 3 which is an imaging means is brought close to or touches the medium surface to image a dot pattern, the dot diameters of the key dots 102, the information dots 103, or the reference grid point dots 104 are based on the appearance and the quality of the paper. Considering the printing accuracy, the resolution of the image pickup unit 7, and the optimum digitization, 6 to 16% of the spacing between the adjacent reference grid point dots 104 (0.03 when the spacing between the reference grid point dots is 0.5 mm). ~ 0.08 mm) is desirable. However, depending on the size of the photographing port of the scanner 3 which is an imaging means, it is necessary to be formed so that a predetermined dot pattern is imaged in the imaging region, and the arrangement size of the dot pattern of one block and the diameter of the dots are appropriately set. Of course, it is decided. Further, in consideration of the required amount of information for the imaging area and misidentification of various dots 102, 103, 104, it is desirable that the distance between the reference grid point dots 104 is about 0.3 to 0.5 mm in length and width. Considering the misidentification of the reference grid point dot 104 and the information dot 103, the deviation of the key dot 102 is preferably about 20 to 25% of the grid spacing.

On the other hand, although not shown, when a non-contact scanner or a built-in camera of a mobile phone, which is an imaging means, captures a dot pattern at a predetermined distance from the medium surface, the predetermined dot pattern is captured in the imaging region. It needs to be formed, and it goes without saying that the arrangement size of the dot pattern of one block and the diameter of the dots are appropriately determined.

It is desirable that the distance between the information dot 103 and the virtual grid points surrounded by the four reference grid point dots 104 is about 15 to 30% of the distance between the adjacent virtual grid points 5. This is because if the distance between the information dots 103 and the virtual grid points 5 is closer than this distance, the dots are easily recognized as large lumps, which makes the dot pattern 101 unsightly. On the contrary, if the distance between the information dot 103 and the virtual grid point 5 is longer than this distance, it is difficult to determine which of the adjacent virtual grid points 5 is the information dot 103 having vector directionality. Because it becomes.

As shown in FIG. 8, one dot pattern is a dot pattern composed of 4 × 4 block areas, and 2-bit information dots 103 are arranged in each block. FIG. 9 shows an example of the dot code format of the information dot 103.

As shown in FIG. 10, a parity check, a code value, an X coordinate, and a Y coordinate are recorded in one dot pattern. The details of the format will be described later.

FIG. 11 is an example of bit display of the information dot 103 and the data defined therein, and shows other forms.

Further, 4 bits can be expressed by using two types of information dots 103, long and short, from the virtual grid point 5 surrounded by the reference grid point dots 104, and setting the vector direction to 8 directions. At this time, it is desirable that the longer one is about 25 to 30% of the distance between the adjacent virtual grid points 5, and the shorter one is about 15 to 20%. However, it is desirable that the center spacing of the long and short information dots 103 is longer than the diameter of these dots.

The information dot 103 surrounded by the four reference grid point dots 104 is preferably one dot in consideration of appearance. However, if it is desired to ignore the appearance and increase the amount of information, a large amount of information can be obtained by allocating one bit for each vector and expressing the information dot 103 with a plurality of dots. For example, the vector of concentric eight directions, an information dot 103 surrounded by four points lattice dots 4 can represent information of 2 ^8, and 16 pieces of information dots of one block 2 ^128.

FIG. 12 shows an example of bit display of information dots and data defined therein. FIG. 12A shows two dots, (b) shows four dots, and (c) shows five dots. It shows.

FIG. 13 shows a modification of the dot pattern 101, in which (a) is a 6-information dot arrangement type, (b) is a 9-information dot arrangement type, and (c) is a 12-information dot arrangement type, (d). ) Is a schematic diagram of a type in which 1036 information dots are arranged.

The dot pattern 101 shown in FIG. 8 shows an example in which 16 (4 × 4) information dots 103 are arranged in one block. However, the information dots 103 are not limited to being arranged in 16 in one block, and can be changed in various ways. For example, six information dots 103 (2 × 3) are arranged in one block (a), and nine information dots 103 are arranged in one block (a) according to the required amount of information or the resolution of the imaging unit 7. There are 3 × 3) arranged (b), 12 information dots 103 arranged in one block (3 × 4) (c), and 36 information dots 103 arranged in one block (d). ..

<Explanation of dot pattern Direction dot>
Next, a direction dot, which is another form of the dot pattern, will be described with reference to FIG.

In this dot pattern, the direction of the dot pattern is defined by the shape of the block. In FIG. 14A, first, reference points 201a to 201e are arranged. A shape indicating the direction of the block (here, a pentagon facing upward) is defined by a line connecting the reference points 201a to 201e. Then, the virtual reference points 201f, 201g, and 201h are arranged based on the reference point, and the information dot 203 is arranged at the vector end point having a direction and a length with the virtual reference point as a start point. As described above, in the figure, the orientation of the block can be defined by the method of arranging the reference points. The size of the entire block is also defined by the direction dots that define the orientation of the block. Further, it goes without saying that this block may be repeatedly arranged in the vertical and horizontal directions as many times as necessary.

In FIG. 14A, the reference points 201a to 201e and the information dot 203 are all of the same shape, but the reference points 201a to 201e may be larger than the information dot 203. Further, the reference points 201a to 201e and the information dot 203 may have any shape as long as they can be distinguished, and may be a triangle, a quadrangle or a polygon.

FIG. 14B defines information according to whether or not there are information dots on the virtual grid points of the block.

Note that FIG. 14 (c) shows two blocks shown in FIG. 14 (a) connected in the vertical and horizontal directions.

The dot pattern formed on the medium 2 in the handwritten input / output system 1 according to the present invention is not limited to GRID 1, and may be any one that defines XY coordinate information and code information.

For example, Japanese Patent No. 3766678 (GRID2), Japanese Patent No. 3858051 (GRID3), Japanese Patent No. 3858052 (GRID4), and Japanese Patent Application No. 2009-165163 (stream dot pattern) can be used.

<Explanation of dot pattern format>
FIG. 15 is a diagram illustrating a format of a dot pattern formed on the medium 2 in the handwriting input / output system 1 according to the present invention.

As shown in FIG. 3D, the dot pattern is a dot pattern composed of 4 × 4 block areas, and is divided into C _{1-0 to} C _31-30 within this block. The dot code formats of each region are shown in (a) and (b).

FIG. 3A is a format in which the page code, the X coordinate, and the Y coordinate are registered. The page code means the number of pages of the medium 2 when the medium 2 is in the form of a booklet, and the X coordinate and the Y coordinate mean the positions (coordinate values) on the medium, respectively. It is a thing. As a result, when the user reads the medium 2 with the scanner 3, the number of pages of the medium and the coordinate position of the read portion are simultaneously recognized by one reading operation.

FIG. 3B is a format in which a page code, an action code, an X coordinate, and a Y coordinate are registered. The action code means an operation instruction. For example, it is defined only in the area where the icon meaning the operation such as "enlargement" and "reduction" provided on the page is drawn, and is mainly included in the dot pattern used in the icon area described later. ing. Further, when the action code is defined on the entire writing area of the page and the necessary processing is performed during or after the handwriting input, the instruction processing may be associated with the action code and executed. For example, instruction processing refers to processing such as character recognition during or after handwriting input or transmission to another information processing device. Although the use as a page code is described here, a plurality of specific writing area codes in parentheses may be provided in one medium surface, and those writing areas should be defined by a unique code. Therefore, it is possible to instantly determine where the writing is performed. Of course, it may be defined together with the page code so that it can be determined which specific writing area of which page. Further, when the handwritten content belongs to a predetermined category, the recognition rate is remarkably improved by collating with the dictionary of that category. Needless to say, the content to be input by handwriting may be not only characters but also symbols and illustrations. Of course, the action code may be similarly defined together with the specific write area code and the instruction processing may be executed.

In FIG. 3C, if the dot pattern formed on the medium surface is imaged by the imaging means by providing the writing area flag and analyzed by the dot pattern analysis unit 8, it can be immediately determined whether or not the dot pattern is the writing area. If it is not a writing area, the output and / or instruction processing of the information associated with the code value and / or the coordinate value is executed, and if it is a writing area, a predetermined processing of handwritten input is executed by the locus recognition means. Of course, the write area flag may be accompanied by the page code (specific write area code) or action code shown in the figures (a) and (b).

As described above, in the dot pattern of the present invention, various code information such as a page code and an action code and XY coordinates can be registered in one format. In addition, only XY coordinates or only code information can be registered, and a flexible format is possible.
<Image pickup unit>
16 to 28 are views for explaining the image pickup unit 7 used in the handwriting input / output system 1 according to the present invention.

The image pickup unit 7 preferably has a size and shape suitable for being held by a person, and most preferably has a pen shape as shown in FIG.

FIG. 3A is a diagram showing the most basic embodiment of the pen-shaped imaging unit 7. In such an embodiment, an instruction protrusion 23 and a transmission unit 24 are provided, and a C-MOS sensor or the like (not shown) is generally built in on the axis of the pen. The transmission unit 24 is for the imaging unit 7 to transmit a signal to another member by using a method such as infrared communication or wireless communication, and is shown for convenience of explanation, but it is not always visible in appearance. It does not have to be in place.

FIG. 3B is a diagram showing another embodiment of the pen-shaped imaging unit 7. In such an embodiment, two buttons for receiving an operation by the user are provided. These two buttons are a transmission button 25 and a retransmission button 26, and the user can instruct by button operation to send a signal from the image pickup unit 7 to another member. Here, the retransmission button 26 is used to retransmit the same signal when the transmission of the signal by the transmission button 25 cannot be transmitted to the receiving member due to the presence of a shield or the like.

FIG. 3C is a diagram showing another embodiment of the pen-shaped imaging unit 7. In such an embodiment, a microphone 27 for receiving voice input by the user and a voice reproduction button 28 are further provided. The microphone 27 is used to record the voice used for voice recognition described later, and the voice reproduction button 28 is used to reproduce the voice guide described later.

FIG. 3D is a diagram showing another embodiment of the pen-shaped imaging unit 7. In such an embodiment, the pen 29 is stored in the internal space of the instruction protrusion 23, and the pen 29 is ejected by pressing the pen storage button 30.

FIG. 17A is a cross-sectional view showing the tip of a pen-type scanner 31, which is one of the embodiments of the imaging unit 7 constituting the handwriting input / output system 1 according to the present invention. Such a pen-type scanner 31 is the same as the lens 33, that is, a light guide 32 having a hollow portion formed inside and having an opening at the tip thereof, a lens 33 arranged at a position facing the opening of the light guide 32, and the lens 33. The LED 34 (irradiation unit 13) as a light source that irradiates the medium surface on which the dot pattern is formed with light of a predetermined wavelength, and the C-MOS sensor 35 (imaging) arranged at a position retracted from the lens 33. It includes an element 12), a PCB 36 at a further retracted position, and a CPU (not shown) installed therein. The hollow portion of the light guide 32 is formed in a tapered shape in which the diameter gradually increases from the lens 33 toward the opening. Here, the CPU has any or all of the above-mentioned dot pattern analysis unit 8, locus recognition unit 9, processing instruction unit 10, and information processing unit 11.

The light guide 32 is made of a transparent or milky white resin, and the inside thereof functions as a light guide path. The irradiation light from the LED 34 travels inside the light guide 32 and irradiates the medium 2 through the opening. When the light guide 32 is made of a milky white resin, the irradiation light from the LED 34 is appropriately diffused as it travels in the light guide 32, so that the light irradiating the medium 2 from the opening becomes more uniform. Can be done.

<Control of LED light emission timing>
FIG. 3B is a diagram showing the light emission timing of the LED 34 for each time. The LED 34 instantaneously emits light at a predetermined intensity at a predetermined timing to expose the medium 2. The C-MOS sensor 35 captures the reflected light from the medium 2 and transmits it as image data to the dot pattern analysis unit 8.

FIG. 3C is a diagram showing the reaction state of the element of the C-MOS sensor 35 for each time. The afterimage of the medium 2 is printed on the element of the C-MOS sensor 35 at the moment when the LED 34 emits light with a predetermined intensity and exposes the medium 2. Here, the predetermined time t _k at the time until the afterimage of imaging element 12 disappears, indicating the time required to hold the analyzable afterimage state by dot pattern analysis unit 8.

If LED34 time _{f t} that has been emitted is equal to or less than the predetermined time alpha, as shown in FIG. 2 (d), since the element of C-MOS sensor 35 is burned without afterimage dot blurs, this afterimage Is transmitted as image data to the dot pattern analysis unit 8. In this case, the transmission is performed during the _{time t l when the afterimage remains in the element.}

On the other hand, if the LED34 time _{f t} which has been emitting largely exceeds the predetermined time alpha, as shown in FIG. (E), the element of C-MOS sensor 35 is burned blurred afterimage dots, correct number Cannot be read.

Such an imaging method is particularly effective when a C-MOS sensor is used, but there is no problem even if a CCD camera is used instead of the above imaging method.

The functions of the optical control unit 14 and the dot pattern analysis unit 8 used in the handwriting input / output system 1 according to the present invention will be described with reference to FIG.

FIG. 1A shows the first timing of the light emission of the LED 34 by the optical control unit 14, the timing of capturing the dot pattern by the image sensor 12 and transmitting it to the storage element, and the timing of analyzing the image data by the dot pattern analysis unit 8. It is a time series that explains the pattern. In the first pattern, light emission _f t _n , imaging _r t _n , and analysis _a t _n are performed in order, and when the analysis _a t _n is completed, the next light emission _f t _{n + 1 is performed until a predetermined time t s elapses.} Is not done. Here, the predetermined time t _s in which the image pickup element 12 is out of the expected value of the time required for imaging, transmitting, analyzing a dot pattern, and defines the maximum value as the predicted imaging, transmission and analysis time.

FIG. 3B shows a second pattern of the light emission timing of the LED 34 by the optical control unit 14, the imaging / transmission timing of the dot pattern by the image pickup device 12, and the analysis timing of the image data by the dot pattern analysis unit 8. It is a series. In the second pattern, the analysis _a t _n ends, the light control unit 14 detects the completion of the analysis, immediately following emission _{f t} n _{+ 1} is performed, the imaging of the next dot pattern _{r t} n _{+ 1,} analysis _a t _{n + 1} is performed.

FIG. 3C shows a third pattern of the light emission timing of the LED 34 by the optical control unit 14, the imaging / transmission timing of the dot pattern by the image pickup device 12, and the analysis timing of the image data by the dot pattern analysis unit 8. It is a series. In the third pattern, by providing a plurality of task processing or light control unit 14 and dot pattern analysis unit 8, the analysis _{a t n-1} simultaneously with captures an image _r t _n of the dot pattern for the next analysis, analysis _a t immediately emission when _n-1 is completed _f t _{n + 1} is performed, followed by already analyzed _a t _n of the image data captured by the image capturing _r t _n is performed. Still further, may start emitting _f t _{n + 1} at the same time analyzing _a t _n, but since the light emitting time of the LED34 is extremely short, the dot pattern analysis unit 8 from the emission LED34 is an analysis of the image data There is almost no change in the total time to finish.

<Pen mounting>
Further, the pen-type scanner 31 is equipped with a pen 37 as a writing instrument. When printing with a writing instrument such as a pen, it is usually used in an inclined state, so that the outer circumference of the pen-type scanner 31 is formed so as to be inclined up to 45 degrees. In this case, by imaging the medium surface near the tip of the pen 37, it is possible to accurately analyze the handwriting, the moving distance, and the like when printing the pen 37 at a position away from the imaging center. In addition, the handwriting can be analyzed more accurately by the offset correction described later.

The pen 37 as a writing instrument can be stored by various methods so as not to leave a handwriting on the medium. As a method of storing, the technique used in a pen such as a general ballpoint pen can be used as it is. For example, a knock type or rotary type storage method may be adopted, or a cap may be put on the storage method.

It is also possible to change the pen like a general three-color ballpoint pen. In this case, not only the pens of different colors may be exchanged, but also an eraser or correction fluid may be used instead of the pens.

Further, instead of the pen 37, an instruction protrusion 23 (stylus pen) may be provided to take an image of the vicinity of the handwriting of the instruction protrusion 23. A storage space may be provided in the instruction protrusion 23 to store the pen 37, and the pen 37 may also be used as the instruction protrusion.

<Offset correction>
As shown in FIG. 17A, in the pen-type scanner 31, the XY coordinate values defined by the dot pattern imaged by the imaging unit 7 and the XY coordinate values defined by the dot pattern on the trajectory actually written by the pen 37 are different. Further, it is necessary to convert the XY coordinate value defined by the dot pattern on the locus into the xy coordinate value in the writing area on the medium surface to perform writing input and operation instruction.

In the handwriting input / output system 1 according to the present invention, a method of correcting the dot pattern on the trajectory actually written by the pen 37 for the XY coordinate values of the dot pattern imaged by the imaging unit 7 and the XY coordinates of the dot pattern coordinate system. The conversion method from the value to the xy coordinate value of the writing area coordinate system will be described below.

<Relationship between the coordinate system of the writing area and the coordinate system of the dot pattern>
FIG. 19 shows an example of the writing area.

In the writing area coordinate system, the lower left corner is (0,0). The xy coordinate value of the writing area is displayed in mm. For example, the coordinate position of (16,40) indicates a position moved 16 mm to the right and 40 mm upward from the lower left corner of the writing area.

In the writing area, a block of a dot pattern in which one set of XY coordinate values of the dot pattern coordinate system is defined is continuously formed. The XY coordinate value indicates the position of the center point of the block. In order to distinguish the coordinate values of the dot pattern coordinate system from the coordinate values of the writing area coordinate system, they are expressed as [X, Y].

If the lower left corner of the writing area is the origin (0,0) of the coordinate system and the lower left corner of the block of the dot pattern that defines [0,0] matches the origin (0,0), XY in the dot pattern coordinate system As shown in FIG. 20, a dot pattern is formed on the coordinate values so that the lower left corner is [-0.5, -0.5] in any writing area. In this case, a unique code value is defined for the dot pattern formed in each writing area to specify each writing area. The same code value may be defined if they do not need to be identified from each other.

Further, as shown in FIG. 22, a dot pattern in which the XY coordinate values are defined is formed on the entire surface of the paper, and a unique code value is defined in the writing area in addition to the XY coordinate values. In this case, the XY coordinate value in the lower left corner of the writing area is [Xn0, Yn0] (n is a number that specifies the writing area).

Next, assuming that one side of each block is L mm, the XY coordinate value is converted into the writing area coordinate system as follows.

When a dot pattern is formed with the XY coordinate value of the origin (0,0) as [-0.5, -0.5] as shown in FIG. 20, the coordinates [X, Y] in the dot pattern coordinate system are the writing area coordinate system. Then,
(X × L + L / 2, Y × L + L / 2).
As shown in FIG. 21, when the lower left corner of the writing area is set as the origin (0,0) of the coordinate system and the center point and the origin of the block of the dot pattern in which [0,0] is defined are matched,
(X × L, Y × L).

The relationship between the writing area coordinate system and the dot pattern coordinate system in the example of FIG. 21 is the same as that of the enlarged view portion of FIG.

As shown in FIG. 22, when the XY coordinate value in the lower left corner of the writing area is [Xn0, Yn0], the coordinates [X, Y] in the dot pattern coordinate system are in the writing area n coordinate system.
((X−Xn0) × L, (Y−Yn0) × L).
<Calculation of XY coordinate values [Xc, Yc] in the dot pattern coordinate system of the imaging center C>
A method of calculating the XY coordinate values [Xc, Yc] of the imaging center C will be described with reference to FIG. 23.
The captured image is temporarily stored in a frame buffer having a predetermined resolution, and the image is analyzed to obtain the number of pixels M constituting the center spacing of adjacent blocks. Since the center spacing of the block is equal to the length L of one side of the block, if the size per pixel of the captured image with respect to L is the value k in the dot pattern coordinate system,
k = L / M.

Similarly, by image analysis, the number of pixels u constituting the distance between the block center P [Xp, Yp] closest to the image pickup center C and the image pickup center C can be obtained. The distance CP from the image center C to the nearest block center P from the image center is
CP = k × u.

Next, when the dot pattern is formed on the paper so that the direction of the dot pattern (direction of the writing area) is upward of the writing area, the angle formed by the direction of the dot pattern and the PC vector is θ'(left). If the rotation is in the positive direction),
[Xc, Yc] = [Xp-CP x sinθ', Yp + CP x cosθ']
Is sought.

<Calculation of xy coordinate value Q (xq, yq) in the writing area coordinate system of the pen tip (when the pen is perpendicular to the paper surface)>
As shown in FIG. 24, the coordinate value of the imaging center C in the writing area coordinate system is (xc, yc), and the coordinate value on the medium surface of the pen tip is Q (xq, yq). As shown in FIG. 25, assuming that the difference between the x-coordinate value and the y-coordinate value of the point C and the point Q is Δxq and Δyq, respectively.
Δxq ＝ xq −xc
Δyq ＝ yq−yc
Is sought.

When the dot pattern is formed so that the direction of the dot pattern is upward in the writing area, the direction of the CQ vector with the imaging center C as the start point and the pen tip as the end point is the direction of the pen-type scanner 31. ..

The angle between the direction of the dot pattern and the direction of the pen-type scanner 31 is θ (left rotation is the positive direction).

Further, as shown in FIG. 26, where s is the distance from the imaging center C to the pen tip Q when the pen-type scanner 31 is perpendicular to the paper surface.
Δxq and Δyq are
Δxq ＝ −s × sinθ
Δyq ＝ s × cosθ
Will be.

Therefore, xq and yq are
xq ＝ xc −s × sinθ
yq ＝ yc ＋ s × cosθ
Will be.

<Calculation of xy coordinate value Q'(xq', yq') in the writing area coordinate system of the pen tip (when the pen is tilted with respect to the paper surface)>
As shown in FIG. 27, when the pen-type scanner 31 is tilted with respect to the paper surface, the distance from the image pickup center C to the pen tip Q changes.

Assuming that the distance from the center of imaging to the pen tip is s'when the inclination of the pen-type scanner 31 from the normal direction of the paper surface (hereinafter referred to as "pen inclination") is α, the above equation is Replaced by s → s',
xq'＝ xc − s'× sinθ
yq'＝ yc ＋ s'× cosθ
Will be.

Also,
s'＝ s / cosα
Therefore, when this is substituted into the equation,
xq'＝ xc − s'× sin θ / cos α
yq'＝ yc ＋ s'× cosθ / cosα
Will be.

<Distance from the center of imaging to the pen tip when the pen tip is stored in the pen body>
As shown in FIG. 28, when the pen tip is stored, the pen-type scanner 31 approaches the paper surface by ΔL, but does not affect s and s', and the calculation formula is the same.

<Calculation method of pen tilt α>
In order to detect the tilt α of the pen, a tilt measuring means such as a tilt sensor may be provided on the pen body.

Alternatively, the brightness BLi, j = 1, p of a predetermined plurality of p points of the captured image is obtained, and the inclination of the pen is tilted from the relational expression α = f (ΔBL) using the difference ΔBL between the brightness of the brightest part and the darkest part. You can also find α.

It is also possible to find α using the Fourier function or sin function with BLi, j as parameters such as α = f (BLi, j = 1, p).

Alternatively, a table corresponding to ΔBL and α may be created in advance to calculate α. The direction in which the pen is tilted can be obtained by measuring the direction from the darkest part of BLi, j = 1, p to the brightest part.

<Media 2>
29 to 42 are diagrams for explaining the medium 2 used in the handwriting input / output system 1 according to the present invention.

FIG. 29 is a diagram showing a medium 2 in which a dot pattern is printed (formed) on the entire surface to form a writing area.

Such a dot pattern defines an XY coordinate value and a page code. Here, the page code means a code for specifying a medium by using code information.

FIG. 30 is a diagram showing a medium 2 in which another dot pattern in which an action code is further defined is printed in an area in which a dot pattern in which a page code and an XY coordinate value are defined is printed.

Here, the action code means a code that instructs the information processing unit 11 to perform a predetermined operation (particularly an operation of an application) in response to a handwritten input to a specified medium by using code information. As a result, security can be improved and erroneous operations can be reduced.

This medium is a set of a plurality of sheets, and different page codes are assigned to each sheet. The page code is, for example, "01" for the first page, "02" for the second page, and "03" for the third page.

In the area where the dot pattern that defines the action code is printed, text, graphics, photographs, and icons can be overlaid and printed to explain to the user what the action code means. it can. For example, the action code in the area printed with the text icon "start" superimposed commands the information processing unit 11 to start the handwriting input / output application. The action code in the area printed with the text icon "Cancel" superimposed indicates that the information processing unit 11 is instructed to discard (cancel) the input result by the handwritten input / output application, or the information processing unit 11 causes the information processing unit 11 to discard (cancel) the input result. When deciding whether or not to execute a predetermined process, the information processing unit 11 is instructed to stop the execution of the process. The action code in the area printed with the text icon "save" is instructed to save the input result by the handwriting input / output application to the information processing unit. The action code of the area printed on which the text icon "end" is superimposed commands the information processing unit 11 to end the handwriting input / output application.

FIG. 31 is a diagram showing a medium 2 in which an area on which a dot pattern defining an XY coordinate value and a page code is printed and an area on which a dot pattern defining an action code and a page code are printed are separated. In this case, various commands can be executed only for the page specified by the page code without selecting which page of the medium is to be targeted. As a result, security can be improved and erroneous operations can be reduced.

FIG. 32 is a diagram showing a first medium on which a dot pattern defining XY coordinate values and a page code is printed, and a second medium which is a paper controller on which a dot pattern defining only an action code is printed. .. In this case, various commands for handwriting input can be executed on any page of the medium. When the writing area and the icon are provided separately, as shown in FIG. 32, the writing area and the icon may be attached to another medium as a sticker, a sticky note, or a card, in addition to the case of using a paper controller.

The action code in the area where the text icon "text input" is superimposed and printed instructs the information processing unit 11 to input characters or figures to be the text in the operation by the handwriting input / output application. The action code in the area where the text icon "comment input" is superimposed and printed instructs the information processing unit 11 to input characters or figures to be comments in the operation by the handwriting input / output application. The action code in the area where the text icon "edit" is superimposed and printed commands the information processing unit 11 to edit characters or figures in the operation by the handwritten input / output application (that is, handwritten input / output). It is also possible to edit files such as documents and images created by applications other than the application for use.) The action code of the area printed on which the text icon "decision" is superimposed causes the information processing unit 11 to execute the process when the information processing unit 11 determines whether or not to execute the predetermined process. Order to decide.

FIG. 33 shows a first medium on which the dot pattern defining the XY coordinate values and the page code A is printed, and a second medium which is a paper controller on which the dot pattern defining the action code and the page code A is printed. It is a figure which shows what became one booklet.

In this case, a dot pattern in which the same XY coordinates are registered on each page may be printed in the writing area. This is because the write area can be specified by using the icon of the controller area in which the action code and the page code are registered when performing the write operation. That is, it can be said that the position and page of the writing area can be changed without changing the XY coordinate values. In this case, various commands can be executed only for the page specified by the page code without selecting which page of the medium is to be targeted. As a result, security can be improved and erroneous operations can be reduced.

In addition to the above, a dot pattern defining an action code for instructing the information processing unit to start the handwriting input / output application may be printed on the entire writing area. In this case, the handwriting input / output application is automatically started when writing is performed on the medium surface.

Further, the writing area on the medium is not limited to one, and a plurality of writing areas can be defined.
(When multiple write areas are provided)

Hereinafter, with reference to FIGS. 34 to 38, a case where a plurality of writing areas are provided will be described with reference to four examples.

<First example>
As shown in FIG. 34, the first example is a case where a dot pattern that defines an XY coordinate value by a different dot pattern coordinate system and a code value that specifies a writing area is formed for each of a plurality of writing areas. ..

In this figure, a dot pattern in which the XY coordinate values in the dot pattern coordinate system and the code values m and n that specify the writing area are defined is formed for each writing area.

By defining a plurality of pieces of information, the code value can include not only the writing area but also the classification number for classifying the page number.

When the page number is included, it is possible to determine which writing area of which page, and there are selection items such as information, processing instructions, and alternatives in the writing area, and when processing as a common code, the defined page number. Or, the information that identifies the selected item can be defined as a code value.

As shown in FIG. 35, different code values can be defined by dividing the area in the writing area. In this figure, the code values are n1, n2, n3, and n4. As a result, a plurality of selection items can be set in one writing area.

In this first example, since the lower left corner of each region is always 0 (origin), there is an advantage that it is easy to handle.

<Second example>
In the second example, as shown in FIG. 36, a dot pattern in which one dot pattern coordinate system defines an XY coordinate value is formed on the entire surface of a predetermined area, and a code value is also defined in the dot pattern in the writing area together with the XY coordinate. If it is.

In this figure, a dot pattern in which the XY coordinate values are defined in one dot pattern coordinate system with the lower left corner of the predetermined region as the origin is formed on the entire surface of the predetermined region. The two writing areas within this predetermined area are defined as XY coordinate values whose lower left corners are [Xm0, Ym0] and [Xn0, Yn0], and code values m and n that specify the writing area, respectively. ing.

As for the code value, as in the first example, the area can be divided and different code values can be defined in the writing area.

It should be noted that writing is not possible in an area other than the writing area where a dot pattern is formed.

According to the second example, by reading the XY coordinate values, it is easy to know where in the predetermined area the writing area is laid out.

<Third example>
The third example is a case where the writing area is further inserted in the writing area as shown in FIG. 37.

In this figure, a dot pattern that defines an XY coordinate value in one dot pattern coordinate system with the lower left corner of the predetermined area as the origin and a code value L that specifies the writing area is formed on the entire surface of the predetermined area. There is. The two write areas within the predetermined write area are defined by the XY coordinate values whose lower left corners are [Xm0, Ym0] and [Xn0, Yn0], respectively, and the code values m and n that specify the write area. Has been done.

According to this example, it becomes easy to arrange a small writing area such as a selection item field in a large writing area where sentences and figures are freely written.

If this is to be realized by the first example, when a writing area is further formed in the writing area, it is necessary to provide a writing area with a hole, and the formation of a dot pattern becomes a complicated work.

Further, if a write area can be formed in the write area in one coordinate system, the write area in the write area can be ignored and the write work can be performed by changing the input mode. It will be possible. For example, when writing to the writing area in the writing area, a setting may be provided so as to be effective only when the input mode is switched or a specific symbol is drawn.

As for the code value, as in the first example, the area can be divided and different code values can be defined in the writing area.

In this example, by reading the XY coordinate values, it is easy to know where the writing area is laid out.

<Fourth example>
In the fourth example, as shown in FIG. 38, one dot pattern coordinate system is defined on the entire surface of a predetermined area, and the XY coordinate value in this coordinate system and the code value for specifying the writing area are defined in the writing area. This is the case when a pattern is formed.

In this figure, one coordinate system with the lower left corner as the origin is defined in a predetermined area, and the lower left corners are [Xm0, Ym0] and [Xn0, Yn0] in the two writing areas in this area, respectively. A dot pattern is formed in which a certain XY coordinate value and code values m and n that specify a writing area are defined.

In this example, unlike the second example, no dot pattern is formed in a range other than the writing area.

In this example, if there is a block copy of printed matter such as graphics and text, and a block copy of dot pattern, the range of the writing area is automatically laid out by arranging a mask according to the coordinate system of the block copy of the printed matter. It is possible to create a block copy of a dot pattern by performing mask processing, and it is possible to easily issue a dot pattern.

Also, when the dot pattern is read, by reading the XY coordinate values, it is possible to know where the writing area is laid out in the predetermined area, so for example, research on where on the paper the advertising effect should be high. Can be used for.

In all of the above explanations, the dot pattern coordinate system has been used, but it is natural that the same applies to the writing area coordinate system converted from the dot pattern coordinate system.

FIG. 39 is a diagram showing a document which is a specific example in which a plurality of writing areas are defined on a medium.

FIG. 3A is a diagram showing a document in which a specific code for instructing the origin of the XY coordinate value and the specification of the writing area is defined in each writing area.

In the figure (a), the writing area of "Frigana" is in the writing area of "Name", the writing area of "Zip code" is in the writing area of "Address", and the writing area is in the writing area of "Map of the surrounding area". The writing areas of "walk (time)", "train (use time)", and "bus (use time)" are defined.

In addition, the writing area for "gender" is divided into the writing areas for "male" and "female", and the writing area for "date of birth" is divided into the writing areas for "year", "month", and "day". ..

The layout of such a writing area is possible by assigning a different specific code to each.

FIG. 3B is a diagram showing the document in which the origin of the XY coordinate values is not defined in each writing area and only the specific code for instructing the specification of the writing area is defined.

With such a configuration, the position of the written information on the entire medium surface can be specified.

FIG. 40 is a diagram showing a state in which handwriting input is actually performed on the document of FIG. 39.

FIG. 3A is a diagram showing a document of FIG. 39A in which handwriting input is actually performed.

FIG. 3B is a diagram showing a state in which the result of processing performed by the information processing unit 11 in response to the handwritten input of FIG. 3A is displayed on the display 6 or output by printing.

Although the details will be described later, the handwritten input / output system 1 according to the present invention outputs the characters and figures as shown in FIG. 3B by recognizing the characters and figures input by handwriting by some methods.

<Print a new dot pattern>
In the handwriting input / output system 1 of the present invention, handwriting input can be performed, the information processing unit 11 can print a reflection of the handwriting input, and the handwriting input can be performed again.

FIG. 41A is a diagram showing the locus information recognized by the locus recognition unit 9 and the medium 2 printed by superimposing the dot pattern. Although not shown, a dot pattern different from the dot pattern printed on the medium 2 in which the handwriting input is first performed may be newly printed on the medium 2 shown in FIG. 41 (a).

FIG. 41B is a diagram showing a medium 2 in which the result of processing performed by the information processing unit 11 is printed by superimposing a dot pattern. Although not shown, a dot pattern different from the dot pattern printed on the medium 2 in which the handwriting input is first performed may be newly printed on the medium 2 shown in FIG. 41 (b). In this case, the XY coordinates are the same, and only the page code may be changed for printing.

<Braille / Bank>
Further, the medium 2 may be further provided with a bank guide 38 (narrow convex portion) or Braille 39.

In FIG. 42, a bank guide 38 is provided on the outer periphery of the writing area, and a Braille 39 protrusion is provided in the writing area together with a dot pattern. By providing the bank guide 38 in this way, the position of the writing area can be known even when a user with a visual impairment uses handwriting input / output, and by providing the Braille 39 and the dot pattern in the same area, the user has a visual impairment. Even when the user uses handwritten input / output, the same input efficiency as that of a healthy person can be maintained.

<Dot pattern analysis unit>
The dot pattern analysis unit 8 obtains the XY coordinate information and the code information defined by the dot pattern by code-analyzing the image data according to the information input / output method using the dot pattern in GRID1 described above.

In the handwritten input / output system 1 according to the present invention, since the dot pattern analysis unit 8 sequentially code-analyzes the locus written (traced) on the medium by the imaging unit 7, the XY coordinate information and the code information are the loci. (Trajectory information).

<Trajectory recognition unit>
The locus recognition unit 9 recognizes characters and figures written on the medium based on the change in the locus information.

The locus recognition method is a method of considering characters and figures as graphics and referring to the XY coordinate patterns of the characters and figures in a database, as in a general OCR (optical character reader).

Here, in the present invention, in particular, the content to be written in the writing area can be instructed in advance by the code information. That is, it is possible to recognize the locus information with high accuracy by referring to the pre-classified character and / or graphic information pattern table. As a specific example, if it is instructed to describe only Arabic numerals in advance, the locus recognition unit 9 refers only to the table of Arabic numerals, so that the recognition rate is increased. If it cannot be recognized, all characters and figures in the database can be referred to as usual.

Further, after performing the above locus recognition, the semantic information of the characters and / or figures written in the writing area, or the words and phrases consisting of one or a plurality of characters is the predefined semantic information corresponding to the code information. By referring to the code table (dictionary), it can be converted into a semantic information code. That is, for example, when the locus recognition unit 9 writes "Bunkyo-ku, Tokyo", it first recognizes patterns one by one, such as "East", "K computer", "Miyako" sentence, "K computer", and "ward". , Refer to the semantic information code table (dictionary) and convert to the semantic information code defined in advance corresponding to "Tokyo" and the semantic information code defined in advance corresponding to "Bunkyo Ward". As a result, the information processing department can recognize that the place names "Tokyo" and "Bunkyo-ku" have been input.

In addition, referring to the semantic information code table (dictionary) also assists in character pattern recognition. For example, if it is difficult for the locus recognition unit 9 to recognize whether the user wrote "East" or "Car" because the user wrote the characters "East" of "Tokyo" dirty, semantic information. If the idiom "Tokyo" is defined in the code table and the idiom "Kyo" is not defined (the idiom "Kyo" usually does not exist), the user wrote the word "East". The locus recognition unit 9 can recognize the locus.

<Timekeeping section>
The handwriting input / output system 1 according to the present invention may further include a time measuring unit 15. The timekeeping unit 15 has a function of detecting time information.

Here, the time information is
(1) The writing history is in the order in which the imaging unit 7 touches the writing area surface and is drawn.

(2) The time until the image pickup unit 7 touches the writing area surface and separates from the writing area surface is set as a set, and the time when the image pickup unit 7 separates from the start of writing is ignored. That is, the time when the imaging unit 7 is separated is omitted, and the handwriting input time and its order are described from the start and end times of (1). As a result, the time for writing characters can be known, the accuracy of character recognition can be improved, and the character input speed of the target person can be known.

(3) By recording all the times when the image pickup unit 7 touches and leaves the writing area surface, it is possible to know how long it took for the questionnaire, product selection, etc., and the comparison with the normal time is performed. By doing so, it is possible to grasp the difficulty, the state of thinking, the character of the writer, the age, the gender, and the like.

(4) In addition, by grasping the moving speed of the pen, it is possible to analyze the splash and difficulty of writing characters, and further to estimate the psychological state and health state of the writer.

<Angle measurement unit / angle change recognition unit>
The handwriting input / output system 1 according to the present invention may further include an angle measuring unit 16 and an angle change recognizing unit 17. The angle measuring unit 16 and the angle change recognizing unit 17 measure and recognize the inclination of the imaging unit 7 with respect to the medium surface.

As the method of angle measurement / angle change recognition, the method of calculating the inclination α of the pen already described can be used as it is. In this case, it is possible to measure and recognize the tilt even if the imaging unit 7 does not have a pen. Further, as shown in FIGS. 43 to 47, the method of calculating the inclination disclosed in the invention of the information output device according to Japanese Patent Application No. 2005-267565 (PCT / SG2006 / 00267) can be used. However, methods other than these may be used.

It should be noted that the recognition of the change in the angle of the imaging unit 7 with respect to the medium surface, which is the reference for measuring and recognizing the inclination, is not limited to the angle when the imaging unit 7 is vertically erected with respect to the medium surface, and the imaging unit 7 is the medium. It shall be possible to use the angle when touching the surface as a reference.

<Rotation angle reading unit / rotation angle change recognition unit>
The handwriting input / output system 1 according to the present invention may further include a rotation angle reading unit 18 and a rotation angle change recognition unit 19. The rotation angle reading unit 18 and the rotation angle change recognition unit 19 read and recognize the rotation angle when the image pickup optical axis of the image pickup unit 7 is set up substantially vertically and rotated.

The method of determining the angle between the orientation of the dot pattern and the orientation of the imaging unit 7 has already been explained in the section of offset correction, but this method is used for reading and recognizing the rotation angle regardless of the presence or absence of a pen. it can. That is, since the orientation of the dot pattern is always constant, the orientation of the dot pattern when the imaging unit 7 is erected vertically above the medium surface and the first image is taken and the angle θ _{1 formed by} the orientation of the imaging unit 7 are predetermined. The difference between the direction of the dot pattern after the lapse of time and the angle θ ₂ (with the left rotation as the positive direction) formed by the direction of the imaging unit 7 can be read and recognized as the rotation angle.

<How to distinguish between touch motion and bouncing motion>
In the handwriting input / output system 1 according to the present invention, a method of discriminating between an operation of touching the image pickup unit 7 on the writing area (touch operation) and an operation of bouncing from the state of touching the writing area (bouncing operation) will be described with reference to FIG.

As shown in FIG. 6A, when the imaging unit 7 touches the writing area, the dot pattern of the writing area can be imaged at the moment when the imaging unit 7 touches the writing area. Then, when the image pickup unit 7 is separated from the writing area, the irradiation light from the LED 34 is not reflected on the medium surface, so that the C-MOS sensor 35 does not detect the irradiation light from the LED 34 (reflected light from the medium surface). As shown in (c), the dot pattern cannot be imaged.

As shown in FIG. 3B, when the imaging unit 7 is bounced from the state of touching the writing area, the dot pattern of the writing area can be imaged at the moment when the imaging unit 7 touches the writing area, that is, here. Up to is the same as the operation of touching the writing area. Then, when the imaging unit 7 is bounced from the state of touching the writing area, the tip of the imaging unit 7 is still directed to the writing area for a predetermined time, so that the irradiation light from the LED 34 is reflected on the medium surface, so that C-MOS The sensor 35 takes an image of the dot pattern on the medium surface for the time being, but since the moving speed of the imaging unit 7 is high, the dot pattern cannot be blurred and analyzed as shown in FIG.

That is, the imaging unit 7 touches the writing area to image the dot pattern, and analyzes the image data captured after a lapse of a predetermined time (eg, 0.2 seconds later) with the moment of analysis as the reference time to analyze the dots. When the pattern is not imaged (reflected light is not detected), the imaging unit 7 is predefined as performing a touch operation, and the processing instruction unit 10 is predefined with respect to the information processing unit 11 as corresponding to the touch operation. Send a processing instruction. When the dot pattern is imaged (although the reflected light is detected), but the dot pattern is blurred and cannot be analyzed, the image pickup unit 7 assumes that the image pickup unit 7 has bounced, and the processing instruction unit 10 informs the information processing unit 11. Then, a pre-defined processing instruction is transmitted as corresponding to the bouncing motion. As shown in FIG. 3E, when the dot pattern can be analyzed, it is nothing but a normal handwriting input operation.

<Processing instruction unit>
The processing command unit 10 transmits a processing command based on the information recognized by the locus recognition unit 9 to the information processing unit 11 together with the locus information.

The processing instruction unit 10 is the instruction for specifying the page based on the page code, the instruction for specifying a certain writing area based on the specific code, and various instructions based on the action code defined by the dot pattern printed on the medium surface as described above. Actually performs to the information processing unit 11.

Further, although the details will be described later, the processing instruction unit 10 instructs the information processing unit 11 to execute various operations in the handwriting input / output application.

Further, a predetermined processing command based on the measurement / recognition result by the angle measuring unit 16 / angle change recognition unit 17 and the reading / recognition result by the rotation angle reading unit 18 / rotation angle change recognition unit 19 is sent to the information processing unit 11. Command against.

<Information Processing Department>
The information processing unit 11 receives a processing instruction performed by the handwritten input / output system 1 according to the present invention and executes various processes.

<Server>
The handwritten input / output system 1 according to the present invention may access the server 5 in which various information is recorded, if necessary.

<Server for trajectory recognition>
For example, information for trajectory recognition can be stored in the server 5. An XY coordinate pattern is defined and stored in a plurality of characters and figures as used in a general OCR (optical character reader).

In addition, classification codes for classifying characters and figures are defined and stored.

Further, for characters and figures, a word or figure consisting of one or more characters, or a semantic information code for outputting a combination thereof is defined and stored by referring to a semantic information code table (dictionary). Has been done.

In the handwritten input / output system 1 according to the present invention, since the writing area can be specified using not only the XY coordinate information but also the code information, even if the amount (number) of the writing area is huge, it can be easily managed. can do. In particular, it is extremely effective when storing written information in a shared global server that can be used by everyone.

<Other servers>
Needless to say, the server 5 can store various information such as voice, document, image, moving image, application, and the like in addition to the information for locus recognition. For example, a handwriting input / output application can be stored. That is, when the information processing unit 11 does not have the handwriting input / output application, it is possible to search the server and execute the handwriting input / output application on the server. However, the handwriting input / output application on the server may be executed with priority.

Instead of the server 5, another handwriting input / output application of the information processing unit 11 may be executed.

As a matter of course, the server 5 can be implemented by changing various forms other than the above embodiment.

<Application for handwriting input / output>
The handwritten input / output application executed in the handwritten input / output system 1 according to the present invention will be described below.

Such an application is a system that reflects writing on a medium surface on a computer by executing various operations.

Writing to the medium surface may be performed while looking at the display 6 on which the processing result is displayed by instructing the information processing unit 11 in real time by the processing command unit 10, or writing without looking at the display 6. It is assumed that the processing instruction unit 10 may give an instruction to the information processing unit 11 after writing.

<Write operation>
Hereinafter, the writing operation in the handwriting input / output system 1 according to the present invention will be described.

The writing operation is performed by writing characters or figures to be input into the writing area on the medium surface using the imaging unit 7.

While the imaging unit 7 is moving on the medium surface, the imaging unit 7 images a dot pattern printed superimposed on the writing area at a predetermined time interval. The later procedure is as described above.

Hereinafter, each operation executed by the handwriting input / output system 1 will be described.

<How to recognize characters and figures>
FIG. 49 is an image diagram illustrating a first operation of inputting characters in the text by handwriting.

When characters are written in the writing area using the imaging unit 7 as shown in FIG. 6A, the imaging unit 7 sequentially images the dot patterns on the trajectory and analyzes the dot patterns as shown in FIG. Part 8 acquires locus information including XY coordinate information and code information.

For convenience of explanation, in FIG. 3B, the interval between the imaging positions of the dot pattern on the locus is wide, but it is actually narrower.

As shown in FIG. 3C, the information processing unit 11 executes the first operation based on the result of recognition based on the locus information, and outputs the result to the display 6.

As shown in FIG. 6D, there is no change in the recognition method between the case of recognizing a figure and the case of recognizing a character, and the recognition result is displayed on the display 6 as shown in FIG. 6E.

However, the XY coordinate information obtained by the dot pattern analysis unit 8 may be input / output as it is without recognizing characters and figures. In this case, the handwriting input / output system 1 according to the present invention functions as a pen tablet.

<Second operation>
As shown in FIG. 50, the second operation is an operation of inputting the characters of the comment by handwriting. In the figure, the comment part is surrounded by a broken line for convenience of explanation, but it is not necessary to actually write the broken line.

The method of recognizing characters or figures is the same as the method of the first operation.

The character of the comment input in the second operation is also recognized by the information processing unit 11 as a comment different from the text.

As shown in FIG. 51, the comment entered between the lines may be reflected as it is in the information processing unit 11, but as shown in FIG. 52, information can be obtained by specifying the part in the text in which the comment is to be inserted underlined. Comments may be listed beside the text in the processing unit 11.

<Third operation>
The third operation is an operation for editing characters input by handwriting.

Each edit is performed by writing characters or figures defined in advance as edit symbols in the information processing unit 11 using the image pickup unit 7.

The method of recognizing characters or figures is the same as the method of the first operation.

FIG. 53 is a diagram illustrating an editing symbol used in the editing operation which is the third operation of the present invention.

The lines that form one closed area mean an edit symbol that selects a character or figure within the area.

Two sets of parenthesized lines that do not close the area mean an edit symbol that selects a character or figure between the two sets of lines.

The check mark means an edit symbol that specifies the target of cutting when cutting and pasting.

The double line means the delete (erase) edit symbol.

"Torr" means a delete (erase) edit symbol.

"Iki" means the editing symbol for canceling the editing symbol "Torr".

A zigzag line of three or more round trips means an edit symbol for canceling an edit.

A double-headed arrow means an edit symbol that swaps selected characters or shapes.

A one-way arrow means an edit symbol that directly specifies the paste destination.

Letters (A), (B), (C). .. .. Means an edit symbol that copies (cuts) the selected character or figure with a flag and specifies the paste destination at a distant position.

However, it goes without saying that editing symbols other than the above may be defined.

FIG. 54 is a diagram showing the medium 2 before executing the third operation. A document input by handwriting by executing the first operation or a document printed as a result of inputting by handwriting is written on the medium 2.

FIG. 55 is a diagram illustrating an editing state on the medium 2 in the third operation.

As shown in FIG. 3A, when writing a line that closes an area, the handwriting input / output application recognizes the area closed by the line as one editing target.

When writing a parenthesis-shaped line that does not close the area as shown in Fig. (B), one area is drawn in the later area by drawing the corresponding parenthesis-closing line as shown in Fig. (C). It is recognized in the handwriting input / output application as the editing target of. In particular, it is used when the editing target spans several lines.

As shown in FIG. 3D, when a “torr” is drawn on the line indicating the editing target in the state where the editing target is selected, the character to be edited is deleted in the handwriting input / output application.

As shown in Fig. (E), when drawing an arrow to specify the edit target from the edit target to a predetermined position with the edit target selected, cut and paste or copy the edit target to the specified area in the handwriting input / output application. & Paste. Here, the paste destination is not only inserted between characters (or at the beginning or end of the line), but as shown in the figure (f), the original character is deleted and overwritten by editing the paste destination. It is also possible to do.

In this case, as shown in Fig. (G), when a check mark, which is a deletion editing symbol, is added on the line indicating the editing target of the copy source, it is processed as cut and paste in the handwriting input / output application and cut. The target of and is deleted. As shown in FIG. 3H, when the check mark is not added, it is processed as copy and paste in the handwriting input / output application.

As shown in the figure (i), if an editing symbol that becomes a flag such as (A) or (B) is added to the editing target, the editing target is copied with the flags (A) and (B) attached. After that, you can paste as many times as you like. Further, as shown in FIG. 6J, it is possible to paste at distant positions such as different pages.

When a double line is drawn on the character as shown in the figure (k), the character with the double line is deleted in the handwriting input / output application.

As shown in FIG. 6 (l), when two editing targets are connected by the editing symbols of the two-way arrows, the editing targets are exchanged in the handwriting input / output application.

FIG. 56 is a diagram showing a sentence in which the result of each editing in the third operation performed on the medium 2 is reflected in the handwriting input / output application.

FIG. 57 is a diagram illustrating an operation of canceling the editing and returning to the state before editing after editing the third operation shown in FIG. 60.

As shown in Fig. (A), if a zigzag line, which is a canceling editing symbol, is added to the area once deleted with a check mark, which is an editing symbol, the deleted area is deleted in the handwriting input / output application. Cancel and restore the area.

As shown in Fig. (B), once the edit symbol (A) mark is added and copy and paste is performed, when a zigzag line is added to the arrow part which is the edit symbol that specifies the paste destination, it is handwritten. In the input / output application, copy and paste is performed to return to the state before inserting characters.

FIG. 58 is a diagram showing a sentence in which the result of canceling the editing in the third operation on the medium 2 is reflected in the handwriting input / output application.

<Hierarchical relationship of each operation>
The hierarchical relationship of each operation will be described below.

Each operation may be an operation in which the third operation is parallel to the first and second operations.

In this case, at the time of the third operation, the characters of the text and the characters of the comment are edited at the same time.

On the other hand, the third operation may be a sub-operation subordinate to the first operation and the second operation, respectively.

In this case, only the characters in the text are edited during the third operation as a sub-operation subordinate to the first operation, and the characters of the comment during the third operation as a sub-operation subordinate to the second operation. Edit only.

As a matter of course, the handwritten input / output system 1 according to the present invention can be modified and implemented as being capable of executing various operations as needed, in addition to the above-mentioned first to third operations.

<Save to computer>
As shown in FIG. 59, by rotating the image pickup unit 7 clockwise at a predetermined rotation angle (example: 10 degrees or more) while touching the image pickup unit 7 on the medium surface, the processing instruction unit 10 outputs the input characters and the result of the document structure (file). The information processing unit 11 is instructed to save the content change result). When the main body of the imaging unit 7 has a function of recording the editing history, the result of the above document structure is transmitted to the information processing unit 11.

As shown in FIG. 60, by rotating the image pickup unit 7 counterclockwise at a predetermined rotation angle (example: 10 degrees or more) while touching the image pickup unit 7 on the medium surface, the processing instruction unit 10 discards the input characters and the result of the document structure. The information processing unit 11 is instructed to do (cancel the change of the file contents).

To recognize the rotation of the imaging unit 7, the above-mentioned method of reading the rotation angle and recognizing the change in the rotation angle is used.

<Starting / Exiting Application>
As shown in FIG. 61, the image pickup unit 7 is rotated clockwise at a predetermined rotation angle (example: 10 degrees or more) while being touched on the medium surface, and the predetermined rotation angle (example: within 1 second) is maintained as it is. Example: 10 degrees or more) By rotating counterclockwise, the processing instruction unit 10 instructs the information processing unit 11 to start the handwritten input / output application.

As shown in FIG. 62, the image pickup unit 7 is rotated counterclockwise at a predetermined rotation angle (example: 10 degrees or more) while being touched on the medium surface, and the predetermined rotation angle (example: within 1 second) is maintained as it is within a predetermined time (example: within 1 second). Example: By rotating clockwise (10 degrees or more), the processing instruction unit 10 instructs the information processing unit 11 to terminate the handwriting input / output application.

<Operation switching operation-Method 1>
The first method for switching operations will be described below with reference to FIG. 63.

In this method, the above-mentioned method of angle measurement and angle change recognition is used.

As shown in FIG. 6A, the image pickup unit 7 is placed on the medium surface, and the image pickup unit 7 is placed on the left side of the medium surface by a predetermined angle (example: 15 degrees or more) with reference to the angle formed by the medium surface and the image pickup unit 7. By defeating it, the processing command unit 10 instructs the information processing unit 11 to start the first operation of inputting the text by hand.

As shown in FIG. 3B, the image pickup unit 7 is erected on the medium surface, and the image pickup unit 7 is placed on the right side of the medium surface by a predetermined angle (example: 15 degrees or more) with reference to the angle formed by the medium surface and the image pickup unit 7. By defeating it, the processing instruction unit 10 instructs the information processing unit 11 to start a second operation of inputting a comment by hand.

As shown in FIG. 3C, the image pickup unit 7 is erected on the medium surface, and the image pickup unit 7 is placed on the upper side of the medium surface by a predetermined angle (example: 15 degrees or more) with reference to the angle formed by the medium surface and the image pickup unit 7. By defeating it, the processing instruction unit 10 instructs the information processing unit 11 to start a third operation for editing the text and the comment.

As shown in FIG. 3D, the image pickup unit 7 is erected on the medium surface, and the lower side of the medium surface is set by a predetermined angle (example: 15 degrees or more) with reference to the angle formed by the medium surface and the image pickup unit 7. The processing instruction unit 10 instructs the information processing unit 11 to end the operation currently being executed and start a preset normal operation (at startup).

Note that FIG. 64 is a plan view seen from directly above, which supplements the description of the first method for switching operations.

<Operation switching operation-Method 2>
The second method for switching operations will be described below.

The operation may be switched by an input means such as a button provided on the main body of the imaging unit 7.

In addition to the image pickup unit 7, the operation may be switched by an external input means such as a mouse or a keyboard.

<Operation switching operation-Method 3>
The third method for switching operations will be described below.

In the imaging unit 7 of the embodiment in which a plurality of pen tips can be stored and switched inside, the operation may be switched in conjunction with the pen tip switching operation.

In this case, since it is possible to color-code each operation, it becomes easy to distinguish each operation on the medium.

Further, when an eraser or correction fluid is attached instead of the pen tip, the information processing unit 11 may also erase characters or figures instead of writing.

<Decision / Cancellation>
When the information processing unit 11 determines whether or not to execute a predetermined process, as shown in FIG. 65 (a), after the image pickup unit 7 is vertically erected on the medium surface and the medium surface is touched once. When the medium surface is touched again within a predetermined time (eg, within 1 second), the processing command unit 10 orders the information processing unit 11 to decide to execute the process.

As shown in FIG. 3B, when the image pickup unit 7 is vertically erected on the medium surface, the medium surface is touched once, and then the touch is not made within a predetermined time (eg, within 1 second), a processing instruction is given. The unit 10 orders the information processing unit 11 to stop the execution of the process.

<Other usage>
The above configuration is premised on leaving a writing history on the medium 2, but the handwriting input / output system 1 according to the present invention performs handwriting input while looking at the display 6 without leaving a writing history on the medium 2. As mentioned earlier, it is acceptable.

<Voice recognition>
Although the present invention is a handwriting input / output system, voice recognition and voice guidance may be performed as a method for assisting handwriting input.

<Voice recording section>
As shown in FIG. 66, the audio recording unit 20 is mainly composed of a central processing unit (MPU) 40. That is, under the control of the central processing unit (MPU) 40, the audio information input from the microphone 41 is processed by the A / D converter 43 and the compression circuit 44 via the amplifier 42, and recorded as digital audio information.

<Voice recognition unit>
The voice recognition unit 21 converts the recorded voice information into the corresponding text information, and the processing command unit 10 transmits a processing command based on the text information to the information processing unit 11 together with the text information.

Here, it is possible to improve the accuracy of voice recognition by capturing a dot pattern in which a classification code is defined before and after voice recognition. That is, the accuracy of voice recognition is improved by referring to the text information in the voice recognition information table corresponding to the classification code without referring to all the text information when searching the database and outputting the text information.

For example, if you say "Ueno", it is difficult to know whether it is the place name Ueno or the person's name Ueno, but after the classification code that means "address" is imaged, the voice "Ueno" is spoken. When input as information, the voice information of "Ueno" is transmitted to the information processing unit 11 together with the category code of address input.

<Switching to voice recording mode>
When the handwriting input / output system 1 according to the present invention performs voice recording instead of handwriting input, it is necessary to switch to the voice recording mode by performing a predetermined operation. The method of switching to the voice recording mode will be described below.

<Trace a specific symbol>
As a first method of switching to the voice recording mode, there is a method of switching to the voice recording mode by the imaging unit 7 writing (tracing, drawing) a specific symbol meaning the switching to the voice recording mode. The locus recognition unit 9 recognizes the locus of a specific symbol meaning switching to the voice recording mode and transmits the locus to the voice recording unit 20 (or by transmitting via the processing instruction unit 10) to record the voice. Part 20 starts voice recording.

<Bounce the pen>
As a second method of switching to the voice recording mode, as shown in FIG. 67, there is a method of switching to the voice recording mode by bouncing the image pickup unit 7 in a state of touching the writing area on the medium surface.

<Switching by voice recognition>
As a third method of switching to the voice recording mode, there is a method of recording a predetermined voice in the voice recording unit 20 and switching to the voice recording mode by the voice recognition unit 21 recognizing the predetermined voice.

When the operation by the above-mentioned various methods is performed, the processing instruction unit 10 instructs the information processing unit 11 to perform voice input instead of handwriting input in the handwriting input / output application.

<Audio output section>
The audio output unit 22 reproduces a predetermined audio file in response to an instruction from the processing instruction unit 10. The audio file that has been digitized and stored in advance is output from the speaker 45 or the earphone 46 via the A / D converter 42 and the amplifier 42.

<Audio guide>
In the handwritten input / output system 1 according to the present invention, voice guidance may be provided to a user who performs handwritten input by using code information.

That is, since the voice guide code for instructing the information processing unit 11 to reproduce the voice is defined in the dot pattern printed in the writing area, the handwriting input is started (the imaging unit 7 outputs the dot pattern). (Imagine), the voice output unit 22 reproduces the voice file that is stored in association with the voice guide code in advance.

However, the voice guide code and the corresponding voice file may be used in addition to the voice guide.

<Grid sheet>
68 to 72 are explanatory views showing an embodiment in which a grid sheet, which is an information input auxiliary sheet, is attached to a display in the handwriting input / output system 1 according to the present invention. This grid sheet has a dot pattern formed on a transparent sheet, and exhibits a function as a touch panel by being laminated and pasted on a monitor surface.

By using the grid sheet disclosed in Japanese Patent Application No. 2007-230776 (PCT / JP2008 / 002188) as the medium 2, the medium can be attached on the monitor surface of the display 6 to form a touch panel. However, a means other than the grid sheet may be adopted.

In addition to using the grid sheet as the medium 2, the grid sheet may be used by superimposing it on the medium 2. In such a usage, firstly, when it is desired to edit a document or an image printed (described) on a medium not for handwriting input / output using the handwriting input / output system 1 according to the present invention, secondly, an imaging unit is used. This is particularly effective when the user feels resistance to pressing the 7 onto the monitor surface.

68 to 69 are explanatory views showing an embodiment in which the grid sheet is attached to the display 6 and used.

FIG. 68 is a diagram in which a grid sheet is used in a general-purpose computer system which is an information processing device. In this embodiment, a grid sheet is attached on a display screen of a liquid crystal display (LCD) such as a personal computer or a CRT (cathode ray tube). The grid sheet is made of a transparent film and has a dot pattern printed on it. This dot pattern is a pattern of XY coordinate values and / or code values by a predetermined algorithm, which will be described in detail later. A scanner, which is a dot pattern reading means, is connected to the computer body. The user touches the grid sheet with the scanner according to the instructions on the fluoroscopic screen. The scanner reads the dot pattern and sends it to the personal computer via the USB cable, and the central processing unit (CPU) of the personal computer analyzes the dot pattern to obtain the XY coordinate values on the grid sheet, and further xy on the display. It is converted to a coordinate value, and it is possible to input in a touch panel format from the information of the coordinate value.

According to this, it is possible to input in a touch panel format by this sheet, and it is possible to provide an inexpensive and highly convenient touch panel. In addition, when browsing an Internet site, it is possible to search for and refer to related information even if the link information is not set.

Although a personal computer is used as the information processing device in FIG. 68, the present invention is not limited to this, and the information processing device includes a personal computer, a PDA, a television receiving device, front and rear projectors, a game device, a karaoke device, and a mobile phone. It may be a telephone terminal device, a POS terminal device, an ATM, a KIOSK terminal, a car navigation system, a pachinko machine, a clock, a smartphone, or the like. In these, a grid sheet is arranged on a display or a display screen of a screen as a touch panel type input device.

FIG. 69 is a cross-sectional view showing the internal structure of the grid sheet described with reference to FIG. 68.

FIG. 3A is a vertical cross-sectional view showing a state in which the grid sheet is touched by the scanner.

As shown in the figure, the grid sheet has a structure in which an infrared reflective layer, a dot pattern layer, and a protective transparent sheet (protective layer) are laminated from the display device side.

The infrared reflective layer has a structure in which an infrared reflective material is vapor-deposited on a transparent sheet for vapor deposition made of a material that transmits visible light, such as vinyl, envelope, polypropylene, and the like. The infrared reflective layer reflects infrared rays emitted from the infrared irradiation means of the scanner and transmitted through the protective transparent sheet to the scanner, and also transmits visible light. At the same time, it blocks infrared rays from the display device. As a result, only the infrared light emitted from the infrared irradiation means of the scanner can be used as the irradiation light, only bright and clear dot patterns can be photographed, and the dot code can be analyzed accurately.

On the dot pattern layer, a dot pattern in which dots made of an infrared absorbing characteristic material such as carbon ink are arranged according to a predetermined rule as described later is printed.

The protective transparent sheet is made of a material that transmits visible and infrared rays, such as vinyl, envelope, polypropylene, and the like. When the dot pattern is repeatedly touched by the scanner, the dots are worn out, which causes a problem that the dot pattern cannot be read accurately. Therefore, by providing a protective transparent sheet, it is possible to prevent the dots from being worn and soiled and to use the sheet for a long period of time.

The scanner contains an IR-LED, which is an infrared irradiation means, an IR filter that cuts a predetermined wavelength component of the reflected light, a C-MOS sensor, which is an image sensor, and a lens. The scanner captures the reflected light of the irradiation light applied to the grid sheet. As described above, since the dot pattern is printed with ink having a characteristic of absorbing infrared rays, only the dot portion is photographed in black in the image captured by the C-MOS sensor.

In this case, the reflective layer mirror-reflected by the infrared reflective layer is photographed in a state where the center is black as shown in FIG. It ends up. Therefore, the dot pattern cannot be photographed without omission. Therefore, it is necessary to diffusely reflect infrared rays and make them enter the lens. That is, as shown in FIG. 70A, an infrared diffusing layer is provided between the dot pattern printing layer and the infrared reflecting layer. The infrared diffusing layer is made of a transparent or translucent material. As a result, as shown in FIG. 3B, the infrared rays emitted from the IR-LED are mirror-reflected by the infrared reflecting layer and diffused by the infrared diffusing layer, and the reflected light in the entire photographing region is incident on the lens.

Although the grid sheet is provided with the infrared diffusing layer in the figure, the present invention is not limited to this, and a filter made of an infrared diffusing material may be provided at the photographing port of the scanner.

FIG. 71 is a cross-sectional view showing various structures of the grid sheet.

FIG. 3A is a grid sheet composed of an infrared reflecting layer having a property of reflecting infrared rays and transmitting visible light, and a dot pattern layer. As shown in the figure, the infrared rays emitted from the infrared irradiation means are first absorbed in the portion of the dot pattern layer where the dots are formed (hereinafter referred to as the dot portion), and then transmitted in the other regions. Next, the transmitted infrared rays are specularly reflected by the infrared reflecting layer, and are transmitted through the dot pattern layer except for the dot portion.

FIG. 3B is a grid sheet provided with an infrared diffusing layer that transmits visible light and diffuses the infrared rays between the infrared reflecting layer and the dot pattern layer. As shown in the figure, the infrared rays emitted from the infrared irradiation means are first absorbed by the dot portion of the dot pattern layer and then transmitted through the other regions. Next, the transmitted infrared rays are diffused by the infrared diffusing layer, then mirror-reflected by the infrared reflecting layer, diffused again by the infrared diffusing layer, and transmitted through the dot pattern layer except for the dot portion.

FIG. 3C is a grid sheet provided with an infrared diffusing layer that transmits visible light and diffuses infrared light on one surface side of the dot pattern layer, that is, on the opposite side of the infrared reflecting layer. As shown in the figure, the infrared rays emitted from the infrared irradiation means are first diffused by the infrared diffusing layer, then absorbed by the dot portion of the dot pattern layer, and then transmitted in other regions. Next, the transmitted infrared rays are specularly reflected by the infrared reflecting layer, and again pass through the dot pattern layer except for the dot portion and diffused by the infrared diffusing layer.

FIG. 3D is a grid sheet composed of an infrared diffuse reflection layer having a characteristic of diffuse-reflecting infrared rays from one surface side and transmitting visible light, and a dot pattern layer. As shown in the figure, the infrared rays emitted from the infrared irradiation means are first absorbed by the dot portion of the dot pattern layer and then transmitted through the other regions. Next, the transmitted infrared rays are diffusely reflected by the infrared diffuse reflection layer, and are transmitted through the dot pattern layer except for the dot portion.

FIG. 3E is a grid sheet provided with a protective layer having a property of transmitting infrared light and visible light on the outer surface of the grid sheet shown in FIG. 6A on the dot pattern layer side. By providing such a protective layer, it is possible to prevent the dots from being worn and soiled and to use the sheet for a long period of time.

The protective layer may be provided not only on the outer surface of the grid sheet shown in FIGS. (A) but also on the outer surface of the grid sheet shown in (b) to (d).

FIG. 3F is a grid sheet shown in FIG. 3A having a characteristic of reflecting infrared rays from the side opposite to the dot pattern layer. By having such a characteristic, it is possible to block infrared rays from the display screen or screen on which the grid sheet of the present invention is attached, and to use only infrared light emitted from the infrared irradiation means as irradiation light. Therefore, only bright and clear dot patterns can be photographed, and the dot code can be analyzed accurately.

In addition to the figure (a), the grid sheets shown in (b) to (e) may also have a property of reflecting infrared rays from the side opposite to the dot pattern layer (other surface side). ..

FIG. 3G is a grid sheet shown in FIG. 3A in which an adhesive layer is provided on the anti-dot pattern layer side of the infrared reflecting layer. The adhesive layer is made of a removable material. The adhesive layer is necessary only when the grid sheet is used by being attached to a display device or a medium, and is not necessary when the grid sheet is used by placing it on a printed matter or sandwiching it as described later. By providing such an adhesive layer, the grid sheet can be easily attached to a display or the like.

The adhesive layer may be provided not only in the figure (a) but also on the anti-dot pattern layer side of the infrared reflective layer of the grid sheets shown in (b) to (e).

FIG. 72 shows another embodiment of the grid sheet. The present embodiment is characterized in that dot pattern layers are provided on both sides of the sheet.

FIG. 3A shows an infrared reflecting layer having a property of reflecting infrared rays to each surface in each direction and transmitting visible light, and a dot pattern layer provided on both sides of the infrared reflecting layer. It is a grid sheet. As a result, the sheet can be used on both sides, which improves convenience.

FIG. 3B shows a grid sheet in which an infrared diffusing layer that transmits visible light and diffuses infrared light from each direction is provided between the infrared reflecting layer and the dot pattern layer on each surface. Is.

FIG. 3C is a grid sheet in which an infrared reflecting layer and an infrared diffusing layer are further provided on the outer surface of the dot pattern layer on each surface.

FIG. 3D is a grid sheet composed of an infrared diffuse reflection layer and dot pattern layers provided on both sides of the infrared diffuse reflection layer.

FIG. 3E is an information input auxiliary sheet provided with protective layers having a property of transmitting infrared light and visible light from each direction on both outer surfaces of the grid sheet shown in FIG. is there.

The protective layer may be provided not only on the outer surface of the grid sheet shown in FIGS. (A) but also on both outer surfaces of the grid sheets shown in (b) to (d).

FIG. 3F is a grid sheet in which adhesive layers are provided on the anti-dot pattern layer side of each infrared reflective layer, that is, on both outer surfaces of the grid sheet in the grid sheet shown in FIG. is there.

The adhesive layer may be provided not only on the outer surface of the grid sheet shown in FIGS. (A) but also on both outer surfaces of the grid sheets shown in (b) to (e).

Since the description of each layer and the description of reflection in the grid sheets of FIGS. (A) to (f) are the same as those described above, the description thereof will be omitted here.
In the grid sheet shown in the figure, different coordinate values and / or code values may be patterned in the dot pattern layer on one surface side and the other surface side of the infrared reflecting layer or the infrared diffuse reflecting layer.

Further, in the grid sheet shown in FIGS. 71 to 72, text, illustrations, photographs and the like may be superimposed and printed on the dot pattern layer or other layers with ink made of an infrared transmitting material or an infrared reflecting material.

<Issues related to grid sheets>
As described above, if the grid sheet is not provided with the infrared diffuse reflection layer, specular reflection is performed on the sheet surface, so that a region where infrared reflected light is not incident is created around the center of the image sensor, and FIG. 69 (b) shows. An eyeball phenomenon occurs in which the black eye is located in the center. Therefore, it is necessary to provide a diffuse reflection layer, which increases the cost.

<Problem solving means>
Therefore, in the case of a grid sheet without a diffuse reflection layer, a method of forming a dot pattern with a diffuse reflection material (diffuse reflection ink) is used. According to this method, as shown in FIG. 73 (a), the dots formed of the diffuse reflection material diffusely reflect infrared rays in all directions, so that the infrared rays that enter the dots are always incident on the image sensor. .. Therefore, as shown in FIG. 73B, the dot pattern is detected as white dots in the black eyeball region in the captured image.

<How to distinguish between white dots and black dots>
An algorithm capable of discriminating conventional black dots printed on a normal medium surface and recognizing the above-mentioned white dots printed in the form of a grid sheet without a diffuse reflection layer will be described.

（数式）(i、j)＝255−BL(i、j)
ただ、この方法では、全てのピクセルにおいて白黒反転計算を行う必要があり、余計な計算時間を要する問題がある。

(Formula) (i, j) = 255−BL (i, j)
However, in this method, it is necessary to perform black-and-white inversion calculation for all pixels, and there is a problem that extra calculation time is required.

Therefore, regardless of this method, a method of changing the threshold value for black dots and discriminating by an algorithm for analyzing white dots in the eyeball region will be specifically described.

First, the analysis algorithm for black dot discrimination will be described with reference to FIG. 74 (a). This algorithm is a scanline algorithm performed line by line, and finds the difference in brightness between pixels that are separated by a predetermined interval from the left end to the right, and whether the difference exceeds or falls below a predetermined positive / negative threshold value. Is determined, and when the value falls below the negative threshold value, the pixel on the right side is determined to be a dot, and it is determined that the dot starts from that pixel. Further, when the value exceeds the positive threshold value, it is also determined that the pixel on the right side is not a dot, and it is determined that the dot ends up to the pixel before that. However, when the dots are determined by the absolute amount threshold value, the image captured by tilting the scanner causes brightness and darkness, and the brightness of the dark portion is significantly reduced, so that the dots in the dark region cannot be determined. However, as described above, the dots can be accurately determined even in the image in the dark region by using the threshold value for determining the difference in brightness between the two points.

Here, the reason for using the difference between pixels separated by a predetermined interval instead of the difference in brightness between adjacent pixels is that there is little change in brightness around the outline in an image captured with a blurred outline of dots. This is because if the pixels for which the difference is taken are too close to each other, the difference is small and the value does not exceed the threshold value, making it impossible to discriminate the dots.

For example, if this predetermined interval is "4" and the difference between the pixel number i and the pixel number i + 4 on the jth line is ΔBL (i, j), the calculation method is as follows.
(Formula) ΔBL (i, j) = BL (i + 4, j) −BL (i, j)
The threshold value used is φ (φ> 0). When the brightness is expressed by 8 bits (bright level 0 to 255), φ = 25 to 50 is appropriate in the bright area, and φ = 8 to 16 is appropriate in the dark area. In this way, by dynamically changing the threshold values of the dark region with less light and darkness and the bright region with large brightness and darkness of the dot portion, the dots can be determined more accurately.
Here, if ΔBL (i, j) is ΔBL (i, j) <−φ with respect to the threshold value φ, it is determined that the dot starts from the pixel at the position i + 4. If ΔBL (i, j)> φ, it is judged that the dot ends at the pixel at the position of i + 3.

In the analysis algorithm of the position of the white dot, this threshold value determination is reversed.

That is, if ΔBL (i, j) has the same threshold value φ and ΔBL (i, j)> φ, it is determined that the dot starts from the pixel at the position i + 4. If ΔBL (i, j) <−φ, it is judged that the dot ends at the pixel at the position of i + 3.

By using this method, it is possible to easily determine the position of the white dot without inverting the black and white in all the pixels.

<Shape of pen scanner that enlarges the eyeball>
When scanning white dots with a scanner using a grid sheet, the size of the black eyes must be increased to widen the detection range of the dot pattern.
Therefore, the shape of the scanner so that the black eyes become larger will be described.

As shown in FIG. 75, assuming an optical path in which infrared rays emitted from the inside of the mouth edge of the photographing opening are specularly reflected on the grid sheet and can enter the image sensor, a light guide is provided outside this optical path. Infrared rays are uniformly applied to the entire surface of the medium. As a result, the infrared rays emitted from the light guide are specularly reflected on the grid sheet, and many areas of the captured image can be black eyes. At this time, it is necessary to provide a light-shielding portion at a position shown in the figure so that excess infrared rays are specularly reflected on the grid sheet and do not enter the image sensor. Due to the structure, when the scanner is tilted, the black eye area is reduced, but the shooting area is expanded and the number of white dots to be imaged increases, so that the dot code can be analyzed without any problem.

<Method of determining whether or not it is in contact with the medium>
If the code on the medium surface is constantly read while the scanner is in contact with the medium surface, the code is different from the user's intention, for example, when the user unintentionally shifts the reading position slightly. Reads and plays different content. If you want to eliminate such a situation, you can set it to ignore even if you read a code different from the first time. However, when tracing on the medium surface with a scanner as in handwriting input, it is necessary to input a code (coordinate value) that is read one after another. Therefore, the error should be ignored, and the next time a new coordinate value is read, it should be treated as a normal input. Furthermore, if a read error occurs due to a slight movement of the scanner, and if the error occurs once, the same code read next is input as the code read the first time, the same content is input. The phenomenon of starting many times (playback such as "Hello, Hello, Hello ..." occurs) occurs, and proper operation cannot be performed.

Therefore, it is determined whether or not the scanner is in contact with the medium surface, and unless the scanner is separated from the medium surface, the same code to be read next is ignored (different codes may be ignored), and the scanner is placed on the medium surface. It is necessary to continue the execution of the content or instruction corresponding to the code read the first time in contact with. Of course, if the scanner is separated from the medium surface and then touches the medium surface again, the content or instruction that is being continuously executed is terminated, and the content corresponding to the newly read code (same code or different code) is used. It goes without saying that the execution of the command and the command is started. If the code cannot be read even if it comes into contact with the medium surface, the execution is continued.

When using this method, if only the black dots arranged on the medium surface are targeted, the brightness of the central part of the scanned image is simply measured, and if it is bright, the pen is in contact with the medium surface. For example, it can be done only by determining that the pen is not in contact with the medium surface. The reason for measuring only the brightness in the central part is that even if the scanner is tilted in the central part, the brightness does not decrease so much that the judgment is not affected, and the measurement time is minimized by setting the measurement area only in the central part. This is because it can be suppressed.

However, in a scanner that can read black dots and also white dots, when reading white dots, it is determined that the pen is not in contact with the medium surface because the central portion of the scanned image is dark.

Therefore, FIG. 76 shows a method for determining whether or not a scanner that can read black dots and white dots is in contact with either a normal medium surface (printing medium surface or the like) and a grid sheet surface. Will be described using.

First, an image is acquired by the image sensor (S10).

Whether or not the acquired image center is bright is measured (S20), and if the center is bright, it is determined that the scanner is normally in contact with the medium surface (S25). The measurement of whether or not the central part of the image is bright is determined by whether or not the brightness of the central part of the image exceeds a predetermined threshold value.

If it is not determined that the central part is bright, a white dot is searched for in the image (S30). When a white dot is detected, it is determined that the white dot is in contact with the grid sheet (S35).

If it is not determined that there are white dots, it is determined that the scanner has not touched the medium surface (S40).

After the determination of S25, S35, S40, the analysis ends.

In order to search for white dots from the dark region in S30, as shown in FIG. 77, it is determined in a predetermined region (always a predetermined amount) whether the peripheral pixels are white dots in a spiral shape from the central pixel of the captured image. Judgment is made based on the minimum area where the dots exist.

In the above, when it is determined in S25 that it is in contact with the normal medium surface, the analysis of black dots (conventional algorithm) is started. When it is determined in S35 that the grid sheet is in contact with the grid sheet, the analysis of white dots (new algorithm) is started.

<Matters newly added due to priority claim>
<Afterimage characteristics of the image sensor>
Incidentally, in the charge of the element shown in FIG. 17 (c), in addition shown in FIG. (C), may be one way disappear afterimage of the imaging element 12 in a predetermined time period t _k is more gradual.

<About the shape of the writing area>
As far as the above description is concerned, the writing area is a rectangle, but when the writing area is specified by using the code information as in the present invention, the shape of the writing area is, for example, a polygon as shown in FIG. 78. , Circular, oval, heart-shaped, star-shaped, or when superimposing a dot pattern on an illustration, the same analysis efficiency is exhibited regardless of the shape along the actual line of the illustration. be able to.

Conventionally, when trying to specify the writing area using only the coordinate information, the coordinates indicating the boundary of the writing area are defined in advance in the table or function, and the input coordinate values fall into the category of the table or function. I had to judge sequentially. For this reason, the analysis becomes complicated, and when the writing area is specified using only the coordinate information, the writing area needs to be rectangular in order to minimize the decrease in analysis efficiency such as the memory used and the calculation speed. In the case of a rectangle, the minimum and maximum values of the XY coordinate values in the rectangular area are stored in the table, and it is determined whether the input coordinate values are within that range. Therefore, even if it is a rectangle, an extra comparative calculation for determining whether the memory used is four times or more and within the writing area is required four times for specifying the writing area by the code information. Further, when there are a plurality of write areas, it is necessary to determine which area it is, and it is predicted that a huge amount of memory used and calculation speed will be required. In the present invention, since the code information directly specifies the write area regardless of the shape of the write area, the write area can be specified without using extra memory or performing calculation.

<Discrimination by combination of placement directions>
Another method for performing automatic operation determination will be described.

When using a dot pattern that defines information by arranging information dots with a distance and direction from the reference position, such as GRID1 and GRID3 (first digitized information), the information dots are placed on the + direction line (1st digitized information). Information can also be defined for the combination of the arrangement directions of the information dots themselves, whether they are arranged on the vertical and horizontal direction lines (on the vertical and horizontal direction lines) or on the × direction line (on the diagonal direction line) (second digitized information). Therefore, when the case where the information dots are arranged on the × direction line is “0” and the case where the information dots are arranged on the + direction line is “1”, the second numerical value in the dot pattern of FIG. 79 (b) is set. The conversion information is "1010010110100101", and the second numerical information in the dot pattern of FIG. 80 (b) is "000000001111111".

In addition to the placement direction of the information dots, the distance (placement distance) at which the information dots are placed from the virtual reference point is set to two or more patterns, so that the distance at which the information dots are placed with respect to the virtual reference point is determined. The second quantification information can also be defined by the combination of.

Of course, the second quantified information may be defined by the combination of both the arrangement direction and the arrangement distance of the information dots.

That is, the second digitized information means information defined by the information dot arrangement law itself, such as the information dot arrangement direction and the arrangement distance.

The inventor has proposed a dot pattern format that defines both code information and coordinate information, but the dot pattern of such a format increases the distribution area of the dot pattern in order to increase the number of information dots. Along with this, the optical reading unit of the pen scanner also needed to have a large reading area for reading the dot pattern.

For this reason, it was difficult to use a format that defines both code information and coordinate information in toys and the like equipped with an optical reading unit that has less reading accuracy, calculation speed, and calculation procedure. However, according to the above method, information dots can be used. It is possible to define a large amount of information even with a small dot pattern with a small distribution area.

The handwritten input / output system according to the present invention firstly feels like writing on a paper medium by a person who has not mastered the techniques for using existing input devices such as a mouse, keyboard, and tablet used for operating a computer. Has industrial applicability as an input device that can be used in. In addition, it has industrial applicability as toys, especially educational toys, mainly for children.

Secondly, the handwritten input / output system according to the present invention can identify the writing area by the code information read at the same time as the coordinate information. Therefore, when the handwritten input content belongs to a predetermined category, the dictionary of that category The recognition rate is significantly improved by collation with.

Thirdly, the handwriting input / output system according to the present invention determines whether or not the writing area is a writing area only by reading the dot pattern without any special operation by the operator, and a writing instrument that everyone is familiar with. You can intuitively realize handwriting input as you use it.

1 Handwriting input / output system 2 Medium 3 Scanner 4 Computer 5 Server 6 Display 7 Imaging unit 8 Dot pattern analysis unit 9 Trajectory recognition unit 10 Processing command unit 11 Information processing unit 12 Imaging element 13 Irradiation unit 14 Optical control unit 15 Time measurement unit 16 Angle Measuring unit 17 Angle change recognition unit 18 Rotation angle reading unit 19 Rotation angle change recognition unit 20 Voice recording unit 21 Voice recognition unit 22 Voice output unit 23 Transmission unit 24 Transmission button 25 Retransmission button 26 Microphone 27 Voice playback button 28 Instruction protrusion Part 29 Pen 30 Pen retract button 31 Pen type scanner 32 Light guide 33 Lens 34 LED
35 C-MOS sensor 36 PCB
37 Pen 38 Bank Guide 39 Braille 40 Central Processing Unit (MPU)
41 Microphone 42 Amplifier 43 A / D converter 44 Compression circuit 45 Speaker 46 Earphone

Claims (1)

A handwriting input sheet comprising a medium having at least a portion of the writing area in which the dot pattern defined coordinate information and code information is formed in the region,
The coordinate information of the dot pattern is defined in the dot pattern coordinate system, which is the coordinate system of the dot pattern.
The dot pattern is read by the imaging analysis means, at least a portion of said code information, write flag for determining whether or not the writing area is provided, et al is,
When it is determined that it is the writing area, the locus recognition means built in or connected to the imaging analysis means executes a predetermined process of handwriting input based on the coordinate information and the code information.
If it is determined that the area is not the writing area, the code information captured by the imaging analysis means and / or the information associated with the coordinate information is output and / or instruction processing is executed.
Handwriting input sheet.

Images