JPH06289983A - Handwritten input information processor - Google Patents

Abstract

PURPOSE:To deal with vertical/horizontal writing and Japanese words by specifying an editing proofreading symbol which is the most similar to a handwriting form without regard to the directional habit of the handwriting form. CONSTITUTION:A tablet controller 58 controls a tablet provided on a display and tablet block 59 as a coordinate input means generating coordinate data corresponding to handwriting by a stylus pen 60. CPU 51 is provided with functions as a symbol discriminating means recognizing the handwriting form based on coordinate data generated by the coordinate input means and specifying the editing proofreading symbol which is the most similar to the recognized handwriting form from among plural editing proofreading symbol previously defined within a system ROM 52 without regard to the directional habit of the handwriting form and an editing proofreading means executing an editing proofreading processing to an editing proofreading object in correspondence to a function assigned to the specified editing proofreading symbol. Thereby, this processor can handle vertical/horizontal handwriting and Japanese words.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a handwriting input information processing apparatus for recognizing and processing characters, figures, etc. by handwriting input from a pen or the like.

[0002]

2. Description of the Related Art Conventionally, when a document or figure is edited by a computer, for example, a menu is displayed on a display, a desired menu is selected from the menu by using a cursor, and a desired character, phrase, sentence Alternatively, there is a method in which a graphic or the like is selected and an editing process is performed. Specifically, there is a system in which a file cabinet, a document receiving tray, or the like is displayed on the display, and this is selected by the coordinate input device.

However, in the above-mentioned method, since the cursor is frequently moved up, down, left, and right to perform the editing, the fixation point is likely to move, the operator's thinking is likely to be disturbed, and the work efficiency is lowered.

For this reason, recently, a computer operated by a pen (stylus pen) has been announced instead of the conventional keyboard and mouse.

This pen-operated computer can be operated by simply touching the menu displayed on the liquid crystal screen provided with a sensor with the stylus pen.
It is possible to input with one hand, easy for beginners to operate,
Further, it is characterized in that it is convenient for, for example, recalling an already created document or figure on the screen for editing.

Here, in the above-mentioned editing, by drawing a command (editing proof symbol), which is a so-called gesture, on the screen with a stylus pen, a process such as inserting or deleting a document is performed.

As the gesture, for example, the one shown in Table (1) of Table 1 is used.

[0008]

[Table 1]

[0009]

By the way, in the conventional computer (handwriting input information processing device) for pen input as described above, a gesture corresponding to horizontal character input such as Western characters is basically used. The problem is that the Japanese language support is not sufficient.

That is, as the gesture, the same shape as that included in the Japanese character / symbol group is used for different meanings. For example, among the gesture functions shown in Table (1) of Table 1, the gesture showing the function of "line feed" is
The same shape is used as the end of the designated character string in Japanese (": angle brackets).

Also, Japanese can use both vertical writing and horizontal writing, but since the above-mentioned conventional gesture is developed for horizontal writing only, it is not accurately specified for vertical writing.

Therefore, an object of the present invention is to provide a handwriting input information processing apparatus which can handle both vertical writing and horizontal writing and Japanese and has improved operability.

[0013]

DISCLOSURE OF THE INVENTION The handwriting input information processing apparatus of the present invention has been proposed in order to achieve the above-mentioned object, and is a display means for displaying an editing and proofreading object and coordinate data according to handwriting. Recognizing the coordinate input means for generating, and the coordinate data generated by the coordinate input means based on the edit calibration symbol most similar to the recognized handwriting shape from among the plurality of predefined edit calibration symbols. , A symbol discriminating means for specifying regardless of the directionality of the handwriting shape, and an edit and proofreading process for the editing and proofreading object according to the function assigned to the editing and proofreading symbol specified by the symbol discriminating means. And a calibration processing means.

Here, the function assigned to the editing proof symbol identified by the symbol discriminating means is interpreted,
In the case of a function that changes according to the attribute of the edit proof object adjacent to the handwriting shape entry position, there is a function interpreting unit that uniquely interprets the function according to the direction of the handwriting shape with respect to the edit proof object. Then, the edit / proofreading processing means performs the edit / proofreading processing corresponding to the function interpreted by the function interpreting means on the edit / proofreading target.

When the edit proofreading object is a character, the edit proofreading symbol specifies a range from a start point to an end point for converting the edit proofreading object into kana-kanji and a kana-kanji conversion instruction in a continuous handwriting. Kana to Kanji conversion symbol (Kana to Kanji conversion gesture).

At this time, the kana-kanji conversion gesture may have a left-handed ending shape, a right-handed ending shape, a left-handed starting shape, or a right-handed starting shape.

Further, the edit proof symbol cancels the effect of the edit proof processing command (gesture command) and cancels the input of the conversion gesture and / or the coordinate data to instruct the conversion to the coordinate data, and the conversion to the gesture command. It is a conversion gesture to instruct.

Further, the edit proof symbol may be a lead line symbol (lead line gesture) for setting a parameter according to the attribute of the edit proof and an input end gesture for instructing the end of input.

In addition, the handwriting input information processing apparatus of the present invention holds information corresponding to the edited proofreading symbol corresponding to the executed editing proofreading process, and displays the information according to the edited proofreading symbol on the display means. In addition, it has the function of modifying the function of the previously entered edit calibration symbol with the edit calibration symbol entered on the display of information according to the edit calibration symbol, and further When collectively performing the edit proofreading processing corresponding to each function of the proofreading symbol, it is possible to have a function of immediately executing the function of the particular edit proofreading symbol for the particular edit proofing symbol.

[0020]

According to the handwriting input information processing apparatus of the present invention, the symbol discriminating means specifies the editing proof symbol most similar to the recognized handwriting shape regardless of the directionality of the handwriting shape. It will be possible to handle both writing and horizontal writing.

[0021]

Embodiments of the handwriting input information processing apparatus of the present invention will be described below with reference to the drawings.

As shown in FIG. 1, the handwriting input information processing apparatus of the present embodiment has a display / tablet block 59 having a liquid crystal display unit as a display means for displaying an object to be edited and calibrated such as characters and figures, and a stylus. A tablet provided on the display / tablet block 59 as a coordinate input means for generating coordinate data according to the handwriting by the pen 60, a tablet controller 85 for controlling the tablet, and coordinate data generated by the coordinate input means. For example, a gesture most similar to the handwriting shape recognized from the plurality of editing calibration symbols (hereinafter referred to as gestures) defined in advance in the system ROM (read only memory) 52 is recognized. Symbol identifying means for identifying regardless of the directionality of handwriting shape and the symbol And a CPU (central processing unit) 51 having a function as an editing and proofreading processing means for performing an editing and proofreading process on the editing and proofreading object according to a function assigned to a gesture specified by another means. is there.

In other words, the CPU 51 of the handwriting input information processing apparatus according to the present embodiment interprets the function assigned to the gesture specified by the symbol identifying means as well as the function as the symbol identifying means, and the handwriting shape is obtained. In the case of a function that changes according to the attribute of the editing proofreading object adjacent to the entry position, the function of the function interpreting means that uniquely interprets the function according to the direction of the handwriting shape with respect to the editing proofing work, It has a function of an editing and proofreading processing means for performing an editing and proofreading processing corresponding to the function interpreted by the function interpreting means on the editing and proofreading object.

That is, in FIG. 1, the apparatus of the present embodiment controls the above-mentioned C which controls each constituent element via the main bus.
The PU 51, the system ROM 52 in which the program of the system and the contents of each table to be described later and the information of the above-mentioned plurality of gestures are stored, and the work RAM 55 in / from which data in the process is written / read out.
A display / tablet block 59 including the liquid crystal (LCD) display section and the doublet, a stylus pen 60 as a handwriting input means, and the CP via the main bus.
A tablet controller 58 connected to the U51 and connected to the display / tablet block 59 to control the tablet, and connected to the CPU 51 via the main bus, and controls the liquid crystal display unit of the display / tablet block 59. An LCD controller 57, a video RAM 56 controlled by the LCD controller 57 and the CPU 51 for writing / reading image display data, a keyboard 54 for receiving a user's character input and editing operation by keystrokes, and the keyboard. The keyboard controller 53 controls 54.

The overall structure (appearance) of the handwriting input information processing apparatus of this embodiment is as shown in FIG. That is, in this FIG. 2, the apparatus of this embodiment has, for example, characters, figures, and the stylus pen 60 (the stylus pen 3 in FIG. 2) on the display / tablet block 59.
Handwriting is input and displayed, and a microcomputer (the CPU 51 and each controller 57,
58, 53, ROM 52, RAM 55, 56, etc.) and performs various operations of this embodiment described later by a program. A liquid crystal display unit of a display / tablet block 59 (1 in FIG. 22) and a user (user, operator). Is displayed above
A display whose coordinate values are measured by pointing an arbitrary position on the liquid crystal display of the tablet block 59. The tablet input device of the tablet block 59, the pen 3 (tablet and stylus pen 60 in FIG. 1), and the keyboard. 2 (keyboard 54 in FIG. 1).

Here, prior to the description of the specific contents of the apparatus of this embodiment, the features and basic idea of the embodiment of the present invention will be described below.

In the embodiment of the present invention, for example, Table (2)-
By using the gestures shown in Table (6) of Table 6,
It is assumed to have the following features. In addition,
Table 2 (2) in Table 2 shows the conventional gesture (Table 1 above).
Gesture) in the present embodiment, which corresponds to the (gesture of). Table 3 (3) shows the basic gestures, and Table 4 (4) shows the character string manipulation gestures.
Table 5 (5) of Table 5 shows gestures corresponding to high-speed operations, and Table 6 (6) of Table 6 shows gestures used in the batch execution type editing described later.

[0028]

[Table 2]

[0029]

[Table 3]

[0030]

[Table 4]

[0031]

[Table 5]

[0032]

[Table 6]

As a feature of this embodiment, first, a gesture shape set corresponding to Japanese is used. That is, in order to support vertical writing in Japanese, a gesture shape that does not make the user aware of the direction of the symbol is selected. For example, the table above that can be used in vertical writing
A table that can be used with gesture shapes such as "blank", "tab", and "line feed" shown in (2) and table (4), and rotated orientations.
(2) Alternatively, the shape of the "range designation" or "insertion" gesture shown in Table (3) is used. In addition, operations that are often used for conversion and correction, such as the gesture shape of “conversion” shown in Table (3) and the gesture shape of “text selection” (“candidate selection”) shown in * of Table (3). We have prepared gesture shapes that can be drawn quickly.

Second, a basic gesture that is easy to remember is used. That is, it is possible to perform one operation with a few basic gestures. For this reason, the meanings of operations are unified so that the same gesture can be used even if the objects of operations are different if the functions have the same meaning. For example, the "Move" gesture shown in Table (3) can be used to move files and figures, and the "Direction" gesture shown in Table (3) can be used to send scrapbook pages and kana-kanji conversion candidates. Can be done.

Thirdly, it is possible to easily perform auxiliary input by using the leader line gesture shown in Table (3). This leader line gesture facilitates setting attributes to objects, calling a candidate selection sheet for Kana-Kanji conversion, inputting commands that cannot be specified by the gesture, and referring to file information.

In this embodiment, approximately all data chunks that can be processed by the CPU, that is, files, bitmaps, coordinate values, key inputs, icons, linear figures, program codes, and other processing are performed. A block of data that can be processed at this time is called an object. The method of selecting a chunk of data is arbitrary, and an object may include another object. For example, characters, character strings, paragraphs, etc. are each objects.

Fourth, the repeated operation can be easily performed. That is, the "execute" gesture shown in Table (5) can be used to repeatedly execute the immediately preceding operation. For example, after setting red and bold, select a character that you have forgotten to set and enter the “input end instruction” gesture shown in table (5) above to make the selected character red and bold. it can. In addition, it is possible to repeatedly execute search, replace, etc. by using the "direction" gesture in Table (3). For example, after designating a search character string, by inputting the gesture of “direction” in Table (3) on the searched character string, it is possible to skip to the next searched character string.

In addition, the gesture-on-gesture function, which will be described later, enables more versatile and efficient operation, and can be extended to a character-simultaneous character-type gesture, which will be described later, using a leader line gesture. Also,
It is also available for batch execution type editing.

Next, the basic idea of the present invention will be described.

In this embodiment, the gesture is defined as a method of transmitting the user's intention to the computer by handwriting. However, since there are innumerable shapes and combinations of handwriting and it is impossible to give meaning to all of them, in reality, a shape such as a symbol that can be drawn with a small number of strokes is used for the gesture.

Among the symbols that can be described with a pen, a free-form curve of arbitrary shape with an arrow at one end (a free-form curve of arbitrary shape with (>) at one end, or an arrow whose long line portion is a free-form curve) There are some shapes that include an amorphous part. In the present embodiment, a symbol having a shape including such an irregular portion is also a kind of gesture.

Further, in the case of designing a gesture, the design of the gesture is a task of assigning a computer interpretation to the shape (symbol) drawn by the pen. However, there are actually some ideas about computer interpretation. For example, it is possible to consider assigning a shape directly to each function of the application, and it is also possible to decompose each function into a slightly finer level of meaning and assign a shape to these basic functions.

In the former case, since the processing of the coherent meaning can be executed by one gesture, high-speed input can be expected if the user gets used to it, but the number of gestures to be remembered is large, and it is difficult to remember the shape from the meaning in association. The problem of occurs. The latter may require some gestures to execute one process and the input amount may be larger than that of the former, but on the other hand, there are few gestures to be remembered.
Further, there is an advantage that the symbols can be remembered by analogy with the symbols that are used everyday.

From these things, in this embodiment, the latter idea is basically adopted, and the improvement of the former is dealt with by incorporating a device for improvement. Actually, basic meanings such as selecting, erasing, moving, setting, direction, etc. are assigned to a shape, and this is used as a gesture set.

Here, when a basic meaning is assigned to a shape, how to interpret the basic meaning and operate the computer becomes a problem. because,
This is because the meaning is basic, and several operations are possible depending on the difference in interpretation, and the operation method of the actual application is determined by it. Therefore, even if there is some guideline for the meaning interpretation, the unity of operation cannot be achieved.

Therefore, in the present embodiment, in order to deal with this problem, an application example for each scene is associated with meaning and action as described below, and a standard operating environment can be realized. . The basic meanings that form the core of the above gesture set are selected based on the following ideas.

First, the computer can be operated in its entirety by using only basic gestures. Secondly, the concept used in desk work is used so that even an unskilled person can fully understand it. Third, the general concept of computer operation is used for comfortable and high-speed operation. Fourth, attention is paid so that the meanings can be easily superimposed (reconstructed).

Therefore, the meanings are classified into two categories, that is, the basic gesture and the high speed gesture, so that the most basic operation of the basic gesture and the comfortable operation of the high speed gesture are realized.

Next, the shape of the gesture is selected based on the following conditions.

As the first condition, the symbols that are usually used on paper can be used as they are. Second
As a condition, all shapes should have the same meaning no matter which direction they are drawn in. As the third condition, prepare a shape that can be drawn quickly with one stroke for each meaning. Fourth
As a condition, a different shape is prepared for a gesture having a shape similar to other symbols and characters. As a fifth condition, a shape that allows gestures to be overlapped with each other is selected.

With these features, it is possible to realize an easy-to-remember operation in which the normally used symbols can be used as they are from the first condition, and to enable a natural operation in which the desired symbol can be used in the desired direction from the second condition. It is possible to create a basic system with a one-stroke gesture that is easy to implement even for skilled users who use shapes that can be drawn quickly from the condition 4 and that avoids misrecognition from the condition 4 It is possible to perform an operation with a high degree of freedom in which the skilled person can select one, and it is possible to set a shape that allows the gesture operation from the fifth condition.

In this embodiment, a plurality of shapes may be assigned to the respective meanings so as to meet the above conditions. In that case, the basic shape is defined as the basic gesture, and the other shapes are classified as the different-shaped gestures as shown in Tables (2) to (6). By categorizing in this way, the user can perform all the basic operations if he / she learns the basic gesture for the time being, and can learn other shapes as necessary and use them conveniently.

Further, as an advanced form of the editing operation using the gesture of this embodiment, a batch execution type in which a number of editing symbols are input in advance and the editing operation is executed at a stroke when an instruction to execute the editing is given. Editing is also possible. By using this batch execution type editing, it is possible to smoothly shift from the conventional proofreading work, and the editing instruction can be expressed in a visually recognizable shape.

In this embodiment, the following operations are performed to deal with the batch execution type editing.

First, both a gesture that is easy to input and a gesture that is easy to understand are prepared, and secondly,
Adopting a leader line for facilitating character parallel writing type in which editing instructions are directly written, and thirdly, writing a gesture on a gesture that has already been written, and adding or changing meaning, gesture-on gesture Introducing the idea called, 4th, introducing the function to display the process input by the method other than the gesture such as by the keyboard after converting it into the shape of the gesture, and 5th, in the batch execution type editing. Even if there is an exception, the idea called immediate execution gesture that is executed immediately after input is introduced.

With these measures, the present embodiment can easily cope with batch execution type editing. Further, it is possible to perform an edit in which an instruction is not executed for the first time with a combination of several gestures without completing the input with one gesture, which is an intermediate state between the batch execution type edit and the sequential execution type edit.

Further, in the present embodiment, the gesture is applied, for example, when a file is specified, a basic setting operation of the computer is performed, a document / graphic editing operation, Japanese character recognition, conversion, and cancellation operation are repeated. Is able to.

The meaning of each gesture in the correspondence between the shape and function of each gesture shown in Tables (2) to (6) will be described below.

First, "help" presents information such as an operating method and auxiliary information. According to this help, an interactive window for presenting the explanation automatically opens, and the user selects a menu in the window to browse desired information. In this case, it is a natural operation for the user to draw a gesture on the object that the user wants to explain, but it is also required that the user can reach the desired description wherever the gesture is drawn. Therefore, it is desirable that the dialogue window provides both an option for presenting the explanation for the object under the handwriting and an option for presenting the desired explanation by the question and answer.

In the "cancel", the operation performed immediately before or the processing of the gesture written under the handwriting is canceled and the state before the operation is returned. When undoing twice in a row, there are two types of actions: undoing the previous operation (multiple undo) and undoing the undo operation itself (toggle undo). The selection method of the two operations will be described later.

Here, this "cancel" operates as follows for each target object. For example, in the case of writing on the gesture, the operation of the gesture described below is canceled (returned to the state before execution). For example, in the case of a window that opens temporarily, such as an attribute input window or a kana-kanji conversion window, first invalidate all the input on the window up to the present, then close the window that was temporarily opened, and finally draw it. Return to the condition before drawing the line.
For windows other than the above, cancels the previous operation on the specified window (returns to the state before execution).

A gesture-on gesture can be used to select the multiple cancellation and the toggle cancellation. When "up feed" is drawn on the cancel gesture, the previous process is canceled as the multiple cancel, and when "down feed" is drawn, the toggle cancel is performed and the canceled process is restored. Also, if "Cancel" is drawn again, all the canceled processes are restored.

"Open" opens the object under the handwriting. This "open" is a function that allows you to operate the target in the object. For example, when you open a file, the editor starts, when you open the application software, the application software starts, and when you open the printer icon, the printer The model, the print waiting status, etc. are displayed, and the input of the print target is prompted.

Here, "open" operates as follows for each target object. For example, in the case of application software icons (such as items in the list included), start the application software. For example, in the case of a file folder icon, the file folder is opened and the contents are displayed. Further, for example, in the case of an icon of a file that can be browsed, an application for browsing the target file is activated (the operation target is the designated file).

In the "position" / "object designation", the designated position is set as the designated position, or the designated object is additionally set as the designated object. Unless the input is limited to position specification operation,
If there is an object in the vicinity, it is regarded as an object designation operation. When an object is specified, it is specified in addition to the object specified up to immediately before. This addition operation continues until some operation is performed on the specified object. Further, when an object which has already been designated is designated, the object is excluded from the designated objects.

Here, when the position is specified, the position itself hit with the pen is usually set. However, if there are special objects near the end points of the figure, line points, etc., these points will be set. It is also possible to make the designated position. When specifying an object, an object existing within the range set from the position hit with the pen is set as the specified object, but if no object exists in the set range, the object specification is not executed. Which one to choose when there are two or more objects depends on the situation.

If the specified object is waiting for the opportunity to proceed to the next step,
It serves as an opportunity to move to the next step. This gesture is called "next feed". The waiting object is yes
When the input of s, no, etc. is required, it is assumed that the value (default value) predetermined by the program is approved.

When an object has a hierarchical structure, it is 1 which level of the object should be designated.
I do not understand it by inputting points. The application may determine which layer of the object is designated.
However, it is necessary to clearly inform the user which layer of the object has been selected. Further, for the object designated by the gesture, the hierarchy of the object designated by the direction designating gesture may be moved up and down.

The gesture operates as follows for each target object. First, for an object waiting to proceed to the next step, it is assumed that a default value has been input to the object, and the waiting state is released. For example, when the dialog window with the default value of yes is the target object, the gesture functions in the same way as when the yes button is pressed, and in the dialog window with the default value of no, it functions as when the no button is pressed. , The search cursor for character string search detects the next matching character string, and the search cursor for character string replacement replaces the character string indicated by the search cursor for the character string replacement and then searches for the next matching character string. In the phrase being converted into Kana-Kanji, the next candidate of the phrase being converted into Kana-Kanji is displayed.

In other objects, if the designated object is not currently in the designated state, the designated object is added to the designated object, and the fact that the designated object is added to the designated object is emphasized by some method. And display (reverse, blinking, color change ...). If the designated object is already in the designated state, the designated object is excluded from the designated objects, and the designated object is returned to the normal display.

Further, when the waiting object is instructed to proceed to the next step, the proceeding direction is the direction just before. For example, reverse playback starts when a still image that has been stopped during reverse playback is released by the gesture. When changing the direction, the gesture of the direction is required.

In the "range designation", 2 from the corner of the handwriting
The direction of 90 degrees divided by the line segment of the book is the designated area. That is, for example, a gesture in the shape of "(that is, a horizontal curly bracket) has the specified area in the lower right direction from the corner. In addition, it is combined with another (") type gesture to specify the rectangular range. Becomes In addition, a pair of vertical bracket-shaped gestures for vertical writing also specifies a rectangular range.

However, it has string-like continuity like a character string,
Moreover, when drawn for an object that is folded row by row, the entire string-shaped section sandwiched between two gestures may be the target of designation. For example, this is the case as shown in FIG. However, ◯ and × in the drawing of FIG. 3 mean characters, and GE in the drawing is a gesture.

In the example of FIG. 3, the range of x in the figure is the target to be designated, as in the case of the Japanese hook and bracket. On the other hand, when it is desired to specify the rectangular range, it may be done as shown in FIG.

In FIG. 4, when there is no pair of parentheses, or in the case of a combination which does not form a rectangle like GE ₁ and GE _{2 in} the drawing of FIG. 4, the length is infinitely long in the opening direction. It becomes the state of the specified area with a certain depth.

That is, as shown in the example of FIG. 5, all the sentences under the parentheses are in the designated state until the end of the document.

However, in the case of FIG. 5, it is very burdensome for the computer to set all the objects below the parenthesis in the state where the first parenthesis is drawn, and a gesture to be paired is input. Sometimes it becomes a big burden to remove an object that is out of range from the specification. Therefore, in such a case, the object does not have to be in the selected state until a pair of parentheses appears or an operation is performed on the designated object.

When the parentheses are drawn in the area designated by the parentheses, there arises a problem of correspondence between the parentheses. In such a case, it is necessary to confirm with the user which parentheses correspond and decide.

Here, the "range designation" operates as follows for each target object. That is, only in the above-described case, two pairs operate as a string-shaped range specifying operation. In other cases, a set of two works as a rectangular range specifying operation. Only one of the gestures operates as a range designating operation of the four half planes. Further, when the designated range as the coordinate is not explicitly obtained, if the object exists within the range designated by the gesture, the object designation operation is performed. It is desirable to judge that an object that falls within the specified range and is out of the specified range is specified if most of the objects are included in the specified range.

Further, the shape of the gesture is completely the same as the shape of the Japanese hook. Therefore, in a situation in which both characters and gestures can be input, the input of the gesture and the input of the hook and bracket cannot be distinguished only by the handwriting shape. Therefore, it is necessary to distinguish between the two depending on the situation, but it is further desired to mount a gesture for switching between the gesture and the coordinate data in order to remedy misrecognition.

In the "direction", the object below the starting point of the handwriting is sent in the direction in which the handwriting was blown. With the gesture, not only the actual direction on the two-dimensional plane but also the abstract direction expressed by words such as up and down and front and back can be specified. The actual movement in the abstract direction designation differs greatly depending on the target object (below the starting point of the handwriting), but basically it means the operation of moving to the next adjacent object with respect to the ordered object. .

FIG. 6 shows the designation of directions. That is, in FIG. 6, N is above, S is below, W is to the left, E is to the right, NE is to the upper right, SE is to the lower right, and N is to the upper left.
W, diagonally lower left is SW. The gesture in the N direction is called "upward feed", the S direction is called "downward feed", the E direction is called "right feed", and the W direction is called "left feed".

When the actual (visible) positional relationship is clear and the positional relationship and the direction assigned to the abstract order are different, the operation of the actual positional relationship is implemented. For example, regarding page feed, the operation to read the right page is the same when referring to the book that opens on the right and the book that opens on the left (therefore, the operation to read the next page). Is the opposite direction. This operation is called "next feed".)

Among the correspondences between the abstract order and the directions, the following are defined. For example, the top and bottom (NS) represent the top and bottom of the hierarchical structure, and are used when specifying grouped objects. The front and back (WE) represent the same level order relationship, and are used for displaying the next candidate of the recognition result. When operating an object that has a vertical relationship but not a front-back relationship, or an object that has a front-back relationship but not a vertical relationship, (front = up) (rear = down) can be assigned to increase the operable directions.

Attention must be paid to the difference between the operation of sending an object and the operation of sending a viewpoint. For example, when a horizontal writing word processor is used, a subsequent portion of the document appears by drawing this gesture in the N direction with respect to the portion corresponding to the paper. However, when searching for a character string, it is unnatural to draw a gesture in the N direction to search for the next character string. This is because the former is an operation to send the paper roll upward, whereas the latter (search for the next character string) is an operation to send the search cursor to the next search character string (below). The latter should draw a gesture in the S direction to send the cursor downwards.

It should be noted that the "next feed" function by the "position / object designation" gesture of hitting the tablet with a pen should be made available in as many situations as possible to avoid user confusion.

Here, the "direction" operates as follows for each target object. That is, in the case of a window treated as paper on the application, scrolling is performed in the designated direction. In the case of an object that has already been designated by object designation, etc.
The object in the upper layer is designated by the N-direction gesture, and the object in the lower layer is designated by the S-direction gesture. For example, in the case of a graphic object, a group that is one level higher (larger) than the one currently specified by the gesture in the N direction is specified, and one group (fine) that is one more lower than that currently specified by the gesture in the S direction is specified. Specify a group. Further, for example, in the case of a document object, a gesture in the N direction designates one character → chapter → word → (bunsetsu →) sentence → paragraph → section → chapter → whole sentence, and a gesture in the S direction makes the whole sentence → chapter → Section → paragraph → sentence →
(Sentence →) Word → Specify one character. Details of the designation of the document object and the group will be described later. Furthermore, in the case of the search cursor (highlighted part such as inversion on the search character string), the gesture in the N direction is used to search the previous candidate,
Gesture in S direction to search next candidate, gesture in W direction to search previous candidate (horizontal writing), next candidate search (vertical writing),
The next candidate search (horizontal writing) and the previous candidate search (vertical writing) are instructed by gestures in the E direction. In the case of an object being converted, the gesture in the W direction indicates the previous candidate, and the gesture in the E direction indicates the next candidate.

The shape of the gesture is very simple, and it may be considered that it is difficult to distinguish it from other description alone. However, it is possible to make a sufficient distinction by detecting information other than the shape such as the speed at which the handwriting is made.

In the "erase", if the area under the handwriting is the designated area, all the areas currently designated are cleared, and if the object under the handwriting is designated, all the currently designated objects are erased. , If the area under the handwriting is not in the specified state, erase the object under the handwriting. Confirmation is requested in the case of a fatal deletion operation. When erasing without specifying an object in advance, the size and outline of the gesture and the size and outline of the nearby object are compared to determine the object to be erased.

Here, the above "erase" operates as follows for each target object. That is, when the target is a gesture, the operation of the target gesture is canceled and the original state is restored. For example, if you draw a gesture (×) that overlaps a gesture (○) that encloses a character that surrounds it, the specified area is returned to the normal state, and you gesture (x) that you erase it by overlapping the leader line. ), The setting window is closed by discarding the settings. If it is a file icon, the file is deleted. Further, in the case of a graphic object, the graphic object is deleted. If it's a string object, delete the string object,
The gap generated by this erasure is reduced.

The "move" gesture moves the currently designated area and object. If not specified, the object below the starting point of the handwriting is moved. The destination is the end point of the handwriting. If a special object exists at the move destination, the move may not be performed. For example, when the file icon is moved over the printer icon, it is interpreted as an instruction to print the moved file, and the movement is not actually performed, and the file icon returns to its original position.

Here, this "move" gesture operates as follows for each target object. That is, when the end point is a special object that represents a specific function, the target object (moving object) is set as a target that exerts the function, and processing corresponding to the function is performed. If the end point is other than a special object that represents a specific function or if the target object cannot be processed by the function, normal movement processing is performed. For example, in the case of a file icon, the file is moved. In this case, if the start point and the end point are located beyond the boundary of the directory, the movement is between directories, and if they are in the same directory or folder, the display position of the file icon is moved. In the case of a graphic object, the target object is moved in parallel by the coordinate difference between the start point and the end point. In the case of a character string object, the target object is inserted in the character string at the end point position, and the original target object is deleted (the character is also packed).

The "lead line" gesture refers to or sets the information set in the currently designated area or object. If not specified, the object below the starting point of the handwriting is targeted. A window for reference and input of information is automatically opened near the end point of handwriting.

At this time, by changing the type of window to be opened depending on the target object, the menu can be conveniently called by the user. For example, it is possible to perform switching so that a conversion candidate list window is called when a line is drawn from a phrase that is being converted into Kana-Kanji and an attribute setting window such as a line type and color is called when a line is drawn from a line figure.

The gesture is also used when the parameters roughly set by the gesture operation are input again in detail. For example, if you draw a line from the movement gesture mark that is still displayed on the screen,
The movement distance, angle, coordinates before movement, coordinate values of the movement destination, etc. are displayed, and the amount of movement can be finely adjusted by changing the values.

Here, this "leading line" gesture operates as follows for each target object. That is, in the case of a gesture, detailed parameters of the gesture operation are referred to and set. In the case of a file icon, information such as the file name, file creation date, creator, size, and application used at the time of creation is presented in the window. Changes such as file name and read / write permission flag are accepted on the window. If several files are specified, a list will be presented. In the case of an application icon, information such as the application name, copyright, purchaser name, and size is displayed in the window. There is a selection branch on the window for the user to make detailed settings. By selecting it, the environment settings of the application can be set. For graphic objects,
Display attribute values such as color and line type in the window. A selection branch for setting an attribute value and a selection branch for other operations exist on the window, and attribute setting and various operations can be performed. In the case of a character string object, attribute values such as color, character type, size, character spacing, and line spacing are displayed in the window. There are choices for setting attribute values and selection branches for other operations on the window, and attribute setting and various operations can be performed. In the case of a phrase being converted into Kana-Kanji, a conversion candidate list window for displaying a conversion candidate list is opened. In the case of a character after character recognition, a character recognition window for displaying a list of recognition candidates is opened.

In the "conversion" gesture, a conversion operation is performed on the currently designated object. If no object is specified, the object under the handwriting will be the target. It is possible to enter a shortened type that combines the shape for conversion at the beginning and end of underlines and sidelines. The conversion operation varies greatly depending on the target object, but the basic meaning is to activate the conversion function and display the first candidate.

Here, the above-mentioned "conversion" gesture operates as follows for each target object. That is, in the case of a gesture, a candidate window for correcting a gesture recognition error is opened. In the case of the file icon, a window for converting the file data format opens. In the case of a figure object, the figure is recognized and the recognition result is displayed. In the case of a character string object, Kana-Kanji conversion is performed. In the case of characters after character recognition, a character recognition window for displaying a list of recognition candidates and single-kanji conversion is opened.

In the "insert" gesture, an object is inserted into the pointed portion of the handwriting (insertion instruction portion). Opens a window for insertion at the position specified by handwriting.

This "insert" gesture operates as follows for each target object. That is, in the case of a graphic object, a window for instructing graphic insertion is opened at the position specified by the handwriting in the direction specified by the handwriting. In this case, it is possible to insert a blank space and a selected figure. In addition, a vertical line-shaped margin is created for an insertion symbol whose vertex is facing downward or upward, and a horizontal-line margin is created for an insertion symbol whose vertex is facing left or right. As for the shape that overlaps the line that creates the margin, either move it without transforming it or transform it so that it is longer by the margin and redisplay it on the margin zone. Is desirable. Furthermore, when the margin width is specified, only that value becomes the margin width.
In the case of a character string object, a character string input window is opened to prompt the user to enter an insertion character string. If the target object is a character string whose range is specified, the character string of the target range is transferred to the window for character string input. After the window is opened, the character string to be inserted is selected and moved to the window, so that the character string already input is inserted.
In the case of a database list, a window for adding a new item opens when you draw an insertion symbol in the direction of the item. Items can be rearranged by selecting existing items and moving them to the window. Also, if you insert an insertion symbol in the data direction, a menu window for inserting new data opens. Data can be rearranged by selecting existing data and moving it to the window.

The "copy" gesture copies the currently designated area and object. If not specified, the object below the starting point of the handwriting is copied. The copy destination is the end point of the handwriting. If a special object exists at the copy destination, the copy may not be performed.

This "copy" gesture operates as follows for each target object. In other words, when the end point indicates a special object that represents a specific function, such as a printer icon, if the specific function modifies the target object, first copy the target object. (The copy destination is at the same position as the target object), the copied object is set as the operation target, and the process corresponding to the specific function is executed. If the end point is a special object that represents a specific function and the specific function does not change the target object, set the target object as the operation target and execute the process corresponding to the specific function. (However, the latter may be treated the same as the former). If the end point is any other object, normal copy processing is performed. For a file icon,
Copy the file. If the start point and the end point are located beyond the boundary of the directory, the copy is beyond the directory, and if they are in the same directory, the copy is inside the directory. When it is necessary to give the copied object a name different from that of the original object, the computer may do this automatically, or the user may be required to enter the name. In the case of a graphic object, a copy is generated at a position where the target object is moved in parallel by the coordinate difference between the start point and the end point. In the case of a character string object, the same character string as the target character string is inserted at the end position.

In the "cut" gesture, the object in the writing portion (or its immediate vicinity) of the handwriting is moved to a special area called a cut buffer. If the specified operation has been performed in advance, move the specified object to the cut buffer. The cut buffer is an object such as a workbook, and is prepared by the OS (basic software) or application software. The object moved to the cut buffer is returned to the application software by an operation called paste. This makes it possible to move efficiently across pages. It can also be cut and used for the purpose of deleting an object. When the cut buffer function is realized by the OS, the application software shares the cut buffer, so that the objects of the application software can be delivered via the media.

This "cut" gesture operates as follows for each target object. For example, if the target is a character string, the target character string is moved to the cut buffer,
Subsequent strings are packed in the place where the target string was.
If the graphic object is the target, the target graphic is moved to the cut buffer. Also, when other objects are targeted, if the cut buffer cannot accept the object, it does not work.

What the cut buffer accepts depends on the system. It may be possible to accept a file, a printer, or all other things from a character string, or to accept only a character string.

In the "copy" gesture, the object at the tip of the key for writing the handwriting (or in the vicinity thereof) is copied to the cut buffer. If the specified operation has been performed in advance, the specified object is copied to the cut buffer. Used in combination with the paste operation for copying to a distant place or continuous copying.

This "copy" gesture operates as follows for each target object. If the target object can be accepted by the cut buffer, the target object is copied to the cut buffer, and if the target object cannot be accepted by the cut buffer, it does not operate.

In the "paste" gesture, the object in the cut buffer is copied to the tip of the key for writing the handwriting. If a character string object exists at the tip of the key and the object in the cut buffer is also a character string, character string insertion is performed. In either case, it does not affect the contents of the cut buffer. If the designated operation has been performed in advance, the designation will be canceled. Combine with cutting and copying operations to move to a remote location,
It is used for copying and continuous copying.

This "paste" gesture operates as follows for each target object. For example, when there is no target, the object in the cut buffer is copied at the tip position of the key. If the target is a gesture, the object in the cut buffer is copied to the tip position of the key (ignoring the gesture specification). When the character string object is the target and the object in the cut buffer is the character string, the character string in the cut buffer is inserted into the target character string. Further, when the character string object is the object and the object in the cut buffer is other than the character string, the object in the cut buffer is copied to the tip position of the key. When other objects are targeted, the object in the cut buffer is copied to the tip position of the arrowhead (However, when the contents of the cut buffer are objects that can be copied to the tip position of the arrowhead, otherwise it works. Without warning).

In the gesture of "input end instruction", the operation such as input and selection currently being performed is ended and the process is executed. It is used for re-executing the immediately preceding process, executing editing during batch execution type editing, and ending input operations for which the timing of termination is unclear, such as detailed settings using leader lines. It is also used to convey the trigger for the end of specification when specifying the process first and then the target object. If the processing is not instructed, the immediately preceding processing is executed for the designated object.

The execution operation immediately after the cancellation operation can be interpreted in two ways, meaning that the cancellation is re-executed and that the instruction that has just been canceled is re-executed. Which interpretation is desired depends on the user and the situation, so it is desirable to ask the user to confirm which one should be executed unless otherwise specified by the user.

This "input end instruction" gesture operates as follows for each target object. For example, in the case of a window that opens temporarily such as an attribute input window or a kana-kanji conversion window, all the input up to the present is valid, the window that was opened temporarily is closed, and if it is an attribute window, you can set attributes. Execute the process. In the case of other windows, when the process and the object are designated, the designated process is performed on the designated object. Further, in the case of a window other than the window that is temporarily opened, such as the attribute input window or the kana-kanji conversion window, if only the object is specified, the process executed immediately before is performed on the specified object. When the type of the specified object is inappropriate for the process executed immediately before, the process executable for the specified object is selected in time series from the executed process group. Similarly, in the case of a window other than the window that is temporarily opened, such as the attribute input window or the kana-kanji conversion window, if neither the process nor the object is specified, the process executed immediately before is executed.
However, it is executed again (same object)
It is only when the operation is meaningful even if both the processing such as enlargement and the object are omitted.

The "blank" gesture inserts a blank character between the character before and the character after the point where the pen is lowered.

This "blank" gesture operates as follows for each target object. For example, if it is written on the character string, a blank character is inserted between the character before and the character after the point where the pen is lowered.

The "tab" gesture inserts a tab character between the character before and the character after the point where the pen is lowered.

This “tab” gesture operates as follows for each target object. For example, if it is written on the character string, a tab character is inserted between the character before and the character after the point where the pen is lowered.

In the "line feed" gesture, the character string is line feed at the part delimited by the handwriting.

This "line feed" gesture operates as follows for each target object. For example, when it is written on a character string, the character string is broken at a position designated by a portion of the handwriting that crosses the character string.

The "connect" gesture connects the character string objects before and after the handwriting.

This "connect" gesture operates as follows for each target object. For example, if it is entered between the lines of a character string, the end and the beginning of the lines before and after are connected (line breaks are deleted).

In the "gesture coordinate data switching" gesture, the handwriting recognized as a gesture is switched to coordinate data. Alternatively, the handwriting recognized as coordinate data is recognized as a gesture.

This "gesture coordinate data switching" gesture operates as follows for each target object. For example, when it is written on the gesture, it is assumed that the gesture operation is canceled and the handwriting interpreted as the gesture is input as the coordinate data. When the gesture is entered on the object generated by inputting the coordinate data from the pen handwriting, the object is deleted, and the coordinate data generated by the object is treated as the gesture input by the shape. If it is difficult to interpret, present a list of recognition candidates to the user for instructions.

In the change of operation at the batch execution type editing, when the gesture of "cancel" is written on another gesture, the gesture description described below is erased and the cancel gesture is temporarily changed. Leave (disappear on next input). Alternatively, a change cancellation mark “Iki” is added to the gesture described below. Attribute input window,
In the case of a window that is temporarily opened, such as the Kana-Kanji conversion window, all the inputs on the window up to the present are invalidated, the window that was opened temporarily is closed, and the leader line is deleted. In the case of other windows, the gesture description entered immediately before is erased and the cancel gesture is left temporarily (disappeared by the next input). Alternatively, the mark “Iki” for canceling the change is added to the gesture that was just entered.

Further, in the change point of the operation at the batch execution type editing, the gesture of “input end instruction” is the gesture of suspending the execution when the target object is the gesture of suspending the execution. The process corresponding to is executed. For windows that are temporarily opened, such as the Attribute Input window and the Kana-Kanji conversion window, all the input up to the present is enabled, the window that was opened temporarily is closed, and the operation to be executed is displayed with a character next to the leader line. Fill out with (recognized as a gesture with text writing). In the case of other windows, batch editing is performed.

In the embodiment of the present invention, the following is realized by using each gesture as described above.

First, the kana-kanji conversion gesture will be described.

For example, FIG. 7 shows a kana-kanji conversion gesture in which the range from the start point to the end point of kana-kanji conversion and the kana-kanji conversion instruction in this embodiment are designated by continuous handwriting.
(A), (b), (c), (d) of Fig. 8 or (a), (b), (c), and (d) of Fig. 8, the start side of the smooth handwriting (start point SP side) Alternatively, a shape in which a special shape as shown by 12, 13, 14, 15 is added to the dotted line portion in the figure on the terminal side is used. In addition, in FIG.
(a) and (a) of Fig. 8 show a left-handed shape as a special shape to be recognized, and Fig. 7 (b) and (b) of Fig. 8 show a right-handed shape. 7 (c) and FIG. 8 (c) show the shape in which the starting point is clockwise, and FIG. 7 (d) and FIG. 8 (d) show the shape in which the starting point is counterclockwise.

The procedure for performing kana-kanji conversion is to recognize the handwriting corresponding to the above gesture, then search for the characters that have passed the closest handwriting and arrange them in the passing order of this handwriting, and then Kana-Kanji conversion is performed on the character string that has passed through the immediate vicinity.

That is, for example, it is assumed that the character string "Today is good" is displayed as shown in FIG. 9 (a). At this time, with the stylus pen 60 of the apparatus of the present embodiment, a character string in the character string to be converted into Kana-Kanji (a character string of "Kyoha") is displayed near the character string (b) of FIG.
Draw a handwriting PM as shown in. When the handwriting PM is drawn, the apparatus of the present embodiment determines whether the handwriting PM is a kana-kanji conversion gesture, and if it is kana-kanji conversion gesture, calls the kana-kanji conversion function and is instructed by the handwriting PM. Kana-Kanji conversion is performed on the character string. As a result, the character string shown in (b) of FIG. 9 above is converted into Kana-Kanji, and as shown in (c) of FIG.
The conversion is executed as follows. Note that, as shown in FIG. 9D, a character string may be stuffed and the gesture display may be changed accordingly.

The procedure of Kana-Kanji conversion of this embodiment will be described in detail in the flow chart described later.

The kana-kanji conversion function of this embodiment is shown in FIG.
As shown in (a) and (b), it also works for vertical text strings. That is, the operations are unified for vertical writing and horizontal writing. Further, even for a character string having a special layout as shown in (a) and (b) of FIG. 11, if it is indicated in the same manner as a normal character string like the handwriting PM in the figure of (a) of FIG. Kana-Kanji conversion is possible.

By the way, when the handwriting is drawn in a direction different from the connection of the character strings, such as the handwriting PM shown in FIG. 12A, in the present embodiment, the character of FIG. The columns will be transformed as shown in FIG. On the other hand, for example, by comparing the direction of the character string with the direction of the handwriting and devising a method of selecting the character to be the target of Kana-Kanji conversion, conversion as shown in (c) of FIG. 12 is performed. Of course, it can be set. Further, when the locus PM shown in FIG. 12D is written in the character string, the conversion shown in FIG. 12E is performed in this embodiment. Also in this case, for example, by comparing the direction of the character string with the direction of the handwriting, it is possible to perform the conversion as shown in FIG.

Although the examples of FIGS. 9 to 12 show examples in which the handwriting instructing the conversion remains after the execution of the kana-kanji conversion, it is possible to display the conversion result and erase the handwriting. You can go. Further, as shown in FIG. 9D, the handwriting is transformed and displayed according to the length of the converted character string, so that it is possible to clearly indicate that the conversion operation is in progress.

As described above, according to the present embodiment, the kana-kanji conversion can be performed quickly, and the kana-kanji conversion can be performed by the same operation for vertical writing and horizontal writing, and even for characters displayed in a special layout. Kana-kanji conversion is possible with the same operation.

FIG. 13 shows a flowchart of a kana-kanji conversion gesture. However, in this flowchart, for simplification, the case where the terminal has a counterclockwise shape is shown. The other three shapes can be determined by almost the same means.

In FIG. 13, in step S11, the input of one stroke of handwriting from the stylus pen 60 is awaited. When handwriting is input in step S11, the process proceeds to step S12. In step S12, the handwriting input in step S11 is smoothed, and then the process proceeds to step S13.

In step S13, it is determined whether or not the handwriting includes a portion that intersects with the handwriting of oneself. If there is a portion intersecting the handwriting (Yes), the process proceeds to step S14, and if it does not exist (No), the process proceeds to step S18 described later.

In step S14, it is determined whether or not there is one intersecting portion of the handwriting. When it is determined in step S14 that the number of intersecting portions is one (Yes
If it is determined that the number of intersecting portions is not one (No), the process proceeds to step S18.

In step S15, it is determined whether the handwriting between the intersections is left-handed. If it is determined in step S15 that the handwriting between the intersections is left-handed (Yes), the process proceeds to step S17, and if it is determined that the handwriting is not left-handed (No), the process proceeds to step S18.

In step S17, the length of the intersection (intersection) with the start point SP of the handwriting is the end point E of the handwriting.
It is judged whether or not it is longer than the length of P and the intersection. If it is determined in step S17 that the length between the start point SP and the intersection is longer than the length between the end point EP and the intersection (Yes), the process proceeds to step S19, where the length between the start point SP and the intersection point is the end point EP. If it is determined that it is not longer than the length of the intersection (the length of the end point EP and the intersection is longer than the length of the start point SP and the intersection) (No), step S
Proceed to 18.

In step S19, a character is searched for along the locus from the start point SP to the intersection. After this step S19, the process proceeds to step S110, in which the kana-kanji conversion for the retrieved character is started (executed). This step S110
After that, the process returns to step S11.

The above-mentioned step S when No is judged in each of the above-mentioned steps S13 to S17.
At 18, other processing is performed. That is, since the handwriting input in step S11 is not a kana-kanji conversion gesture, other gestures, figures, characters, etc. are input and the corresponding processing is performed. This step S
After 18, the process returns to step S11.

Next, the function of distinguishing gestures having similar shapes in the apparatus of this embodiment will be described.

As a procedure for distinguishing between gestures having similar shapes, the shape of the input handwriting is first recognized. Based on the shape type of the recognition result, a list of functions corresponding to the shape is created. For each of the functions listed above, a determination area is provided around the input handwriting that is determined by the size and orientation of the handwriting. However, the shape and size of the area differ depending on the type of function. Next, the data existing in the judgment area is searched on the screen, and the direction and size of the data found by this search on the screen are compared with the direction and size of the input handwriting. Also, it is checked whether the positional relationship between the data and the input handwriting matches. This inspection is an inspection as to whether or not it is suitable as data to be operated. A more accurate inspection can be performed by changing the inspection procedure depending on the type of data or function. Furthermore,
As a result of the above comparison and inspection, if there is no matching data due to the relationship of the orientation and the size, the type of function currently being investigated is excluded from the above list. After that, among the listed functions, the one with the highest score of the above-described conformity is adopted as the function indicated by the input handwriting. However, when the handwriting has a shape that is out of the standard as a gesture indicating the function, it is naturally possible to realize the evaluation by discounting the score.

In the present embodiment, to distinguish between gestures having similar shapes, for example, the input handwriting corresponding to the gesture of "range designation" shown in Table (3) of Table 3 and "insertion".
The case where the input handwriting corresponding to the gesture is distinguished is described as an example.

Here, for example, the function of the "range designation" gesture is referred to as a function A, and the function of the "insert" gesture is referred to as a function B.

These two gestures are shown in FIG. 14 (a).
Gesture GE _A shown, as gesture GE _B shown in (b) of FIG. 14, "range specification" and is used as a "insertion", these gestures GE _A gesture GE
_B is very similar in shape only.

From the above, for example, in FIG.
When the character string as shown in (a) is displayed and the handwriting PM as shown in (b) of FIG. 15 is input to this character string, in this embodiment, the character string of FIG. As in the flowchart, it is determined which gesture the input handwriting corresponds to.

FIG. 16 shows a flowchart for distinguishing gestures having similar handwriting shapes.

In FIG. 16, in step S21, a pen input or an input from a keyboard or the like is awaited, and when an input is made in step S21, the process proceeds to step S22.

In step S22, it is determined whether or not the input is handwriting from a pen. This step S2
When it is determined that the input is handwriting in 2 (Yes), the process proceeds to step S24, and when No is determined, the process proceeds to step S23. In step S23, since the input is not a handwriting, processing corresponding to the input (for example, keyboard input) is performed. After step S23, the process returns to step S21. In step S24, the handwriting input is smoothed, and then the process proceeds to step S25.

In the step S25, it is determined whether or not there is a handwriting whose recognition is suspended among the handwriting input before the handwriting input. If the pending handwriting input is found in this step S25 (Yes), the process proceeds to step S26, and if not (No), the step S28 is performed.
Proceed to.

In step S28, the handwriting input (one handwriting input) is recognized. For example, the handwriting input is recognized using the shape recognition table stored in the system ROM 52 or the like. Specifically, processing is performed such that the shape number Sno in the shape recognition table corresponding to the shape of the handwriting input is returned. If there is no shape number Sno corresponding to the handwriting input in the shape recognition table, an unregistered code (a number that means unregistered) is returned. Note that the unregistered code is output for a handwriting input that is not a gesture or a part of any gesture.

After step S28, step S2
Proceed to 9. In this step S29, the shape number Sn
It is determined whether or not o is not registered. That is, in step S29, it is determined whether or not the unregistered code is returned from step S28, and if it is determined that the unregistered code is returned in step S29 (Yes
If it is determined that it is not unregistered (registered) (No), the process proceeds to step S210.
Proceed to 11. In step S210, processing corresponding to the unregistered handwriting input (graphics, characters, etc.) is performed. After step S210, the process returns to step S21.

If it is determined Yes in step S25, the handwriting input is not one stroke, so step S2
In 6, the recognition of multiple strokes is performed. In this step S26 as well as in the above step S28, for example, a system ROM
The shape recognition table corresponding to the gesture of the multi-handwriting held in 52 or the like is used to recognize the multi-handwriting input. After step S26, the process proceeds to step S27. In this step S27, similarly to the above step S29, it is determined whether or not the shape number Sno is not registered (unregistered). If Yes is determined, the process proceeds to step S28, and if No is determined. For that, the process proceeds to step S211.

Step S29 or step S27
When it is determined to be No and the process proceeds to step S211, it is determined in step S211 whether or not the shape number Sno is undetermined (undetermined). If it is determined in step S211 that it is unconfirmed (Yes), the process proceeds to step S219, and if it is determined that it is not undetermined (confirmed) (No), the process proceeds to step S212. Unconfirmed means that the input handwriting can be part of the gesture shape. Note that the above step S2
In step 19, after registering the recognition hold handwriting, the above step S
Return to 21. The registered handwriting is checked in step S25 and used in step S26.

In step S212, the table of the function list corresponding to the shape number Sno stored in the system ROM 52 is referred to. afterwards,
In step S213, the determination area is calculated for each item of the function list.

When the determination area is calculated in step S213, data in the determination area (data such as characters and figures displayed on the screen) is searched for in step S214. The mechanism for finding the data in the designated area on the screen is the one generally used in CAD or the like.

After this step S214, step S
At 215, the direction, size, position of the retrieved data,
Inspect the type. However, in this step S215, the direction, size, and position are inspected, but the comparison method is different between the function A and the function B, and another comparison procedure is called depending on the type of function. Then, in step S216, the function having no matching data is deleted from the function list. Step S after this step S216
In 217, the gesture number for the function with the highest score in the function list is given to the gesture number Gno to be described later, and in the next step S218, the gesture number Gno.
Normal gesture processing is executed for the gesture corresponding to no. Then, it returns to step S21. The gesture number is a number assigned so that each of the functions determined by the gesture can be distinguished.

FIG. 17 shows a flowchart of a specific process of calculating the determination area of the function A in the function list in the flowchart of FIG. 16 (step S213).

In the flowchart of FIG. 17, in step S221, as shown in FIG. 18, a bisector (straight line X) of the angle formed by the start point SP, the vertex ap, and the end point EP of the input handwriting is obtained, and step S222 Proceed to. In step S222, two straight lines (straight line X _1A , straight line X _1A , passing through the apex ap and having an accuracy of 45 ° with the straight line X) are formed.
X _2A ). In the next step S223, an intersection P of the straight line passing through the start point SP and the end point EP and the straight line X shown in FIG. 18 is obtained, and then the process proceeds to step S224. In step S224, the area E _A shown in FIG. 18 on the four half plane including the intersection P surrounded by the straight lines X _1A and X _2A is set as the determination area of the function A.

FIG. 19 shows a flowchart of a specific process of calculating the determination area of the function B in the function list in the flowchart of FIG. 16 (step S213).

In the flowchart of FIG. 19, in step S231, as shown in FIG. 20, a bisector (straight line X) of the angle formed by the start point SP, the vertex ap, and the end point EP of the input handwriting is obtained, and step S232. Proceed to. In this step S232, as shown in FIG.
A straight line Y that passes through the above and is orthogonal to the above straight line X is obtained. In the next step S233, the length L of the input handwriting is acquired, and in step S234, the intersection point P of the straight line X passing through the start point SP and the end point EP and the straight line X shown in FIG. 20 is obtained, and then the process proceeds to step S235. In the step S235, as shown in FIG. 20, two straight lines X _1B and X _2B are obtained by translating the straight line X along the straight line Y by a times (a · L) times the length L.
Ask for. In the next step S236, as shown in FIG. 20, a straight line Y ₁ is obtained by translating the straight line Y along the straight line X to the opposite side of the intersection P by b times (b · L) times the length L, and In S237, as shown in FIG. 20, the straight line Y is moved along the straight line X to the intersection P side by the length c of the length L.
Find a straight line Y ₂ that has moved by a factor of (c · L). afterwards,
In step S238, the straight lines X _1B , X _2B , Y ₁ , Y
_The closed region surrounded by ₂ is set as the determination region E _B of the operation B.
The values of a, b, and c are stored in advance in the ROM 52 or the like shown in FIG.

FIG. 21 shows a flow chart of a concrete process of inspecting the direction, size, and position (step S215) for the function B in the flow chart of FIG.

In FIG. 21, in step S241, for example, the first found data of the data in the determination area E _B obtained earlier is set as D. In step S242, for example, 100 points are given to the data D. In step S243, it is determined whether or not the data D is, for example, text data, and if it is determined that it is text data (Yes), the process proceeds to step S244, and if it is determined that it is not text data (No). Then, the process proceeds to step S249 described later.

When it is determined in step S243 that the data D is character data and the process proceeds to step S244, in step S244, a character string for one line including the data D is acquired, and then in step S245. An angle R formed by the character feed direction of the acquired character string and the straight line Y calculated in step S232 of the flowchart of FIG. 19 is calculated. The angle R is -90 ° <R ≦ 90 °.
Is.

Next, in step S246, it is determined whether the absolute value of the angle R is larger than 30 °.
If it is determined in step S246 that the absolute value of the angle R is greater than 30 ° (Yes), the process proceeds to step S249, and if it is determined to be small (No), step S24 is performed.
Proceed to 7.

In step S247, it is determined whether the absolute value of the angle R is 10 degrees or less. If it is determined in step S247 that the absolute value of the angle R is less than or equal to 10 ° (Yes), the process proceeds to step S252 described below, and it is determined that the absolute value of the angle R is greater than 10 °. (No) proceeds to step S248.

In step S248, it is determined whether the absolute value of the angle R is 20 degrees or less. If it is determined in step S248 that the absolute value of the angle R is 20 ° or less (Yes), step S25.
When it is determined that the absolute value of the angle R is larger than 20 ° (No), the process proceeds to step S250.

In step S250, the score of the data D is set to 40 points, and in step S251, the data D is set.
After setting the score of 70 to 70, the process proceeds to step S252.

After steps S250, S251 or in step S247, if it is determined Yes and the process proceeds to step S252, it is determined in step S252 whether the straight line Y intersects the outer quadrangle of the character string. . When it is determined that the intersections are made in step S252 (Y
es), the process proceeds to step S255 described later, and if it is determined that they do not intersect (No), the process proceeds to step S253.

In step S253, it is determined whether the outer quadrangle is on the intersection P side of the straight line Y or not. This step S
If it is determined in 253 that the outer quadrangular shape is on the intersection P side of the straight line Y (Yes), the process proceeds to step S254, and if it is determined that it is not on the intersection P side of the straight line Y (No), step S254 is performed.
Proceed to 255.

At the step S254, the data D
After the process of deducting 40 points from the score of is performed, the process proceeds to step S255.

Here, in the step S243, the data D
If it is determined that is not character data (No), or if the absolute value of the angle R is greater than 30 ° in step S246 (Yes), in step S249,
The score of the data D is 0. This step S24
After 9, the process proceeds to step S255.

In step S255, it is determined whether there is the next data. If it is determined in step S255 that there is the next data (Yes), step S2
In step S256, after the next data is set to D in step S256, the process returns to step S242. If it is determined in step S255 that there is no next data (N
o) ends the process.

That is, in the example of FIG. 15 described above, for example, as shown in (a) of FIG. 22, there is no character data in the determination area E _A for the function A corresponding to the handwriting PM in the figure, and As shown in (b) of FIG. 22, it is understood that two character data (“ki” and “good”) exist in the determination area E _B for the function B corresponding to the handwriting PM in the figure.

[0177] Therefore, since the above functions A had no data in the determination region E _A, becomes no matching data, inspection line by the method shown in Step S241~ step S256 of the flowchart of function B in FIG 21 Be seen. In this embodiment, the result is that the character string "Today is sunny" in FIG. 22B is suitable as the operation target of the operation B. Note that, as a result of the inspection, only the function B has data in which the relationship of the orientation and the size is compatible, and thus the function A is deleted from the function list.

As described above, according to the present embodiment, the recognition accuracy of gestures is improved, and one shape can be treated as different gestures according to the data arrangement on the screen. Further, if a plurality of gestures that are distinguished by drawing a shape while changing the rotation angle are distinguished by the above method, it becomes possible to draw a gesture at a free angle according to the data on the screen. In particular, if all gestures are handled in this way, it is possible to realize an operation system that allows drawing gestures from any direction.

The method of calculating the judgment area is not limited to the above example. In the present embodiment, a determination area calculation method is provided in which different determination areas are provided for different types of gesture functions, and the shape of the determination area is calculated (changed) according to the direction, size, and position of the feature point of the gesture handwriting. Although an example is given, it is possible to calculate the determination region by another calculation method based on this idea.

Next, a conversion gesture for converting the gesture into normal coordinate data will be described.

As a procedure for converting a gesture into normal coordinate data, first, a conversion gesture from gesture to normal coordinate data is recognized. Next, it is determined that this conversion gesture is a gesture drawn on a certain gesture (the gesture below the handwriting is, for example, gesture A). After that, the processing executed by the gesture A is canceled, and the handwriting when the gesture A is input is processed again as simple coordinate data.

In the present embodiment, a gesture having a shape like the handwriting PM shown in FIG. 23 is used as the conversion gesture. Further, here, as the gesture to be converted when the gesture is converted to the normal coordinate data, for example, a locus PM that circles data on the screen (character data in the example of FIG. 24) as shown in FIG. Is taken as an example.

For example, assume that rectangular graphic data as shown in FIG. 25A is displayed on the screen. In this case, it is assumed that the user inputs a handwriting PM as shown in (b) of FIG. 25 in order to input a free-form curve surrounding the rectangular figure, for example. At this time, the computer recognizes the handwriting PM in FIG. 25B as a gesture of selecting the rectangular figure data, and if the rectangular figure data is selected, the following is performed. Do what you do.

That is, when the selected data and gesture handwriting are displayed as shown in FIG. 25 (b),
The user inputs the handwriting PM _{N of the} conversion gesture as shown in FIG. 25 (c). At this time, the apparatus of the present embodiment cancels the operation of the gesture corresponding to the handwriting PM in 25 (b) above, thereby canceling the selected state of the rectangular graphic data. Next, the apparatus of this embodiment re-registers the handwriting PM _o shown in FIG. 25C as coordinate data input. As a result, the handwriting P in FIG. 25 (c) is displayed on the screen.
Free curve data such as M _o is additionally displayed. Note that FIG. 25 (c) shows an example in which the handwriting PM _{N of the} conversion gesture is also displayed as it is, but after the gesture is converted into coordinate data, the handwriting PM _N is erased. You may add the process to do.

Further, in the above-mentioned example, the case where the gesture is converted into the normal coordinate data is taken as an example. However, the device of the present embodiment is the reverse, that is, the one recognized as the normal coordinate data is converted into the gesture. It is also possible.

As a procedure in this case, first, a conversion gesture from normal coordinate data to a gesture is recognized.
Next, it is determined that the gesture is drawn on the data generated by the handwriting input (handwriting data when the data below the gesture is input is referred to as handwriting A). Delete the data generated by this handwriting A,
Recognize the handwriting A as a gesture.

For example, referring to FIG. 25 described above as an example, when rectangular figure data as shown in FIG. 25A is displayed on the screen, for example, the rectangular figure is surrounded. If the handwriting PM as shown in FIG.
It is assumed that the handwriting PM in (b) is recognized as coordinate data of a free curve.

In this case, the user is required to perform the same operation as shown in FIG.
Input the handwriting PM _N of the above conversion gesture as shown in (c). At this time, the apparatus of the present embodiment executes the cancellation of the normal coordinate data of the free curve corresponding to the handwriting PM of FIG. 25B, and then selects the handwriting PM _o of FIG. _25C , for example. Re-register as the gesture input of. As a result, the rectangular graphic data on the screen is selected.

It should be noted that a combination of the conversion gesture for converting the above-mentioned gesture into the normal coordinate data and the conversion gesture for converting the normal coordinate data into the gesture is set as a gesture having the same shape, and if there is a gesture below. It is possible to switch and use the gesture so as to convert it into normal coordinate data, and conversely, if there is normal coordinate data below, to convert the normal coordinate data into a gesture.

From the above, although it is conventionally necessary to perform two inputs of canceling the operation and inputting the handwriting again, the present embodiment makes it possible to correct the input only once. Further, conventionally, the input can be canceled only in the reverse order of input of the handwriting, but according to the present invention, any handwriting can be converted into normal data while the handwriting display of the gesture remains on the screen. Furthermore, according to the present embodiment, any handwriting can be converted into a gesture while the data display remains on the screen.

FIG. 26 shows a flowchart of the processing of the above-mentioned conversion gesture for converting the above-mentioned gesture into normal coordinate data or converting the normal coordinate data into a gesture.

In FIG. 26, 1 is set in step S32.
When the stroke input of one stroke is awaited, the stroke of the input stroke is smoothed in step S33.

In the next step S34, it is judged whether or not the shape of the input handwriting corresponds to the conversion gesture shown in FIG. If it is determined in step S34 that the input shape is the shape corresponding to the conversion gesture (Yes), the process proceeds to step S36, and if it is determined that the input shape is not the shape corresponding to the conversion gesture (No), the process proceeds to step S35. This step S3
In step 5, if the input handwriting is another gesture, a character, or a figure, the corresponding processing is performed.

If YES in step S34, in step S36, it is determined whether or not there is a gesture under the input handwriting. When it is determined in step S34 that there is a gesture below the input handwriting (Yes)
If it is determined that there is no (N
For o), the process proceeds to step S37.

If it is determined Yes in step S36, the process of canceling the previously executed process by the gesture under the input handwriting is performed in step S38, and then the handwriting of the gesture under the input handwriting is performed in step S310. Re-register as normal coordinate input. This step S31
After 0, the process returns to step S32.

When it is determined No in step S36, it is determined in step S37 whether or not there is data (data such as characters and figures) under the input handwriting. If it is determined in step S37 that there is data below the input handwriting (Yes), the process proceeds to step S39.
If it is determined that there is not any (No), the process returns to step S32.

In step S39, it is determined whether or not the data under the input handwriting is data drawn with a pen. If it is determined in step S39 that the data under the input handwriting is the data drawn by the pen (Yes), the process proceeds to step S311. If it is determined that the data is not the data drawn by the pen (No), the process proceeds to step S32. Return.

In step S311, the data under the input handwriting is deleted, and then in step S312, the handwriting of the data under the input handwriting is re-registered as a gesture. After step S312, the process returns to step S32.

Further, the procedure of the gesture of the lead line in the present embodiment is as follows.

First, two coordinate points are acquired by a series of operations. For example, two coordinate points are input by sequentially striking two points on the tablet with a pen. In this case, if the operation is confusing with other operations, the operation may be switched in advance with a button or the like so as to accept only this operation. Also, for example, 1
The starting point and the ending point of the handwriting are regarded as two coordinate points and acquired.
In this case, if the operation is confusing with other operations, the shape recognition mechanism may be used to recognize a special shape in the middle part of the handwriting, and the operation of the present invention may be triggered by the recognition. The operation may be switched in advance using a button or the like so that only this operation is accepted.

Next, the data below the first input point on the screen is set as the processing target. For example, when input is made with one handwriting, the area under the coordinates of the starting point is inspected, and if there is an icon below that, the icon is set as the processing target.
If there is a character string below, the one character closest to the starting point is set as the processing target. If the selection operation has already been performed and the data to be selected is below, all the data selected by the selection operation are to be processed.

Further, the position of the point input later on the screen is used as the display and input location of the information regarding the data to be processed. For example, when an operation is input with one handwriting, the coordinates of the end point are displayed and set as an input location, and if there is a window system, a window for displaying information may be opened at the coordinates. Note that a window whose frame cannot be seen may be opened. Or set a square area near the coordinates of the end point,
You may draw an outer frame on the screen. It should be noted that it is possible that the frame is not drawn.

Also, the type of information displayed at the designated location
It is also possible to switch the display format depending on the type of data to be operated. Here, as a method of inputting information from a designated place, selection of a menu, input of numerical values or character strings as data, etc. can be considered. When the character string “Tortsume” is input to the function character string input position of the leader line, a process of deleting the character string and filling the space) can be accepted.

Further, in order to keep the input information on the screen even after the input, the input information is converted into an abbreviated character string or symbol, and the tip of the handwriting of the lead line is drawn after the input is completed. It is also possible to display it on and keep the display with the handwriting of the leader line. In this case, for example, a button "Leave display on screen and finish" may be attached to the information display portion, or a mode in which information is displayed and retained by an abbreviation may be set in advance.

Here, when the present invention is carried out by using the shape recognition mechanism, it is necessary to bring the starting point and the ending point of the handwriting to arbitrary positions. In addition, in the present embodiment, the shape of the leader line gesture is set so as to satisfy the above condition as shown in FIG.
The shape is as shown in (a) or (b). For example,
By using this gesture shape shown in (a) of FIG. 27, even if the start point SP and the end point EP of the handwriting PM are brought to arbitrary positions, the characteristic shape 25 surrounded by a dotted line in the middle of the handwriting PM 25
It is possible to sufficiently distinguish it from other gesture shapes only by the folded-back portion such as 2. In addition, the shape of the gesture shown in FIG. 27B is also a characteristic shape 25 surrounded by a dotted line in the figure in the middle of the handwriting PM.
Only the loop part such as 4 can be sufficiently distinguished from other gesture shapes. Therefore, even if the leader line is drawn from any position in any direction, the recognition can be sufficiently performed.

In this embodiment, the shape shown in (a) or (b) of FIG. 27 described above is used as the leader line gesture, and the existing shape recognition mechanism is utilized to make an arbitrary position, direction and size. I try to recognize the drawn line.

Further, the apparatus of this embodiment is provided with two buttons, for example, "lead line mode" and "lead line mode release" on the keyboard. However, this is used for the sake of simplicity of description, and it is needless to say that it can be omitted when implementing the present invention. Further, the apparatus of this embodiment uses the existing window system.

Here, for example, it is assumed that a character string as shown in FIG. 28 (a) is displayed on the screen. In the present embodiment, an example will be given in which an operation for changing a numeral portion in the character string into a Gothic character is performed. In this case, in the apparatus of this embodiment, for example, the "leader line mode" button is pressed and the locus PM is input as shown in FIG. 28 (b). At this time, for example, work R
The position of the start point SP is stored as the coordinate 1 in the AM 55, and the position of the end point EP is stored as the coordinate 2. After that, the number under the coordinate 1 is selected as the operation target. The existing mechanism is used to search the data at the position from the coordinate position.

For example, if the found data is a character, the menu item used for the character is selected using the correspondence table shown in Table (7) of Table 7. In addition to using the correspondence table of this table (7), it is naturally conceivable to select a possible operation and register it in the list.

[0210]

[Table 7]

Next, the current attribute of the selected character is set in the data display portion of the menu. For example, character size = 9 points, character type = Mincho, and character color = black.

Next, the menu window is displayed at the position of the coordinate 2 and the selected menu item and the set current attribute are displayed inside. The displayed screen is like a window 293 shown in FIG. 28 (c). In this menu window 293, some items appearing in the correspondence table of the above table (7) are displayed from the beginning, and the overflowed portion can be operated when other operation is pressed.
The apparatus of this embodiment is provided with three buttons, that is, a cancel button, an end button, and an end button with the display left so that the menu can be ended. Also, display the menu,
The existing window system mechanism is used to determine the data input / end button for each item.

Next, for example, the above menu window 2
Press the "Gotic" button indicated by item 294 in the menu shown at 93 with a pen. This causes Gotick to appear in the area indicated by item 296 in the menu (this is handled automatically by the window system). here,
The input is completed by pressing the "End display and leave" button indicated by item 295 with the pen. The window system erases the menu window. Instead of pressing the "Gotic" button of item 294, it is also possible to directly input the character string "Gotic" using keyboard input or handwritten character input in the area shown in item 296. At that time, it is naturally conceivable to allow the input of abbreviations such as Gothic → Gochi and aliases such as Gothic → Gothic.

After that, referring to the information from the window system, since the end condition is “end with display left”, the input change instruction is validated. The instruction is extracted from the information of the window system and set to the data to be processed. As a result, as shown in FIG. 28 (d), the characters are changed to the Gothic type only in the numbers. At this time,
Since the ending condition is "end leaving display", the handwriting of the gesture of the leader line is left. For example, selecting "end" will erase the gesture handwriting.

Further, the character string used for the setting is displayed at the position of coordinate 2. As a result, the character “Gotic” shown in FIG. 28D is displayed. With this, it is possible to understand at a glance what kind of operation was performed even in the device in which the Gothic font cannot be used on the screen. Especially in batch execution type editing, the font is not changed on the spot, which is a very convenient function. In addition, this display is ON / OF
Naturally, it is conceivable to add a mechanism that allows F. Further, it is naturally conceivable to correct the display position of the character string in order to make this display easier to see. Furthermore, operations that are not specified by a character string can also be converted into a character string and displayed, and the character string can be converted into an abbreviation (eg Gotic → Gochi).
It is also possible to display. Of course, when some designations are made, it is natural to list the character strings.

As a modification of this embodiment, it is possible to set the window opened before the coordinate 2 to be a transparent window indicated by the dotted line 326 in FIG. 29, and set the entire handwritten character recognition area inside the window. Further, when a gesture GE ₃₂₅ for inputting an instruction to be described later as shown in FIG. 29 is enabled in the window, it becomes possible to use various functions by inputting handwritten characters. In FIG. 29, the gesture _GE322 having the shape shown in FIG. 27 (a) is used.

As described above, according to the present embodiment, it becomes convenient to set detailed information for data, it becomes easy to refer to the attributes set for the data, and information such as attributes is set. It is possible to provide a method of displaying the one that is difficult to express on the screen. In addition, at the time of batch execution type editing, the entered instructions can be confirmed at a glance.

FIG. 30 shows a flowchart of specific processing in the above-described leader line gesture.

In FIG. 30, in step S51, input is awaited, and if there is input, it is determined in step S52 whether the input is handwriting. If it is determined in step S52 that the input is handwriting (Yes), the process proceeds to step S53, which will be described later, and if it is determined that the handwriting is not handwriting, the process proceeds to step S57.

In step S57, it is determined whether or not the input is the "leader line mode" button. If it is determined that the input is the leader line mode (Yes), step S57.
In step S58, the flag Hflag indicating the lead line mode is set to "1" in step S58, and step S51 is performed.
Return to. If it is determined in step S57 that the input is not the "lead line mode" button (No), the process proceeds to step S59.

In this step S59, it is judged whether or not the above-mentioned input indicates the "leader line mode release" button, and if it is judged that the leader line mode is released (Y
es), the process proceeds to step S510, and the step S5
The flag Hflag indicating the leader line mode at 10 is "0" (that is, a flag indicating cancellation of the leader line mode).
Stands and returns to step S51. If it is determined in step S59 that the button is not the "lead line mode release" button, in step S511, a process corresponding to the above-mentioned input, such as a key input, is performed, and then step S51 is performed.
Return to.

On the other hand, if it is determined in step S52 that the input is handwriting (Yes) and the process proceeds to step S53, the flag Hflag indicating the leader line mode is "0" (indicating that the leader line mode is released). Flag) is determined. If it is determined in step S53 that the flag Hflag is "0" (Yes), the process proceeds to step S54, and if it is determined that it is not "0"("1") (No), the process proceeds to step S520 described later. move on.

If it is determined Yes in step S53, the shape of the input handwriting is recognized in step S54, and then the process proceeds to step S55.

In step S55, it is determined whether or not the input handwriting has the shape of a lead line. If it is determined that the input handwriting has the shape of the lead line (Yes), the process proceeds to step S520, and the shape of the lead line. If not (No), the process proceeds to step S56. Step S
At 56, after processing corresponding to the input handwriting (other gesture, character, figure, etc.), the process returns to step S51.

If No in step S53 or Yes in step S55, step S53
At 520, the coordinates of the starting point SP of the input handwriting are set to coordinates 1,
In step S521, the coordinates of the end point EP of the input handwriting are set to coordinates 2.
And After step S521, step S522 is performed.
Proceed to.

In step S522, the data in the vicinity of the coordinate 1 is searched, and then the process proceeds to step S523. In step S523, for example, the system ROM 5 described above is used.
From the menu item tables stored in 2 and the like, one corresponding to the data found by the search in step S522 is selected.

After the above step S523, step S
In step S524, the value currently set in the data is registered in the display field, and then in step S525.
The menu is displayed at the position of coordinate 2 and the data is input.

Next, in step S526, the data input from the menu is acquired, and the flow advances to step S527. In step S527, for example, it is determined whether or not a cancel operation has been acquired and ended from the menu, and when it is determined that the cancel operation has been acquired and ended (Yes), step S531. Proceed to. If it is determined in step S527 that the cancel operation has been completed without being acquired (No), step S52 is performed.
Go to 8.

In step S528, the input operation (operation acquired from the menu) is executed, and then step S5
Proceed to 29.

[0230] In step S529, it is determined whether or not the operation acquired from the menu has been turned to the end operation. When it is determined in step S529 that the end operation has been acquired and the end has been completed (Yes), the processing proceeds to step S531, and when it is determined that the end operation has not been acquired (No) in step S530.
Proceed to.

In step S530, a process of displaying the character string representing the input function from the coordinate 2 is performed, and in step S531, the handwriting of the leader line is erased.
After these steps S530 and S531, step S5
Return to 1.

Next, the input end instruction gesture will be described.

As a processing procedure of the input end instruction gesture, first, the input end instruction gesture is recognized.
At this time, on the window with the gesture drawn,
Search for data that has been entered / selected but not yet processed, or gestures that have been entered but for which no operation has been performed. If the found data is data, it is considered that all the found data (group) have been input, and the data is processed. When the process to be executed is not designated, it is possible to execute the latest process that can be executed on the found data among the processes executed in the past. Also, if the gesture was discovered, the gesture (s) found
For each, the corresponding processing is executed. If nothing is found, it is not necessary to do anything, but it is also possible to execute the processing performed immediately before again on the immediately previous data. If it is impossible to execute it again, it can be ignored.

A detailed description will be given below. It is assumed that a character string as shown in FIG. 31 (a) is displayed on the display screen. Here, it is assumed that the drawing line gesture GE ₃₃₁ is used to change the number “1” to a Gothic typeface. This gothic typeface processing is registered in the processing content registration list.
Next, the gesture GE ₃₃₅ sets the two characters “2” and “3” to the selected state.

Here, the gesture GE of FIG.
_{When the} handwriting as shown in ₃₃₆ is input, the input handwriting is determined to be the input end instruction gesture. At this time,
If there is selected state data, the latest processing that can be performed on the selected state data is executed on the selected state data. It should be noted that when there is no data in the selected state, the presence of an unprocessed gesture is determined, and when it is present, the discovered gesture (one or more) is executed, for example, in the order of input time series. If there is no unprocessed gesture, nothing is executed.

Next, on the display screen at the stage where the processing is completed, a display in which the numbers are in the Gothic typeface as shown in FIG. 31 (b) is displayed. In this case, it is possible to add a mechanism for erasing the handwriting of the gesture.

As described above, according to the present embodiment, the input end instruction gesture can be used as a gesture indicating the end of the character string input in the input box for inputting the character string, and a number of selection branches can be selected. It can be used as a gesture to indicate the end of selection in the menu. Furthermore,
It can be used as a gesture to specify the target and execute the previous process again. In addition, it can be used as a gesture for instructing execution of editing for batch execution type editing. As described above, the input end instruction gesture activates various different processes in spite of the gesture having the same shape.

FIG. 32 shows a flowchart of the specific operation of the above-mentioned input end instruction gesture.

In FIG. 32, the input of one stroke is awaited in step S601. When one stroke is input, the shape of the handwriting of the input stroke is recognized in step S602. In the next step S6021, as a result of the shape recognition, it is determined whether or not the input handwriting is a selection gesture. If it is determined in step S6021 that the input handwriting is the selected gesture (Yes), the process proceeds to step S610, and if it is determined that the input handwriting is not the selected gesture (No), the process proceeds to step S6022. In step S6022, as a result of the shape recognition, it is determined whether the input handwriting is a leader line gesture. If it is determined in step S6022 that the input handwriting is a leader line gesture (Yes), the process proceeds to step S606, and if it is determined that the input handwriting is not the leader line gesture (No), the process proceeds to step S6023. In step S6023, as a result of the shape recognition, it is determined whether the handwriting input is an input end instruction gesture. Step S6023
If it is determined that the input handwriting is the input end instruction gesture (Yes), the process proceeds to step S603, and if it is determined that the input handwriting is not the input end instruction gesture (N).
In step o6024, the process advances to step S6024 to perform other processing. That is, since the handwriting input in step S601 is not a selection gesture, a leader line gesture, or an input end instruction gesture, a process corresponding to them is performed as an input of another gesture, a figure, a character, or the like. After this step S6024, step S60
Return to 1.

If it is determined in step S602 that the gesture is the selected gesture, the characters in the area surrounded by the selected gesture are selected in step S610, and the process returns to step S601.

If it is determined in step S602 that the gesture is a leader line gesture, the input of characters is awaited in step S606. If there is character input in step S606, it is determined in step S607 whether character input is completed. If it is determined in step S607 that the character input is not completed (No), the process returns to step S606, and if it is determined that the character input is completed (Yes), the process proceeds to step S608.

In step S608, the process corresponding to the input content is executed (eg, gotification), and then the process name is registered in the process name saving table in step S609 (eg, the process name is "selected character string"). "Gotic"). After step S609, step S6
Return to 01.

If it is determined in step S602 that the gesture is an input end instruction gesture, then step S6 is performed.
In 03, it is judged whether or not there is data in the selected state. If it is judged that the data exists (Yes), the step S
The process proceeds to step 604, and if it is determined that there is none (No), the process proceeds to step S611.

At step S604, step S60 is performed.
After executing the latest process that can be executed on the selected data in the process names registered in step 9, step S601
Return to. In step S611, it is determined whether or not an unprocessed gesture exists, and when it is determined that an unprocessed gesture exists (Yes), step S612.
For example, the process of executing the one or more gestures found in step S <b> 601 in the order of input time series is performed, and the process returns to step S <b> 601. If it is determined in step S611 that there is no unprocessed gesture (No), step S601.
Return to.

Further, in the present embodiment, the executed operation is also displayed. That is, as a procedure in this case, first, for example, an operation input from a keyboard or a file is expressed by a gesture operation. If one gesture cannot be expressed, it is converted into several gestures and expressed. Furthermore, if the gesture cannot be expressed by any means, an icon representing the operation is used instead. Next, the handwriting corresponding to the converted gesture is displayed at the location on the screen that is the operation target. After that, it is handled in the same way as the operation input as a gesture.

[0246] For example, character data enclosed by dotted line in the figure in FIG. 33 (a) is referred to as being selected by the gesture GE _371. In the present embodiment, it is assumed that the character data in the selected state is set to the Gothic typeface and the function designation is input by the keyboard, for example. Although the keyboard input is used in this embodiment, the function may be designated by menu selection with a mouse / pen, for example. When designating a function, for example, characters input from the keyboard are buffered, the input is terminated when the return key is input, and the character string is interpreted to obtain the content of the issued function. The gesture number corresponding to the issued function designation is obtained from the table (8) in Table 8. Here, for the sake of simplicity, one gesture number is given to one function, but an economical expression method can be adopted by an appropriate compression method.

[0247]

[Table 8]

In this embodiment, the gesture number 30002 is obtained from the "gothic typeface" in the table (8) showing the correspondence between the function and the gesture number. Next, the parametric graphic data corresponding to the gesture number is searched from the table (9) in Table 9. Then, parameters such as central coordinates of data to be processed by the function are added to the obtained graphic data. In this example, the center coordinates of the circumscribing quadrangle of the character data in the portion surrounded by the dotted line in FIG. 33 (a) in the selected state are given.

[0249]

[Table 9]

In the present embodiment, the parametric figure data "... gochi" is obtained from the gesture number 30002 in the table of this table (9). Next, the authenticity determination is performed based on the gesture number, and the process moves to the corresponding gesture display process.
In this embodiment, the designation of conversion to the Gothic typeface is issued as described above, and first, the display process of the Gothic typeface gesture is performed from the parametric graphic data and the corresponding parameter. As a result, the lead line GE shown in FIG. 33 (b) in which the character "gochi" is added to the end of the lead line
₃₇₂ and characters 373 are displayed. After that, the Gothic typeface is performed. In this embodiment, the GE ₃₇₁ shown in FIG.
By converting the character data enclosed in parentheses into a Gothic typeface, the numbers in the Gothic typeface are displayed as shown in (c) of FIG.

If the display of the generated gesture conflicts with the information on the existing screen and causes visual or operational inconvenience, it is possible to perform processing such as moving / deforming the generated gesture. Furthermore, it is feasible to include a mechanism for enabling / disabling this mechanism.

As described above, according to this embodiment, it is possible to understand at a glance what the operation was just done by looking at the screen. In addition, it is possible to suspend execution of commands input from the keyboard during batch execution type editing and display them on the screen. Furthermore, a series of operation procedures that have been input in advance can be displayed as gestures on the screen and can be confirmed and corrected before execution (using batch execution type editing).

FIG. 34 shows a flowchart of the operation procedure for displaying the above-mentioned executed operation. Here, for the sake of simplicity, the gesture executed by handwriting input is “select”,
There are only three types, "Gothic typeface" and "Cancel".

In FIG. 34, in step S701, it is determined whether or not there is an input from the keyboard. If it is determined that there is keyboard input (Yes), the process proceeds to step S710 described later, and if it is determined that there is no input. Advances to step S702.

In step S702, it is determined whether or not one stroke of handwriting has been input. If it is determined that one stroke of handwriting has been input (Yes), the process proceeds to step S721. Step S701 for (No)
Return to.

In step S721, it is determined whether the input handwriting of the one stroke is a selection gesture. When it is determined in step S721 that the gesture is the selection gesture (Y
es) proceeds to step S704, and when it is determined that the gesture is not the selection gesture (No), step S72 described below is performed.
Go to 2.

If it is determined Yes in step S721, the selection gesture is displayed in step S704, the operation of the selection gesture is executed in step S705, and the process returns to step S701.

If it is determined No in step S721, in step S722, it is determined whether the input handwriting is a Gothic typeface gesture. If it is determined in step S722 that it is a Gothic typeface gesture (Yes), the process proceeds to step S706, and if it is determined that it is not the Gothic typeface gesture (No), the process proceeds to step S723 described below.

If it is determined Yes in step S722, the gothic typeface gesture is displayed in step S706, the operation of the gothic typeface gesture is executed in step S707, and then the process returns to step S701.

Further, if it is determined No in step S722 above, it is determined in step S723 whether or not the input handwriting is a cancel gesture. If it is determined that the input gesture is a cancel gesture, the process proceeds to step S708, and the cancel gesture is performed. If it is determined that it is not, the process returns to step S701.

If it is determined Yes in step S723, the cancel gesture is displayed in step S708.
After the operation of the cancel gesture is executed in step S709, the process returns to step S701.

On the other hand, if it is determined Yes in step S701, the character input from the keyboard is input in step S710, and it is determined in the next step S711 whether or not the input from the keyboard is completed. Be seen. When it is determined in step S711 that the input is not completed (No), the process returns to step S710, and when it is determined that the input is completed (Yes), step S7 is executed.
Proceed to 12.

In step S712, the above-mentioned table (8)
The character string input in the above step S710 is searched from the table shown in (1) to obtain the corresponding gesture number. Next, in step S713, the figure data corresponding to the gesture number is obtained from the table showing the correspondence between gestures and figures shown in Table (9), and the flow proceeds to step S714.

At step S714, parameters such as size and position are added to the graphic data. The parameter to be added is calculated from the size of the display screen and the position of the data to be processed. Then, it progresses to step S715.

[0265] In step S715, it is determined whether or not the retrieved data is a selection gesture, and if it is determined in step S715 that it is a selection gesture (Yes), the process proceeds to step S704, and this step S704. Thereafter, the same processing as described above is performed. Also,
If it is determined in step S715 that the gesture is not the selection gesture (No), the process proceeds to step S716.

In step S716, it is determined whether or not it is a Gothic typeface gesture.
If it is determined in 16 that the gesture is a Gothic typeface gesture (Yes), the process proceeds to step S706, and after step S706, the same process as described above is performed. Furthermore, when it is determined in step S716 that the gesture is not a Gothic typeface gesture (No), the process proceeds to step S717.

In step S717, it is determined whether the gesture is a cancellation gesture. If it is determined that the gesture is a cancellation gesture in step S717 (Yes), the process proceeds to the step S708, and the same steps as those described above are performed. Processing is performed. Further, step S717
When it is determined that the gesture is not the cancel gesture (No), the process returns to step S701.

In this embodiment, as an example of the gesture-on-gesture operation described above, cancel, erase, lead line, input end instruction, object designation, direction (“upward feed”, “downward feed”, “right feed”). , "Left feed" gesture is drawn on top of (or starts from above) another gesture, it acts as a special gesture that modifies the movement of the gesture below. The procedure of the gesture-on-gesture in this embodiment is as follows.

First, in the case of the above cancellation gesture,
Recognize the gesture of cancellation, determine that it is a gesture drawn on the gesture (the gesture below the handwriting is gesture A), and if it is judged that it was drawn on the gesture, The gesture processing is switched to correct the operation of gesture A as follows. That is, the processing executed in the gesture A is canceled, and the gesture A does not perform any processing. In addition, in the gesture of normal cancellation,
The cancellation of the straight line processing is executed.

In the following procedure, when determining whether the input handwriting is drawn on a gesture, the input handwriting indicates the gesture even if the input handwriting and the gesture do not intersect. When it is clear that the input handwriting is drawn on the gesture.

In the case of the erasing gesture, the erasing gesture is recognized, it is determined that the gesture is drawn on the gesture (the gesture below the handwriting is referred to as gesture A), and the gesture is drawn on the gesture. If it is determined that the gesture has been inserted, the process of the erasing gesture is switched and the operation of the gesture A is corrected as follows. That is, the processing executed in the gesture A is canceled, and the gesture A does not perform any processing. It should be noted that in the normal erasing gesture, the figures and character files under the handwriting of the erasing gesture are erased.

In the case of the above-mentioned leader line gesture, the gesture of the leader line is recognized, it is determined that the leader line is drawn from another gesture (the gesture indicated by the starting point of the leader line is gesture A), and the gesture. When it is determined that it is drawn on the above, the processing of the gesture of the leader line is switched, and the operation of the gesture A is corrected as follows. That is, a window displaying the parameters of the process executed by the gesture A is opened in the window in which the leader line is opened, an input to the window is accepted in the same manner as a normal leader line gesture, and the gesture A according to the corrected parameter is input after the input is completed. Re-execute the process. When the operation is different when the process is re-executed and when the process is canceled and re-executed like the copy process, the process executed by the gesture A is canceled once and the process is performed again according to the previously specified parameter. Just run it. Further, when a plurality of types of gestures are candidates as the recognition result when the handwriting of the gesture A is recognized, the candidate list is also displayed in the window and used to change the type of the gesture A itself. You can also In that case, the process executed in the gesture A may be canceled once, and then the specified type of process may be executed. It should be noted that in the normal leader line gesture, reference and setting of attributes for graphics, character strings, and other data are performed.

In the case of the input end instruction gesture, the gesture of the input end instruction is recognized, it is determined that the gesture is drawn on the gesture (the gesture below the handwriting is referred to as gesture A), and the gesture is made. When it is determined that it is drawn on the above, the processing of the gesture of the input end instruction is switched, and the operation of the gesture A is corrected as follows. That is, when the execution of the process of the gesture A is suspended, such as when performing batch execution type editing, the suspended state of the gesture A is released and the process is immediately executed. In addition, when performing batch execution type editing, processing of all gestures that are suspended in the gesture of the normal input end instruction is executed.

In the case of the above-mentioned object-specified gesture, the object-specified gesture is recognized, it is determined that the gesture is drawn on the gesture (the gesture below the handwriting is referred to as gesture A), and the gesture is touched. If it is determined that the gesture is drawn in, the processing of the gesture designated by the object is switched, and the operation of the gesture A is corrected as follows. That is, it is determined whether or not there is a "next" process corresponding to the process executed in the gesture A, and if there is, the "next" process is executed. The “next” process is, for example, a “next matching string search” process for a character string search process, and a “next further” process for cancellation. The process of "canceling the previous process" is the process of "the next". In order to determine the presence / absence of the “next” process, a table indicating the correspondence between each process and the “next” process may be prepared in advance. It should be noted that in the gesture for designating a normal object, a process of bringing the instructed data into a selected state is executed.

In the case of the right feed and left feed gestures described above,
When right-feeding and left-feeding gestures are recognized, it is determined that the gesture is drawn on the gesture (the gesture below the handwriting is referred to as gesture A), and it is determined that the gesture is drawn on the gesture. Switches the processing of the right feed and left feed gestures and corrects the operation of gesture A as follows. That is, in the case of a right-feed gesture, it is determined whether or not there is the "next" process corresponding to the process executed in the gesture A, and if there is, the "next" process is executed. In the case of a left feed gesture, it is determined whether or not there is a process “before” corresponding to the process executed in the gesture A, and if there is, the process “before” is executed. The process “before” is, for example, the process “searching for the previous matching character string” is “before” for the process of character string search and “previous” for the process for canceling. The process "cancellation of the previous cancellation process" is the process "before". In order to determine the presence / absence of the process “before”, the same process as the process “next” may be performed. In order to make the correspondence between the direction of the handwriting of the right feed and left feed gestures and the processing natural, in accordance with the type of gesture A, the processing to switch to the "previous" for right feed and "the next" for left feed May be added.

In the case of the above-upward and down-feed gestures,
First, it recognizes the gestures of up-feed and down-feed, determines that it is a gesture drawn on the gesture (the gesture below the handwriting is called gesture A), and determines that it is drawn on the gesture. If done, top feed,
Switch the processing of the downward feed gesture, and make a gesture A
The operation of is modified as follows. That is, in the case of the upward feed gesture, it is determined whether or not there is the above-mentioned "above" process corresponding to the process executed in the gesture A, and if there is, the above-mentioned "above" process is executed. "Moreover"
For example, the process of “selecting a group at the next higher level” is the process of “above” for the process of selecting a group, and the process of searching for a character string in a horizontal document is The process of “searching for the previous matched character string” is the process of “above”. If there is no "above" process, it is determined whether or not there is the above-mentioned "before" process corresponding to the process executed in gesture A, and if there is, the above-mentioned "before" process is executed. In the case of the downward feed gesture, it is determined whether or not there is a process "below" corresponding to the process executed in the gesture A, and if there is, the process below "below" is executed. The process "below" is, for example, the process "selecting the group at the next lower level" is "below" for the process of group selection.
The process of "searching for the next matching character string" is a process below the process of searching for a character string in a horizontally written document. In order to determine the presence / absence of the processing “above” and “below”, the same processing as the case of the above “next” may be performed. The level of the group will be described later.

Specifically, the following is done.

First, the shape of the "cancel gesture" for canceling the immediately preceding operation is as shown in the above-mentioned table (3),
In this embodiment, the case of canceling the Kana-Kanji conversion gesture will be described.

For example, assume that character string data as shown in FIG. 35 (a) is displayed on the screen. At this time,
5 of handwriting PM ₃₉₂ for kana-kanji corresponding to convert the gesture as shown in (b) and the inputted kana-kanji conversion gesture corresponding to the handwriting PM ₃₉₂ when the kana-kanji conversion is performed by the kana gesture Causes the character string to be, for example, a character string 40 as shown in (c) of FIG.
Suppose it has been converted to 3. In this example, kana-kanji conversion by handwriting PM ₄₀₄ is also performed at the same time. That is,
In this example, it is assumed that an incorrect conversion was made in the Kana-Kanji conversion performed first. In this case, as shown in FIG. 35D, the handwriting PM ₄₁₃ having the above-mentioned “cancel” gesture shape.
On the above Kana-Kanji conversion gesture handwriting PM ₃₉₂ . As a result, the gesture under the handwriting PM ₄₁₃ is searched. In this case, the Kana-Kanji conversion gesture is discovered, and the operation of canceling the operation of the Kana-Kanji conversion gesture is performed. The display after the cancellation is executed is as shown in FIG. It is also possible to add a mechanism for deleting the canceled gesture from the screen.

In addition to the above examples, it is possible to correct the gesture in the following combinations, for example. That is, arbitrary gesture + erase → gesture cancellation arbitrary gesture + leader line → confirmation / correction of processing parameters or list of gesture recognition candidates Menu conversion gesture + next feed → next candidate conversion gesture + left feed → previous candidate conversion Gesture + right feed → next candidate conversion gesture + up feed → previous candidate group display conversion gesture + down feed → next candidate group display cancel gesture + next feed → one more operation cancel cancel gesture + up feed → cancel operation one step Cancel gesture + Downward feed → Cancel yet another operation Select gesture + Upward feed → Expand the range of selection Range Selective gesture + Downward feed → Decrease the level of selection range.

[0281] Note that, as described above, selection + upward feed, or
Examples of the level of the selection range at the time of selection + downward feed include the following.

For example, in the case of text data,
In the case of Japanese, for example, 1 character → 1 from the lower level.
Phrase → 1 sentence → 1 paragraph (in the case of a section with a heading, 1 item → 1 paragraph → 1 chapter) → move to a higher level like whole sentence. In the case of English, for example, 1 character → 1 word →
1 sentence → 1 paragraph (in the case of a section with a heading, 1 item → 1 section → 1 chapter) → move to a higher level like full sentence.

In the case of graphic data, the graphic data moves from the lower level to the higher level in the order of, for example, element graphic → parts → group → large group → large / large group → ... → All figures. The element graphic is a unit of the finest graphic such as a line segment, a polygonal line, an arc, an ellipse, and a spline that can be handled by software. A part is a collection of element figures. However, some software does not have the level of parts. Usually, element graphics are collected to form a meaningful figure. For example, there is a line segment or a polygonal line like the part "arrow". Of course, one element graphic may be a part. For example, a component "resistor" composed of a polygonal line. These are usually entered by referring to a database. Element graphics input by handwriting may be treated as parts.

The group is a collection of some (may be one) element figures, parts, and groups.
Groups can also be made up of groups, and containment relationships can occur between groups. In this case, the included group is called a lower group, and the included group is called an upper group. Higher level of figure data,
The lower relationships are as shown in FIG.

As described above, according to the present embodiment, it is possible to accurately execute a gesture-based process using a lead line, and the number of processes that can be executed by a combination of gestures is increased. Therefore, the gesture shape to be remembered is reduced. be able to. Also, until now, the last processing (in order)
I could only cancel it, but I will be able to cancel any process that has handwriting on the screen. Furthermore, it is possible to easily distinguish between the “process of canceling the cancellation process” and the “process of canceling the cancellation process”. It should be noted that "cancel to the end" is to input the next feed gesture on the cancel gesture, and "to cancel cancel" is to input the erase gesture on the cancel gesture.

FIG. 37 shows a flowchart of specific processing of the above-described gesture-on gesture.

In FIG. 37, in step S81, the input of one stroke of handwriting is waited, and when there is one stroke of handwriting input, the input handwriting is smoothed in step S82.

After smoothing in step S82, it is determined in step S83 whether or not there is a handwriting whose recognition is suspended among the handwriting input before the handwriting input. If there is a handwriting held in step S83 (Yes), the process proceeds to step S85 described later, and if not (No), the process proceeds to step S84.

At the step S84, the handwriting input (one handwriting input) is recognized. For example, system R
The handwriting input is recognized using the gesture recognition table held in the OM 52 or the like. That is, the gesture number Gno corresponding to the shape of the handwriting of one stroke is recognized.
Perform a process that returns. If there is no gesture number Gno corresponding to the shape of the handwriting of one stroke in the gesture recognition table, an unregistered code (a number that means unregistered) is returned.

After step S84, step S8 is performed.
Proceed to 7. In step S87, it is determined whether or not the gesture number Gno is not registered.
That is, in step S87, the above step S84 is performed.
If it is determined that the unregistered code is returned from step S87, and if it is determined that the code is not registered in step S87 (Yes), the process proceeds to step S88, and it is determined that the code is not registered (registered). If yes (No), the process proceeds to step S89. In step S88, processing corresponding to the unregistered figure, character, etc. is performed. After step S88, the process returns to step S81.

If it is determined Yes in step S83, the handwriting input is not one stroke, so step S8
In 5, the recognition of multiple strokes is performed. In this step S85 as well as in the above step S84, for example, the system ROM
The multi-handwriting gesture recognition table held in 52 or the like is used to recognize the shape of the multi-handwriting. After this step S85, the process proceeds to step S86. In step S86, similarly to step S87, it is determined whether or not the gesture number Gno is not registered (unregistered). If Yes is determined, the process proceeds to step S84. If No is determined. Step S89
Proceed to.

The above step S87 or step S86
If No is determined and the process proceeds to step S89, it is determined in step S89 whether or not the gesture number Gno is undetermined (undetermined). If it is determined in step S89 that it is unconfirmed (Yes), the process proceeds to step S820, and if it is determined that it is not unconfirmed (confirmed) (No), the process proceeds to step S812. It should be noted that, in step S820, after registering in the recognition hold handwriting, the process returns to step S81.

In step S812, it is determined whether or not the input handwriting has a gesture. If it is determined that the gesture is present (Yes), the process proceeds to step S814, and if it is determined that there is no gesture (No). If so, the process advances to step S813.

If it is determined No in step S812, normal gesture processing corresponding to the gesture of the input handwriting is performed in step S813, and then the process returns to step S81.

If it is determined Yes in step S812, the type of gesture under the input handwriting is acquired in step S814, and the gesture number Dn is added to the gesture.
o is given (the gesture number of the gesture found by the search is set to Dno), and in step S815, in order to obtain a procedure to be executed as a gesture-on-gesture from the gesture of the input handwriting and the gesture number of the gesture below the input handwriting. The procedure number k corresponding to the gesture number Dno and the gesture number Gno is obtained from the two-dimensional array (table) GGtbl in which the procedure number is registered in advance.

After that, in step S816, the procedure corresponding to the procedure number k is registered from the one-dimensional array (table) GGfunc in which the entries of the procedure for actually performing various gesture-on-gesture operations are registered in advance in the order of gesture numbers. The storage address in the ROM 52 of No. 1 is obtained and the procedure is executed. After this step S816, the process returns to step S81.

The tables GGtbl and GGfunc in the above-mentioned flowchart will be described below.

Here, for example, a case where seven gestures of “data selection”, “kana-kanji conversion”, “lead line”, “upward feed”, “downward feed”, “cancel”, and “erase” are implemented. For example, the contents of each array are shown. GG
tbl has the content shown in Table 10 (10). Vertical axis is Dn
The value of o, the horizontal axis is the value of Gno, and the content of the table is the value of k. Although the type of gesture is entered on both the vertical and horizontal axes, Gno and Dno are actually numerical values indicating the corresponding gesture. GGfunc has the content shown in Table 11 of Table 11. The part of the content of the procedure is actually R in FIG.
The addresses where the procedures are stored are lined up on the OM 52.

[0299]

[Table 10]

[0300]

[Table 11]

The table GGfunc shown in Table 11 is used.
In the case where the procedure number 20 is used to cancel a number of operations using the procedure number 9 procedure, all the cancellations are canceled at once and there is no cancellation gesture.

Also, for other gesture-on gestures, the same as above can be realized if the procedure number is correctly registered in the table.

The procedure for the immediate execution gesture in this embodiment is as follows. First, the shape of the handwriting is recognized and the type of gesture to be executed is obtained. Here, if the type of gesture obtained as a result of recognition is a special type that is set in advance, the gesture is immediately executed even while the batch execution type editing function is being used.
For other types, the gestures are accumulated so as to be executed collectively later.

Here, a description will be given using a character erasing gesture as shown in FIG. 38A and a canceling gesture as shown in FIG. 38B.

Here, for example, it is assumed that the "batch execution type editing" function is already turned on. On the screen, for example,
It is assumed that the character string data shown in (a) is displayed. At this time, for example, the handwriting P as shown in (b) of FIG.
If you input M (character string deletion gesture), the character string deletion gesture is not an immediate execution gesture.
The corresponding process is once registered and accumulated in the process registration buffer prepared on the RAM 55 of FIG. In this case, since the process has not been actually executed, the screen does not change. However, it is also possible to delete the input handwriting and then print out the recognized gesture in a clean copy. Next, if the above-mentioned character string erasing gesture is incorrect (character string deletion is incorrect), for example, (c) of FIG.
The handwriting PM ₄₇₂ shown in Fig. 39 is used as the handwriting PM in Fig. 39 (c).
₄₇₁ (Handwriting PM in FIG. 39 (b)). The cancellation gesture corresponding to the handwriting PM ₄₇₂ is an immediate execution gesture, and the procedure corresponding to the immediate execution gesture is executed.

In the case of the present embodiment, the RO is determined in order to determine whether or not the gesture is an immediate execution gesture, and to determine the processing to be executed when the gesture is an immediate execution gesture.
A one-dimensional array (table) is prepared on M52. This table is called IGtbl. Each element of IGtbl stores a number indicating the type of processing to be executed corresponding to the immediate execution gesture. A number representing "non-immediate execution", for example, 0 is stored in the element corresponding to the gesture other than the immediate execution type gesture. For example, it is assumed that the gesture number corresponding to the gesture of deleting a character string is 3.
The erase string gesture is not an immediate gesture.
Therefore, 0 indicating "non-immediate execution" is stored in the third element of IGtbl. Further, if the gesture number of the cancel gesture is 5, for example, the cancel gesture is an immediate execution type gesture.
The fifth element of bl stores a number other than the number indicating "non-immediate execution", for example, 10. By using the above table, it is possible to determine whether or not the input gesture is an immediate execution type gesture, and when it is an immediate execution type gesture, the type of processing to be executed can be obtained by a number. . In the case of the present embodiment, the ROM 5 of the procedure for performing the process is calculated from the number indicating the type of process obtained by the IGtbl.
A table is prepared on the ROM 52 to obtain the storage address on the memory 2. This is called IGfunc. IGfu
In each element of nc, the storage address on the ROM 52 of the processing procedure to be executed corresponding to the immediate execution gesture is stored. For example, if a cancellation gesture is input and a number representing the processing type of 10 is obtained by IGtbl, if the tenth element of IGfunc is referenced, the storage address on the ROM 52 of the procedure for performing the processing corresponding to the cancellation gesture. , For example, address 6500 can be obtained. Table (12) of Table 12 shows I in the above example.
Gtbl is shown in Table 13 (13) and IGf in the above example.
It shows the contents of unc.

[0307]

[Table 12]

[0308]

[Table 13]

When the operation of the immediate execution gesture is completed, the screen display returns to the state shown in FIG. 39 (a). Note that the immediate execution gesture operates immediately even during batch execution type editing.

From the above, in this embodiment,
Even when an erroneous editing operation is input by a gesture during batch execution type editing, the gesture can be corrected without switching the operation. In addition, by immediately correcting an erroneous editing operation, only the editing symbol to be executed remains on the screen, so that the final editing operation can be easily understood only by looking at the screen.

FIG. 40 shows a flowchart of the specific operation of the above-described immediate execution gesture.

In FIG. 40, a pen input or a keyboard input or the like is awaited in step S91, and when an input is made in step S91, the process proceeds to step S92.

In step S92, it is determined whether the input is a handwriting from a pen. This step S9
When it is determined that the input is handwriting in 2 (Yes), the process proceeds to step S94, and when it is determined as No, the process proceeds to step S93. In step S93, since the input is not a handwriting, a process corresponding to the input (for example, keyboard input) is performed. After step S93, the process returns to step S91. In step S94, the handwriting input is smoothed, and then the process proceeds to step S95.

After the smoothing in step S94, it is determined in step S95 whether or not there is a handwriting whose recognition is suspended among the handwriting input before the handwriting input. If the pending handwriting is found in this step S95 (Yes), the process proceeds to step S97 described later, and if not (No), the process proceeds to step S96.

In step S96, the handwriting input (one handwriting input) is recognized. For example, system R
The handwriting input is recognized using the gesture recognition table held in the OM 52 or the like. That is, the gesture number Gno corresponding to the shape of the handwriting of one stroke is recognized.
Perform a process that returns. If there is no gesture number Gno corresponding to the shape of the handwriting of one stroke in the gesture recognition table, an unregistered code (a number that means unregistered) is returned.

After the above step S96, step S9
Proceed to 9. In step S99, it is determined whether or not the gesture number Gno is not registered (unregistered). That is, in step S99, it is determined whether or not the unregistered code is returned from step S96. If it is determined that the code is unregistered in step S99 (Yes), the process proceeds to step S910 to unregister. If it is determined that it is not (registered) (No), the process proceeds to step S911. In step S910, processing corresponding to the unregistered figure, character, etc. is performed.
After the step S910, the process returns to step S91.

If it is determined Yes in step S95, the handwriting input is not one stroke, and thus step S9 is performed.
In 7, the recognition of multiple handwritings is performed. In this step S97 as well as in the above step S96, for example, the system ROM
The multi-handwriting gesture recognition table held in 52 or the like is used to recognize the shape of the multi-handwriting. After this step S97, the process proceeds to step S98. In step S98, similarly to step S99, it is determined whether or not the gesture number Gno is not registered (unregistered). If Yes is determined, the process proceeds to step S96, and if No is determined. Step S91
Go to 1.

The above step S99 or step S98
When it is determined No in step S911 and the process proceeds to step S911, it is determined in step S911 whether or not the gesture number Gno is unconfirmed (unconfirmed). If it is determined in step S911 that it is unconfirmed (Yes
s), the process proceeds to step S912, and when it is determined that it is not undetermined (confirmed) (No), step S913.
Proceed to. In step S912, after registering the recognition hold handwriting, the process returns to step S91.

In step S913, it is determined whether the batch execution mode is set or not. If it is determined that the batch execution mode is set (Yes), the process proceeds to step S915. If it is determined that the batch execution mode is not set (No), step S914 is set.
Proceed to.

If it is determined No in step S913, a normal gesture process that is not the collective execution mode is performed in step S914, and then the process returns to step S91.

If it is determined Yes in step S913, the gesture number Gn of the handwriting is calculated in step S915.
From o, the procedure number k corresponding to the process to be executed as the immediate execution type gesture is obtained from the one-dimensional array (table) IGtbl in which the procedure number is registered in advance.

After step S915, step S
Proceed to 916. In step S916, it is determined whether the procedure number k is a non-immediate procedure. When it is determined in this step S916 that it is not non-immediate (Yes)
To step S917, and if it is determined to be non-immediate (No), the process proceeds to step S918.

In step S917, the non-immediate gesture is registered in the above-described process registration buffer, and then step S9.
Returning to step 1, in step S918, the storage addresses on the ROM 52 in FIG. 1 of the procedure for actually performing various gesture operations are registered in advance in the order corresponding to the element values of IGtbl from the one-dimensional array (table) IGfunc. The storage address of the procedure corresponding to the procedure number k in the ROM 52 of FIG. 1 is obtained, and the procedure is executed. After this step S918, the process returns to step S91.

IGtbl of Table (12) and I of Table (13) described above.
If Gfunc is used, it is 6 in step S918.
The procedure stored at address 500 is executed. FIG. 41 shows a flowchart of the procedure stored in this 6500 address. In FIG. 41, step S1001
Then, the gesture displayed in the input handwriting is searched. In the next step S1002, the above step S10 is performed.
At 01, it is determined whether or not the gesture is found. Here, if the gesture is found (Yes), the process proceeds to step S1003, and if it is not found (N).
In step o), the process proceeds to step S1004 described below. In step S1003, the discovered gesture is set as a processing target, and the process proceeds to step S1005. If it is determined Yes in step S1002, the processing target has not been directly specified. Therefore, in step S1004, the last input gesture is set as the processing target, and step S1004 is performed.
Proceed to 1005. Step S1003 or step S
When the process proceeds from step 1004 to step S1005, cancellation processing for the target gesture is executed. In this cancellation processing, processing for deleting both the target gesture and the input handwriting is also performed. Note that step S100
In 4, the processing other than the gesture may be set.

In this embodiment, the tables (3) to 3 in Table 3 to the tables in Table 6 are used.
By using the shape shown in (6) and using the comprehensive gesture set consisting of the above-mentioned gestures, the mechanisms corresponding to the above-mentioned gestures can coexist without contradiction. That is, all the above-mentioned mechanisms can be used by appropriately connecting the above-mentioned flowcharts. Further, the shapes used for the respective gestures can be sufficiently distinguished by inclining in arbitrary directions and drawing. Therefore, it becomes possible to construct an operation system in which a gesture can be written from any direction. In addition, you can use all of the above-mentioned convenient mechanisms by simply remembering the small number of shapes that you normally use on paper, and you can use it even in batch execution type editing without conflict.

From the above, according to the present embodiment, each gesture of Table (3) of Table 3 to Table (6) of Table 6 can handle vertical writing and horizontal writing like Japanese, Gestures have the same meaning when used in different orientations. In addition, gestures that allow quick drawing of frequently used instructions for conversion and correction are also available. Furthermore, it is possible to operate the device as a whole with only basic gestures, and define basic functions whose meaning does not change depending on the processing target,
The meaning of gesture is unified in various situations. Therefore, it becomes possible to perform appropriate processing according to the situation in which the gesture is input.

[0327] Also, for example, it is possible to call the auxiliary menu quickly by using a lead line, and since the target is specified, the contents of the auxiliary menu can be easily limited and selected, and the menu position can be set by the user. Can be set freely.

Further, for example, the immediately preceding process can be easily repeated using the "execute" gesture, and the "next" can be easily repeated using the direction gesture.

In addition, the user can move from the existing gesture set without difficulty, can be used for batch execution type editing without contradiction, and the meaning can be combined and corrected by drawing the gesture in the gesture. In addition, it is possible to support a gesture of writing together characters.

[0330]

As described above, in the handwriting input information processing apparatus of the present invention, the editing and proofreading object is displayed, the coordinate data corresponding to the handwriting is generated, and the handwriting shape is generated based on the generated coordinate data. The editing proof symbol that most closely resembles the recognized handwriting shape is identified from among multiple predefined editing proofing symbols regardless of the directionality of the handwriting shape. By performing edit proofreading processing on the edit proofreading object according to the assigned function, it is possible to handle vertical writing, horizontal writing and Japanese, and further improve operability. .

[Brief description of drawings]

FIG. 1 is a block circuit diagram of a signal processing circuit system of a device of this embodiment.

FIG. 2 is a diagram showing the overall configuration of the apparatus of this embodiment.

FIG. 3 is a diagram for explaining an example of designating a string-shaped section sandwiched between two gestures in the range designation.

FIG. 4 is a diagram for explaining an example of designating a rectangular range in the range designation.

FIG. 5 is a diagram for explaining an example of a combination that does not form a rectangle in the range designation.

FIG. 6 is a diagram for explaining designation of directions.

FIG. 7 is a diagram showing a shape of a kana-kanji conversion gesture corresponding to horizontal writing.

FIG. 8 is a diagram showing a shape of a kana-kanji conversion gesture corresponding to vertical writing.

FIG. 9 is a diagram for explaining an example of performing kana-kanji conversion on a horizontally written character string.

FIG. 10 is a diagram for explaining an example of performing kana-kanji conversion on a vertically written character string.

FIG. 11 is a diagram illustrating an example of performing kana-kanji conversion on a character string having a special layout.

FIG. 12 is a diagram for explaining another example in which kana-kanji conversion is performed on a character string having a special layout.

FIG. 13 is a flowchart showing an operation of a kana-kanji conversion gesture.

FIG. 14 is a diagram illustrating an example of a gesture having a similar shape.

FIG. 15 is a diagram illustrating an example of a character string and a gesture when distinguishing gestures having similar shapes.

FIG. 16 is a flowchart showing the overall operation of distinguishing gestures having similar shapes.

FIG. 17 is a flowchart of determination area calculation of function A.

FIG. 18 is a diagram for explaining a determination region of a function A when distinguishing similar shapes in the present embodiment.

FIG. 19 is a flowchart of a determination area calculation of a function B.

FIG. 20 is a diagram for explaining a determination region of a function B when distinguishing similar shapes in the present embodiment.

FIG. 21 is a flowchart of the inspection of the direction, size, and position of the function B.

FIG. 22 is a diagram for explaining a determination area for a function A and a function B.

FIG. 23 is a diagram showing a shape of a gesture for converting a gesture into normal coordinate data and converting a normal coordinate data into a gesture.

FIG. 24 is a diagram showing an example of a converted gesture (trajectory).

FIG. 25 is a diagram for explaining a procedure of converting a gesture into normal coordinate data.

FIG. 26 is a flowchart showing an operation of converting a gesture to normal coordinate data and converting a normal coordinate data to a gesture.

FIG. 27 is a diagram showing the shape of a gesture of a lead line.

FIG. 28 is a diagram for explaining an operation of a leader line gesture and a leader line gesture.

FIG. 29 is a diagram showing a modified example of a leader line gesture.

FIG. 30 is a flowchart showing an operation of a leader line gesture.

FIG. 31 is a diagram for explaining an input end instruction gesture.

FIG. 32 is a flowchart showing an operation of an input end instruction gesture.

FIG. 33 is a diagram for explaining display of executed operations.

FIG. 34 is a flowchart of an operation of displaying executed operations.

FIG. 35 is a diagram illustrating a gesture-on gesture.

[Fig. 36] Fig. 36 is a diagram for explaining upper and lower sides of graphic data.

FIG. 37 is a flowchart showing an operation of gesture-on gesture.

FIG. 38 is a diagram showing an example of an immediate execution gesture.

FIG. 39 is a diagram for explaining a procedure of an immediate execution gesture.

FIG. 40 is a flowchart showing an operation of an immediate execution gesture.

[Fig. 41] Fig. 41 is a flowchart illustrating an operation of immediate cancellation of a motion of an immediate execution gesture.

[Explanation of symbols]

 1 ... Liquid crystal display device 2, 54 ... Keyboard 3, 60 ... Stylus pen 51 ... CPU 52 ... System ROM 53 ... Keyboard controller 55 ... Work RAM 56 ··· Video RAM 57 ··· LCD controller 58 ··· Tablet controller 59 · · · Display · tablet block

Claims (15)

[Claims] 1. A display unit for displaying an editing and proofreading object, a coordinate inputting unit for generating coordinate data according to handwriting, and a plurality of editing and proofreading data in which coordinate data generated by the coordinate inputting unit are defined in advance. From among the symbols, recognized based on the editing proofing symbol most similar to the recognized handwriting shape, the symbol discriminating means for identifying regardless of the directionality of the handwriting shape, and the editing proofing symbol identified by the symbol discriminating means. An handwriting input information processing apparatus, comprising: an editing and proofreading processing unit that performs an editing and proofing process on the editing and proofing target according to an assigned function. 2. A function that interprets the function assigned to the edit proof symbol specified by the symbol determining means and changes according to the attribute of the edit proof object adjacent to the writing position of the handwriting shape, A function interpreting unit that uniquely interprets a function according to the orientation of the handwriting shape with respect to the editing and proofreading object, wherein the editing and proofreading processing unit interprets the function with respect to the editing and proofing target. The handwriting input information processing apparatus according to claim 1, wherein editing and proofreading processing corresponding to the function is performed. 3. The editing proof symbol, when the editing proofreading object is a character, specifies a range from a start point to an end point for converting the editing proofreading object into kana-kanji characters and a kana-kanji conversion instruction in a continuous handwriting. The handwriting input information processing apparatus according to claim 1, wherein the handwriting input information processing apparatus is a kana-kanji conversion symbol. 4. The handwriting input information processing apparatus according to claim 3, wherein the kana-kanji conversion symbol has a shape that is counterclockwise to the end point. 5. The handwriting input information processing apparatus according to claim 3, wherein the kana-kanji conversion symbol has a shape that is clockwise at the end point. 6. The handwriting input information processing apparatus according to claim 3, wherein the kana-kanji conversion symbol has a shape that is counterclockwise from the starting point. 7. The handwriting input information processing apparatus according to claim 3, wherein the kana-kanji conversion symbol has a shape in which the starting point is clockwise. 8. The handwriting input information processing apparatus according to claim 1, wherein the edit proof symbol is a conversion symbol for instructing conversion from an edit proof processing command to coordinate data. 9. The handwriting input information processing apparatus according to claim 1, wherein the edit proof symbol is a conversion symbol for instructing conversion of coordinate data into an edit proof processing command. 10. The edit proof symbol is a conversion symbol for instructing a conversion from an edit proofreading processing command to coordinate data and a conversion from a coordinate data to an edit proofreading processing command. 9. The handwriting input information processing device according to item 9. 11. The handwriting input information processing apparatus according to claim 1, wherein the edit proof symbol is a leader line symbol for setting a parameter according to an attribute of an edit proof. 12. The handwriting input information processing apparatus according to claim 1, wherein the edit proof symbol is an input end symbol for instructing the end of input. 13. The information according to the edited proofreading symbol corresponding to the executed editing proofreading process is held, and the information according to the edited proofreading symbol is displayed on the display means. Handwriting input information processing device. 14. A function of modifying the function of the previously inputted edit proof symbol with the inputted edit proof symbol on the display of information according to the previously inputted edit proof symbol. The handwriting input information processing apparatus according to claim 13. 15. A function of immediately executing the function of a specific edit calibration symbol for a specific edit calibration symbol when collectively performing the edit calibration process corresponding to each function of a plurality of edit calibration symbols. The handwriting input information processing apparatus according to claim 1, further comprising: