JPH07104964A - Selection determining method - Google Patents

Abstract

PURPOSE:To solve the location deviation at the time of double clicks by acquiring a series of positional information and status information as stroke information and performing a gesture recognition processing when the status information from a pointing device is on and positional information is in a gesture recognition possible area. CONSTITUTION:A location/status detection part 11 receives the positional information and status information from a tablet 21 and converts it into the information that a computer requires according to conditions. A gesture recognition part 13 receives stroke data from the location/status detection part 11 and executes a processing that stroke data is recognized as a gesture. A command processing part 14 issues the code of a command according to the result that the gesture recognition part 13 recognizes and passes the code to an application processing part 15. The application processing part 15 makes the processing result be displayed via a display control part 12 and makes the result be outputted to a printer 23 via a printing output part 16 as necessary.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to input / output control of a computer, and more particularly to a user interface, i.e., when an application program of the computer requires item selection and decision by an operator in order to proceed. The present invention relates to a method for controlling or processing a signal from a display device and controlling an output to a display device.

[0002]

2. Description of the Related Art As a first conventional example, there has been a method of determining a selection using a keyboard shown in FIG. When the computer requires item selection and determination by the operator in order to proceed with the processing of the application program, the computer displays a plurality of items (“selection item 1” in the figure as shown in FIG. 4A) on the display screen. , Etc., but specifically,
Items that need to be selected in each application program, such as "12 points" and "9.5 points" when selecting font size. ) Is displayed. Normal menu screen display or
This is called opening the dialog box or zooming. In the present application, the term "item selection and decision" will be considered including the case where a plurality of application programs are prepared and which one is to be started, that is, the case of so-called menu program selection decision. .

In the initial state when the screen of FIG. 4A is opened, "selected item 1" is normally displayed in a state different from other items. The different states are displayed by displaying different colors in the case of a color display, and by displaying in reverse or thicker the frame in the case of a monochrome display. This part, which is displayed in a state different from other items, may be simply referred to as a cursor, but here, the shape is particularly elongated in the horizontal direction, and it is referred to as a cursor bar 24. . When the selection item that the operator desires to select is “selection item 5”, the operator operates the arrow key 40 of the keyboard and presses the downward arrow four times. Then, the computer receives the signal as a command for screen operation, and moves the cursor bar 24 on the display screen to the position of the selection item 5 (the operator can move to different positions in the display state one after another). It looks like the cursor bar 24 is moving).

When the cursor bar 24 moves to the position desired by the operator, the operator presses the execute button 41 to determine the selection of the item (see FIG. 4B). Here, the execute button is sometimes called a decision button or a confirmation button, and in the first conventional example for making a selection decision by keyboard operation, one of the keys of the keyboard, for example, The command is assigned to one of the return key and the function key, and when the operator depresses the key, the computer receives the signal and processes it as a decision command.

As a second conventional example, there has been a method for determining selection by using a mouse. FIG. 5 is a diagram showing a second conventional example.
The mouse 42 is an input device (pointing device) configured to send position information (relative coordinates) and status information (click signal) to a computer.
The display screen of FIG. 5 is almost the same as that shown in FIG.
Although not shown, when the mouse 42 is used,
A so-called mouse cursor is displayed on the screen. The mouse cursor is, for example, an index in the shape of a human finger and indicates a position pointed by the mouse. When the operator moves the mouse, the mouse cursor moves on the screen and the cursor bar shown in FIG. 5A moves.

[0006] Normally, a mouse is provided with two buttons, a right button and a left button. Pressing the left button once is called a click, and pressing the left button twice in succession is called a double click. I'm calling. In the selection decision method with the mouse, in order to prevent erroneous operation,
Items are often selected and determined by an operation of selecting by clicking (see FIG. 5A) and determining by double clicking.

As a third conventional example, there is a mouse emulation (pseudo mouse mode) using a stylus pen. The stylus pen is used as a position indicator of a coordinate detecting device called a digitizer or a tablet. The coordinate detection device has a specific position detection surface regardless of whether it is a pressure-sensitive type, an electromagnetic induction type, an optical type, or the like, and corresponds to a position pointed by a position indicator on it. The input device is configured to output the coordinates to be transmitted to the computer side. The position indicator of the coordinate detection device is
There are a general-purpose cursor having a puck shape and a stylus pen having a writing instrument shape. In the present application, when a pointing device is mentioned, a cursor as a position indicator of the coordinate detection device,
Think of it as a concept that includes a stylus pen and a mouse.

Just as a mouse is provided with a button for sending status information to the computer, the cursor or stylus pen as a position indicator also has a switch, and the computer has not only coordinate information but also switch information. Is configured to deliver. In the case of a stylus pen, a switch that detects the pressing force of the pen tip, a switch that detects the pressing force in the direction perpendicular to the pen shaft (side switch), or both of them is provided. Is sent to the computer (via the coordinate detection device).

In the selection determining method (mouse emulation mode) in which the stylus pen is used instead of the mouse, the XY coordinates (absolute coordinates) detected by the coordinate detecting device are the tablet built in the computer. It is converted to relative coordinates by the driver's action and passed to the mouse driver. Further, the status information regarding the pressing of the pen tip is converted into a signal for pressing the left button of the mouse by the function of the tablet driver, and is passed to the mouse driver. Here, the tablet driver and the mouse driver are so-called device drivers, which are programs installed as software of the computer. As a result, similarly to the mouse operation, the selection determining method can be realized by the stylus pen or the cursor as the position indicator.

[0010]

In recent years, a stylus pen has been attracting attention from the viewpoint of man-machine interface, but it is more than just a mouse emulation, a selection method more suitable for a writing instrument-shaped position indicator. The inventor of the present invention considered that there must be. A first object of the present invention is to provide a selection determination method suitable for a stylus pen.

Further, one of the problems in performing mouse emulation with a stylus pen is the problem of misalignment during double clicking. In the case of a mouse, it is not so difficult to click the left button quickly twice without moving the mouse. However, when double-clicking with a stylus pen, the indicated position usually shifts between the first click and the second click. Therefore, in order to detect the double click, it is necessary to perform a process having a flexibility to accept a certain amount of displacement. The inventor of the present invention thought that the processing ability could be devoted to a processing more suitable for a stylus pen if the processing needs to be performed softly. It is a second object of the present invention to provide a selection determining method which solves the problem of displacement when the stylus pen is double clicked.

Furthermore, the most important feature of the stylus pen is that it is not only used as a pointing device, but also as a tool for inputting handwritten characters. In the application as a pointing device, coordinate values were mainly treated as individual data and combined with status information to realize a selection determination method. However, in the case of handwritten character input, the information of the combination of coordinate values and status information is not individual data, but
It is treated as a series of data (stroke data), and its continuity is given meaning. The problem of separating the mouse emulation (or pointing, to put it a little broader) from the acquisition of stroke data (inking) when handling a stylus pen is a common problem for those who aim for pen computing. A third object of the present invention is to solve the problem of separating pointing and inking. Here, the stroke data is information that is sometimes referred to as ink data, and is information that gives a meaning to the connection of coordinate values from pen down to next pen up. This stroke data is the target of character recognition processing and gesture recognition processing.

As a result of analyzing the problems as described above, the inventor of the present invention has thought that the gesture command may provide a selection determining method for solving these problems. In the past, there was no idea to provide a plurality of areas in which the gesture command can be recognized and use it for selection determination.
However, when considering only the problem of selecting and determining a plurality of items, the stroke data is processed together with the information indicating in which area the gesture command has been made by providing a plurality of areas in which the gesture command can be recognized. I thought that this would solve the above three purposes.

[0014]

In order to achieve the above object, in the selection and determination method according to the present invention, the operator operates the pointing device while checking the display screen to obtain the position information and the status information. In a computer selection / determination method of inputting to a computer, and based on which the computer selects and determines a desired item of an operator from among a plurality of items, an item display area for displaying the plurality of items is provided on the display screen. Together with each item display area, a gesture recognizable area is provided in the movable range of the pointing device corresponding to the item display area, and the display state of the item display area is changed according to the change of the position information from the pointing device. Status information from the towing device is on and the location information is the When the area is a recognizable area, a series of position information and status information is acquired as stroke information, and the gesture recognition processing is executed for the stroke information, and the item corresponding to the gesture recognizable area is displayed according to the result of the gesture recognition processing. It is a choice and a decision.

In the above structure, preferably, the condition for acquiring the stroke information is not only the status information from the pointing device being on and the position information being the gesture recognizable area, but in addition to that, It is assumed that the change in the position information is a predetermined value or more.

Further, in the above structure, it is desirable that
The dust recognizable area has an elongated rectangular shape, and the target of the dust recognizing process is a line segment extending in the longitudinal direction of the dust recognizable area.

In the above structure, preferably, the pointing device is a stylus pen.

Further, in the above structure, preferably, the position detection surface of the stylus pen and the display screen are arranged in a stack, and the selection item display area and the vestier recognizable area are overlapped with each other.

[0019]

Since the selection / determination method according to the present invention is configured as described above, the selection / determination command can be input by operating the pointing device, and the stroke data used for the gesture recognition can be input. Since it is possible to simultaneously input the information indicating which selection item the coordinate information held by has been selected, it works to improve the operability.

Further, when the stroke length used for the gesture recognition is equal to or longer than a predetermined length, the stroke data is acquired to avoid an input due to an erroneous operation.

By making the gesture recognition area into a long and narrow rectangular shape and making the recognition target a line segment extending in the longitudinal direction, the selection item is expressed by a character string, and the gesture command for selecting and determining the selection item is simple. Make something

Using the pointing device as a stylus pen provides a rational fusion of pointing and inking.

Laminating and arranging the position detection surface and the display screen works so as to match the visual recognition position and the input position,
Realize a selection decision method more suitable for a stylus pen.

[0024]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of a selection determining method according to the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram for explaining an embodiment of the selection determining method according to the present invention in terms of operations performed by an operator. In the most preferred embodiment, the pointing device 20 used is a stylus pen (hereinafter abbreviated as "pen" as needed), and the position detection surface of the tablet and the display screen of the display device are laminated. Will be placed. Also, the tablet is not only when the pen is down, but also when the pen is floating, that is, when the position detection surface is approaching at a certain height, although it is not on the position detection surface. It is assumed that the coordinates are read and sent to the computer even when the force is not applied enough to push the pen tip switch. For example, "Pentop" (pentop is a trademark of Wacom Co., Ltd.) provided by the same applicant as this application satisfies this condition. in this case,
The operator takes the pen tip of the stylus pen on the desired selection item without making any distinction between the selection operation and the decision operation, that is, without being aware of where the cursor bar 24 on the display screen is. A line segment is drawn in the region in the longitudinal direction (see FIG. 1A). The selection is determined by this simple operation. The end when drawing a line segment
It is made by pen-up (see FIG. 1 (B)).

In the embodiment in which the position detection surface and the display screen are separated, the operator is forced to be aware of the distinction between the selection operation and the decision operation. In that case, the operator moves the pen with the pen floating from the position detection surface, or with the pen tip attached but no pressing force applied. In response to this, the cursor bar 24 moves, and while the operator visually recognizes the movement, when the cursor bar 24 reaches a desired position, an operation of drawing a line segment in the longitudinal direction is performed. This makes a selection decision.

If the coordinate detecting device does not detect the coordinates when the pen is floating, the operation becomes more complicated, but the same operation can be performed by using a side switch together. Further, although the present invention is suitable for a stylus pen, the same operation can be performed with a cursor as a position indicator or a mouse instead of the stylus pen.
When performing a gesture input with a mouse, a so-called drag-and-drop operation is performed.

FIG. 2 is a functional block diagram showing an apparatus for realizing the selection determining method according to the present invention. Computer 1
Reference numeral 0 has a general computer configuration, and includes a tablet 21 (coordinate detection device) as an input device and a display unit 22 as an output device. Tablet 21
As described above, the display unit 22 and the display unit 22 are preferably configured such that the position detection surface and the display screen are stacked and arranged so that the input position and the output position match.

The position / status detection unit 11, the display control unit 12, the gesture recognition unit 13, the command processing unit 14,
The application processing unit 15, the print output unit 16 and the like each have a space for convenience of playing a role of the CPU when the CPU of the computer 10 reads a necessary program according to the time and executes the process according to the program. Placed separately and arranged to show their relationship,
That is, it is shown as a functional block diagram.

Roughly speaking of each function, position /
The status detection unit 11 is a unit that receives coordinate information and status information from the tablet 21, and converts the coordinate information and status information into necessary information according to the conditions.
The gesture recognition unit 13 is the position / status detection unit 1.
This is a part that receives the stroke data from 1 and recognizes the stroke data as a gesture. The command processing unit 14 includes the gesture recognition unit 1
3 is a part that issues a command code in accordance with the result recognized by 3 and passes it to the application processing section 15.

The system shown in FIG. 2 does not necessarily require a keyboard, but if the operator wants to use the keyboard together, it can be used by connecting it. At that time, the code input from the command assignment key of the keyboard is passed to the application processing section 15, but the command processing section 14 issues a code similar to that and passes it to the application processing section 15. I can think.

If the pointing device is a mouse instead of a stylus pen, the tablet 2
1 replaces the mouse, and the position / status detector 11, the gesture recognizer 13, and the command processor 14 replace the mouse driver. At that time, the gesture recognition unit 13 becomes a unit for recognizing drag and drop.

The application processing section 15 is a section for executing the application program of the computer, and displays the processing result of the application program on the display section 22 via the display control section 12. In addition, the printer 23 is made to output via the print output unit 16 as required by the operator. In addition,
The print output unit 16 and the printer 23 are not directly related to the selection determination method according to the present invention, but the computer 1
0 is a peripheral device normally provided.

When the application processing unit 15 executes various processes, the operator's input required is to open the display control unit 12 by opening the menu screen, dialog box, zoom, etc. required by the application processing unit 15. This is done by displaying it on the display unit 22 via the prompt, the operator's input, and the operator responding to the input and inputting through the tablet 21. Hereinafter, in order to avoid complication, the above-described optimum embodiment will be described in more detail.

Now, the application processing unit 15 works to display the image on the display unit 22 via the display control unit 12 as shown in FIG.
It is assumed that a screen as shown in, that is, a plurality of selection items are displayed. When the pen tip is approaching the position detection surface regardless of whether the pen is down or up, the tablet 2
1 is the XY position indicated by the stylus pen
The coordinates are sent to the position / status detection unit 11. Then, the position / status detection unit 11 constantly sends the coordinate information to the application processing unit 15 (or the application processing unit 15 constantly monitors the coordinate information). The application processing unit 15 is provided with a table in which coordinates and items are associated with each menu screen, so that the display control unit 12 and the display unit 22 can perform processing for changing the display state such as reverse display of selected items. Running through. As described above, regarding the display state changing process, from the tablet 21, the position / status detecting unit 11, the application processing unit 15, the display control unit 12,
Processing of the flow of the display unit 22 is performed.

In addition to the above processing, there is a processing flow of the tablet 21, the position / status detection unit 11, the gesture recognition unit 13, the command processing unit 14, and the application processing unit 15. While the display state changing process (pointing process) described above is based on only coordinate information, this process is a combination of coordinate information and status information, that is, coordinate information when the status information is on (stroke data). ), It can be positioned as an inking process.

The inking process will be described in more detail. Each part of the selection item displayed on the display unit 22 is called an item display area. Then, the areas of the tablet 21 corresponding to the respective item display areas are respectively positioned as the recognizable areas. Preferably, each of the regions is provided as a region having a long and narrow substantially oblong shape. This is because it is suitable for expressing the contents of items with a character string and arranging many items. It is also suitable for determining whether or not a designated position belongs to the area on the tablet. The size of the gesture recognizable area is preferably a size that covers the item display area, and may be slightly larger.

When the operator pen downs in the area where the gesture can be recognized and has moved by a predetermined distance or more before the next pen up, the position / status detector 11 acquires the stroke data and the gesture recognizer 13 Hand over the processing to. The gesture recognition unit 13 analyzes the stroke data, recognizes that it is a selection determination command if it matches a predetermined recognition target, here, if it is a line segment extending in the longitudinal direction, and executes command processing. Part 1
The process is handed over to 4. The command processing unit 14 issues a code corresponding to the selection determination command and passes it to the application processing unit 15.

Here, the data handled by the gesture recognizing unit 13 is stroke data. However, when the target of the vestier to be recognized by the vestier recognizing unit 13 is only one type of tier that is a selection determinant, It is also possible to perform recognition based on only the first and last data among a plurality of data from pen down to pen up. In the present application, the term “geertier recognition” is considered as a concept including such recognition processing.

However, in the case where it is desired to handle a plurality of gestures, for example, when it is desired to input a screen operation command such as scrolling of a menu screen in addition to the item selection decision, the series of data from pen down to pen down That is, the entire stroke data should be the target of recognition processing.

FIG. 3 is a flowchart showing the inking process in the selection determining method according to the present invention. That is, the flow of a series of processes from the position / status detection unit 11 shown in FIG. 2 to the command recognition unit 14 via the gesture recognition unit 13 is shown. As described above, the display state changing process (pointing process) is performed separately from this series of processes.

When the inking process is started (step 1
00), first look at the status information (step 10
1). In the pen-up state, no inking is required, so the process returns. When the pen is down, it is determined whether or not it is a region in which the gesture can be recognized (step 102). It is determined whether or not the XY coordinates are in any of the rectangular areas indicating them. If it is not in any of the areas, no inking is required, and the process returns.

In addition, when the operator temporarily moves down the pen into the gesture recognition area but stops the pen halfway without moving it, the operator returns through 103, 104, and 106, and moves slightly and moves the pen halfway. When it is up, it goes back through 103, 105 and 106. This is to allow the operator to change the policy.

When the movement of the pen in the pen-down state in the gesture recognizable area is a predetermined distance or more, the gesture recognition processing is started (step 107). As described above, if it is sufficient to recognize only one type of command, which is the selection determination command, step 1
It is also possible to recognize the selection decision command only from the result that the result of 06 is YES. However, if there is another type of command to be recognized, pen down (step 101) to pen up (step 1)
The process of acquiring a series of stroke data up to 05) and analyzing it will be executed. The code issuance in step 108 is a process executed by the command processing unit 14. This series of inking processes is repeated many times.

As described above, the selection determining method of the present invention is the traditional selection determining method that follows the conventional mouse,
In other words, it breaks the convention of clicking and double clicking the cursor bar and proposes a new selection determination method. That is, the present invention proposes a selection determination method in which the cursor bar is used as a gesture recognizable area, and a selection command is input to the area to perform selection determination.

Although not shown in FIG. 2, it is general that a computer using a stylus pen is also provided with a character recognition section. The processing of the position / status detection unit 11 in the present invention is excellent in that it can be shared with the processing for passing stroke data to the character recognition unit.

Further, although the selection determining method according to the present invention goes beyond the pen operation by the mouse emulation and brings about remarkable improvement in operability, on the contrary, this selection determining method has become popular. In this case, even if the driver for the pen becomes popular, it also enables the pen emulation with the mouse. This is because drag-and-drop with the mouse can be used as a substitute for the input of the gesture command.

[0048]

[Effect] Since the selection determining method according to the present invention has the above-mentioned configuration, it has the following effects.

It was possible to provide a selection determination method suitable for a stylus pen as a position indicator similar to a writing instrument. Specifically, the operator is relieved from the vertical movement of the pen, which is not suitable for a stylus pen. In addition, the operator is freed from the noise generated during the click and the double click.

The problem of displacement of the pen when double-clicked was solved.

An organic fusion of inking and pointing is possible.

[Brief description of drawings]

FIG. 1 is a diagram showing an operation in a selection determination method according to the present invention.

FIG. 2 is a functional block diagram showing an apparatus that realizes a selection determination method according to the present invention.

FIG. 3 is a flowchart showing an inking process in the selection determining method according to the present invention.

FIG. 4 is a diagram showing a first conventional example, that is, a method for determining selection by a keyboard.

FIG. 5 is a diagram showing a second conventional example, that is, a method of determining selection by a mouse.

[Explanation of symbols] 10 Computer 11 Position / status detection unit 12 Display control unit 13 Gesture recognition unit 14 Command processing unit 15 Application processing unit 16 Print output unit 20 Pointing device 21 Tablet 22 Display unit 23 Printer 24 Cursor bar 40 Arrow Key 41 run button 42 mouse

─────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] January 12, 1994

[Procedure Amendment 1]

[Document name to be corrected] Drawing

[Correction target item name] All drawings

[Correction method] Change

[Correction content]

[Figure 1]

[Fig. 2]

[Figure 3]

[Figure 4]

[Figure 5]

Claims (5)

[Claims] 1. An operator inputs position information and status information into a computer by operating a pointing device while checking a display screen, and based on the input information, the computer selects an item desired by the operator among a plurality of items. In the selection and determination method of the computer for selecting and deciding, the item display area for displaying the plurality of items is provided on the display screen, and each item display area can be recognized in the movable range of the pointing device corresponding to it. An area is provided, and the display state of the item display area is changed according to the change of the position information from the pointing device, the status information from the pointing device is ON, and the position information is in the gesture recognizable area. At some point, a series of location and status information Chromatography obtained as click information, the on stroke information running Jiesuchiya recognition processing, selection determination method and selects and determines an item corresponding to the Jiesuchiya recognizable area as a result of the Jiesuchiya recognition process. 2. The selection determining method according to claim 1, wherein the condition for acquiring the stroke information is that not only the status information from the pointing device is ON and the position information is the gesture recognizable area. In addition to that, the selection determining method is characterized in that the change of the position information is not less than a predetermined value. 3. The selection determination method according to claim 1, wherein the gesture recognizable area has an elongated rectangular shape, and the object of the gesture recognition processing is a longitudinal direction of the die recognizable area. A method for determining selection, which is a line segment extending to 4. The selection determining method according to claim 1, 2 or 3, wherein the pointing device is a stylus pen. 5. The selection determination method according to claim 4, wherein the position detection surface of the stylus pen and the display screen are arranged in a stack, and the selection item display area and the gesture recognizable area are overlapped with each other. Characteristic selection decision method.