JP2012203563A - Operation input detection device using touch panel - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately detect operation input given as a gesture of a finger.SOLUTION: On the basis of signals from a touch panel 10, a finger contact representative point is recognized by a representative point recognition part 110, and the change pattern is analyzed in a pattern analysis part 120. A sample pattern sampling part 130 instructs a display device 20 to make a prescribed gesture, and stores an analysis pattern obtained at the time in a sample pattern storage part 150 as a sample pattern of the gesture. An operation input pattern sampling part 140 samples an analysis pattern obtained when detection request signals are given from an external unit 200 as an operation input pattern, and a pattern selection part 160 selects a pattern that matches the sampled pattern from the sample pattern storage part 150. A detection result output part 170 returns a detection result signal indicating the gesture corresponding to the selected pattern to the external unit 200.

Description

  The present invention relates to an operation input detection apparatus using a touch panel, and more particularly to a technique for detecting an operation input by a gesture performed using a finger.

  The touch panel is widely used as an input unit for various electronic devices, and is a device that is widely used together with a keyboard and a mouse as an input device for a computer. For example, many notebook personal computers are provided with a touch panel (touch pad) adjacent to a keyboard, and are used as input devices instead of a mouse. In addition, touch panels have been used for tablet PCs, mobile phones, and other electronic devices that do not have a keyboard. In such electronic devices, the display screen constitutes the touch surface of the touch panel, and the touch screen from the user. It is configured to accept operation input.

  In principle, a touch panel is a device that identifies the position touched by a user's finger. By analyzing the temporal transition of the finger touch position, operation input by gestures performed using the finger is detected. It becomes possible to do. In particular, recently, touch panel devices capable of multi-touch input have begun to spread, and techniques for detecting various gestures performed by a plurality of fingers have been put into practical use. If this technique is used, the user can input various instructions simply by performing a gesture using a finger on the touch panel.

  For example, the following Patent Document 1 discloses a configuration of an analog resistive film type multi-touch compatible touch panel device, and Patent Document 2 discloses a configuration of a capacitance type multi-touch compatible touch panel device. Has been. Patent Document 3 discloses a technique for detecting various gesture operations performed by a user using such a multi-touch type touch panel device. Specifically, by combining the movement operation of each finger on the touch surface, operation input can be performed by gestures such as slide and rotation.

JP 2010-26641 A JP 2008-097609 A Special table 2009-525538

  When accepting an operation input from a user as a gesture using a touch panel, it is important to accurately recognize which gesture the user has performed. However, a touch panel is a device that uses a mechanism that electrically detects resistance changes in resistive films, capacitance changes in capacitive elements, etc., so the strength of the output signal depends on the environment such as temperature and humidity in the location where it is used. Has a changing nature. In addition, there are individual differences in finger thickness, pressure on the touch panel, finger contact angle, etc., depending on the user, so it is difficult to accurately recognize the gesture performed by the user with the conventionally proposed technology. is there.

  In particular, when a multi-touch type touch panel device is used, it is possible to mix gestures with one finger, gestures with two fingers, gestures with three fingers, etc., and many gestures can be defined. Can be used. However, as the number of gestures to be defined increases, it becomes more difficult to accurately recognize which gesture the user actually performed. If the gesture performed by the user is erroneously recognized, another operation input that is completely different from the operation input intended by the user is given to the electronic device. It is necessary to redo the gesture after performing the operation of canceling, and it is inevitable that the operability will deteriorate.

  Therefore, an object of the present invention is to provide an operation input detection device using a touch panel that can accurately detect an operation input given as a gesture on a touch surface by a user.

(1) According to a first aspect of the present invention, in the operation input detection device using a touch panel,
A touch panel device that detects a contact state of the user's finger with respect to the touch surface, a display device that displays information to be presented to the user, and a digital processing unit,
Digital processing unit
Based on a detection signal given from the touch panel device, a representative point recognition unit that recognizes a temporal transition of a representative point indicating a representative position of a contact area of a user's finger,
Based on the recognition result of the representative point recognition unit, a pattern analysis unit that analyzes the temporal change state of the contact state and obtains an analysis pattern;
An instruction to perform a plurality of m gestures on the touch surface is displayed on the screen of the display device, and each analysis pattern obtained by the pattern analysis unit for the m gesture operations of the user based on the instructions is sampled. A sample pattern collection unit to collect as a pattern;
A sample pattern storage unit that stores an analysis pattern collected when an instruction to perform the k-th (1 ≦ k ≦ m) gesture is displayed as a sample pattern corresponding to the k-th gesture;
An operation input pattern collection unit that collects an analysis pattern obtained by the pattern analysis unit as an operation input pattern after a detection request signal for requesting detection of an operation input is given from the outside,
A pattern selection unit that compares the collected operation input patterns with a plurality of m sample patterns stored in the sample pattern storage unit, and selects a sample pattern that matches or approximates;
A detection result output unit that outputs information indicating a gesture corresponding to the sample pattern selected by the pattern selection unit to the outside as a detection result signal indicating a detection result of the operation input;
It is made up by.

(2) According to a second aspect of the present invention, in the operation input detection device using the touch panel according to the first aspect described above,
The digital processing unit further includes a user specifying unit that inputs information for specifying a user currently in use and stores the information.
The sample pattern collecting unit collects the sample pattern independently for each user specified by the user specifying unit,
The sample pattern storage unit stores the sample pattern independently for each user specified by the user specifying unit,
The pattern selection unit performs pattern selection by comparing with a sample pattern for the user specified by the user specifying unit.

(3) According to a third aspect of the present invention, in the operation input detection device using the touch panel according to the first or second aspect described above,
The sample pattern collecting unit displays an instruction to perform a gesture separately for each of a plurality of areas obtained by dividing the touch surface of the touch panel device, and collects a sample pattern for each area,
The sample pattern storage unit stores individual and independent sample patterns for each area,
The pattern selection unit recognizes the main area where the gesture is performed, and performs pattern selection by comparing with the sample pattern for the recognition area.

(4) According to a fourth aspect of the present invention, in the operation input detection device using the touch panel according to the third aspect described above,
The touch panel device has a rectangular touch surface,
The sample pattern collection unit collects independent and independent sample patterns for the landscape mode that uses the touch surface in the landscape state and the portrait mode that uses the touch surface in the portrait state,
The sample pattern storage unit stores the sample pattern separately for each of the landscape mode and portrait mode,
The pattern selection unit has a function of recognizing the current mode, and performs pattern selection by comparing with a sample pattern for the recognition mode.

(5) According to a fifth aspect of the present invention, in the operation input detection device using the touch panel according to the fourth aspect described above,
The touch panel device has a sensor for detecting gravitational acceleration,
The pattern selection unit has a function of recognizing the current mode based on the output signal of the sensor.

(6) According to a sixth aspect of the present invention, in the operation input detection device using the touch panel according to the first to fifth aspects described above,
The touch panel device outputs a signal indicating the degree of finger contact for each position on the touch surface,
The representative point recognition unit recognizes a closed region on the touch surface where the contact degree is a value greater than or equal to a predetermined reference value as a contact region, and “a position where the contact degree reaches a peak in the contact region” or “the touch surface is horizontal. In the coordinate system in which the degree of contact is taken in the height direction, the position of the center of gravity of the mountain formed on the contact area is recognized as a representative point.

(7) According to a seventh aspect of the present invention, in the operation input detection device using the touch panel according to the first to sixth aspects described above,
The representative point recognition unit recognizes the coordinate value (x, y) and the current time t of the representative point in the two-dimensional XY coordinate system defined on the touch surface for each contact area, and the value (x, y, t). The representative point data including is output at a predetermined time interval,
The pattern analysis unit obtains an analysis pattern based on the representative point data.

(8) According to an eighth aspect of the present invention, in the operation input detection device using the touch panel according to the seventh aspect described above,
The touch panel device outputs a signal indicating the degree of finger contact for each position on the touch surface,
For each contact area, a representative point recognition unit is further formed on the contact area in a coordinate system in which the “contact level peak value in the contact area” or “the touch surface is a horizontal plane and the contact level is in the height direction”. The volume of the mountain ”is recognized as the intensity value a of the contact area, and representative point data Q (x, y, a, t) including the value (x, y, a, t) is output at a predetermined time interval.
The pattern analysis unit obtains an analysis pattern based on the representative point data Q (x, y, a, t).

(9) According to a ninth aspect of the present invention, in the operation input detection device using the touch panel according to the eighth aspect described above,
The pattern analysis unit analyzes the pattern based only on the representative point data Q (x, y, a, t) whose value a is equal to or greater than a predetermined reference value.

(10) According to a tenth aspect of the present invention, in the operation input detection device using the touch panel according to the seventh to ninth aspects described above,
The pattern analyzer
A representative point data storage unit for storing representative point data;
Based on the representative point data, a stroke recognizing unit for recognizing a stroke composed of an aggregate of a plurality of representative points indicating the same finger movement;
A characteristic recognition unit for recognizing the characteristics of individual strokes recognized by the stroke recognition unit;
A pattern determination unit for determining an analysis pattern by associating one stroke having a unique characteristic or a combination of a plurality of strokes with one pattern based on the recognition result by the stroke recognition unit and the characteristic recognition unit;
It is made to have.

(11) According to an eleventh aspect of the present invention, in the operation input detection device using the touch panel according to the tenth aspect described above,
Stroke identification unit, based on the set of representative point data are temporally continuous output from the representative point recognition unit, a representative point obtained to the i-th time t _i (i-1) th time The stroke is recognized by recognizing two representative points whose distance from the representative point obtained at t _i-1 is equal to or less than a predetermined reference as representative points belonging to the same stroke. .

(12) According to a twelfth aspect of the present invention, in the operation input detection device using the touch panel according to the tenth or eleventh aspect described above,
The characteristic recognition unit indicates at least the stroke length L indicating the sum of the lengths of the connecting lines when a plurality of representative points constituting one stroke are connected in order along the time series, and the degree of curvature of the stroke. The stroke curvature C is calculated as a value indicating the characteristics of the stroke.

(13) According to a thirteenth aspect of the present invention, in the operation input detection device using the touch panel according to the twelfth aspect described above,
The characteristic recognizing unit sets the coordinate value of the i-th representative point Q _i to (x _i , y _i ) and (i−1) -th representative point for a stroke composed of a total of n representative points. The coordinate value of Q _i-1 is (x _i-1 , y _i-1 ), the vector from the representative point Q _{i -1} to the representative point Q _i is V _i , and the (i-2) -th representative point Q _{When a} vector from _i-2 to the representative point Q _i-1 is V _i-1 and an angle formed by the vector V _i with respect to the vector V _i-1 is a displacement angle φ _i ,
L = _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
C = Σ _{i = 3 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 · φ _i
/ Σ _{i = 3 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
In this way, the stroke length L and the stroke curvature C are calculated.

(14) According to a fourteenth aspect of the present invention, in the operation input detection device using the touch panel according to the twelfth or thirteenth aspect described above,
The pattern determining unit classifies each stroke into a plurality of types based on the stroke length L and the stroke curvature C, and determines an analysis pattern by associating a specific analysis pattern with a specific type or a combination thereof. It is what I did.

(15) According to a fifteenth aspect of the present invention, in the operation input detection device using the touch panel according to the fourteenth aspect described above,
The pattern determination unit classifies strokes having a stroke length L less than a predetermined reference value Lth into a type of “point stroke”, sets strokes having a stroke length L equal to or greater than the reference value Lth as “line strokes”, Of these strokes, strokes having an absolute value of curvature C less than a predetermined reference value Cth are classified into types of “linear strokes”. Among “line strokes”, strokes having an absolute value of curvature C of at least the reference value Cth are classified as “ The analysis pattern is determined by classifying it into a type of “curve stroke” and associating a specific analysis pattern with the type of “point stroke”, “linear stroke”, “curve stroke”, or a combination thereof. It is a thing.

(16) According to a sixteenth aspect of the present invention, in the operation input detection device using the touch panel according to the fifteenth aspect described above,
The stroke curvature C is determined so that the sign differs between when the stroke rotates clockwise and when it rotates counterclockwise,
The pattern determining unit handles the type of “curve stroke” based on the sign of stroke curvature C, and further classifies it into two types of “clockwise curved stroke” and “counterclockwise curved stroke” The decision is made.

(17) According to a seventeenth aspect of the present invention, in the operation input detection device using the touch panel according to the fifteenth or sixteenth aspect described above,
The characteristic recognizing unit further calculates a stroke direction B indicating the direction of the entire stroke on the XY plane as a value indicating the characteristic of the stroke.

(18) According to an eighteenth aspect of the present invention, in the operation input detection device using the touch panel according to the seventeenth aspect described above,
The characteristic recognizing unit has a predetermined vector V _i from the (i−1) -th representative point Q _i−1 to the i-th representative point Q _i for a stroke composed of a total of n representative points. When the angle formed with respect to the reference coordinate axis is azimuth angle θ _i ,
B = _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 · θ _i
/ _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
The stroke azimuth B is calculated by the following calculation.

(19) According to a nineteenth aspect of the present invention, in the operation input detection device using the touch panel according to the seventeenth or eighteenth aspect described above,
Based on the stroke direction B, the pattern determination unit handles the type of “straight stroke” by further dividing it into a plurality of W types having different directions, and determines the pattern.

(20) According to a twentieth aspect of the present invention, in the operation input detection device using the touch panel according to the fifteenth to nineteenth aspects described above,
The characteristic recognizing unit further calculates a stroke position P indicating the position of the stroke on the XY plane as a value indicating the characteristic of the stroke.

(21) According to a twenty-first aspect of the present invention, in the operation input detection device using the touch panel according to the twentieth aspect described above,
The characteristic recognizing unit sets the coordinate value of the i-th representative point Q _i to (x _i , y _i ) and (i−1) -th representative point for a stroke composed of a total of n representative points. When the coordinate value of Q _i-1 is (x _i-1 , y _i-1 ),
xp = _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 · x _i
/ _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
yp = Σ _{i = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 · y _i
/ _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
Thus, the coordinate value of the stroke position P (xp, yp) is calculated.

(22) According to a twenty-second aspect of the present invention, in the operation input detection device using the touch panel according to the twentieth or twenty-first aspect described above,
For a pattern including a plurality of strokes, the pattern determining unit grasps the mutual positional relationship by referring to the stroke position P, and determines the pattern by distinguishing patterns having different mutual positional relationships. is there.

(23) According to a twenty-third aspect of the present invention, in the operation input detection device using the touch panel according to the tenth to twenty-second aspects described above,
The pattern selection unit selects an approximate sample pattern that includes a stroke with characteristics that approximate the characteristics of the stroke included in the operation input pattern when there is no sample pattern that matches the collected operation input pattern. Is.

(24) According to a twenty-fourth aspect of the present invention, in the operation input detection device using the touch panel according to the twenty-third aspect described above,
The digital processing unit further includes a history storage unit that stores a history of detection results output in the past by the detection result output unit,
When the pattern selection unit selects an approximate sample pattern, the history is referred to, and a sample pattern having a higher past selection frequency is determined to be a more approximate sample pattern.

(25) According to a twenty-fifth aspect of the present invention, in the operation input detection device using the touch panel according to the twelfth to twenty-second aspects described above,
When the pattern selection unit has no sample pattern that matches or approximates the collected operation input pattern, a corrected operation input in which the stroke with the shortest stroke length L is deleted from the plurality of strokes included in the operation input pattern A pattern is created, and a sample pattern that matches the correction operation input pattern is selected.

(26) According to a twenty-sixth aspect of the present invention, in the operation input detection device using the touch panel according to the first to the twenty-fifth aspects described above,
The touch panel device and the display device are configured by a display device with a touch panel in which a display screen forms a touch surface.

(27) A twenty-seventh aspect of the present invention is an operation input detection device using the touch panel according to the first to twenty-sixth aspects described above,
An external unit having a function of receiving a detection request signal from the operation input detection device and receiving a detection result signal from the operation input detection device;
Thus, an electronic device is configured.

  (28) According to a twenty-eighth aspect of the present invention, the digital processing unit in the operation input detection device using the touch panel according to the first to twenty-fifth aspects is configured by incorporating a program into a computer.

(29) According to a twenty-ninth aspect of the present invention, there is provided a touch panel of a user by a computer to which a touch panel device that detects a contact state of the user's finger with respect to the touch surface and a display device that displays information to be presented to the user are connected. An operation input detection method using a touch panel that detects an operation input to the device,
It consists of a preparation stage for registering a user's sample operation in advance, and a detection stage for receiving and detecting the user's actual operation input,
In the preparation stage,
The computer displays an instruction to perform a predetermined gesture on the touch surface on the screen of the display device, and the user's finger contact based on the detection signal obtained from the touch panel device by the user's gesture operation based on the instruction Recognizing the temporal transition of the representative point indicating the representative position of the region;
The computer analyzes the temporal change mode of the contact state based on the temporal transition of the representative point, and collects the result as a sample pattern corresponding to the gesture;
Is executed for a plurality of m types of gestures so that a sample pattern corresponding to each gesture is collected,
In the detection stage,
When the computer needs to detect the actual operation input, based on the detection signal given from the touch panel device, recognizing the temporal transition of the representative point indicating the representative position of the contact area of the user's finger;
The computer analyzes the temporal change mode of the contact state based on the temporal transition of the representative point, and collects the result as an operation input pattern corresponding to the actual operation input;
The computer compares the collected operation input pattern with a plurality of m sample patterns collected in the preparation stage, and selects a sample pattern that matches or approximates.
A computer detecting that an operation input corresponding to the selected sample pattern has been made;
Is to be performed.

(30) According to a thirtieth aspect of the present invention, in the operation input detection method using the touch panel according to the twenty-ninth aspect described above,
In the preparation stage, the computer performs a process of collecting independent sample patterns for a plurality of users,
At the detection stage, the computer identifies a user currently in use, and performs pattern selection by comparing with a sample pattern for the user.

(31) According to a thirty-first aspect of the present invention, in the operation input detection method using the touch panel according to the twenty-ninth or thirtieth aspect described above,
In the preparatory stage, the computer displays an instruction to perform a gesture separately for each of a plurality of areas obtained by dividing the touch surface of the touch panel device, and collects a sample pattern for each area. Done
At the detection stage, the computer recognizes the main area where the gesture has been performed, and performs pattern selection by comparing with the sample pattern for the recognition area.

  In the operation input detection device using the touch panel according to the present invention, the temporal transition of the representative point indicating the representative position of the contact area of the user's finger is recognized based on the detection signal given from the touch panel device, and the change pattern is Analyzed. In addition, the user can register various gesture analysis patterns as sample patterns in the preparation stage. In the actual detection stage, the pattern of the gesture performed by the user is collected as an operation input pattern, and a pattern that matches or approximates the collected pattern is selected from the sample patterns and corresponds to the selected sample pattern. Information indicating a gesture is output as an operation input detection result. As described above, since the pattern selection that matches or approximates the operation input pattern is performed using the sample pattern registered in advance in the preparation stage, it becomes possible to accurately detect the operation input of the user.

  In particular, even when an operation input by various gestures using a plurality of fingers is detected by using a multi-touch type touch panel device, the features of gesture operations unique to the user are collected in advance and registered as sample patterns. Therefore, it is possible to accurately recognize the gesture operation performed by the user and to detect an accurate operation input.

  Further, in the case of an operation input detection device scheduled to be used by a plurality of users, a user specifying unit for inputting information for specifying a user currently in use and storing the information is provided, and each user is independently provided. If the sample pattern is collected in this way, it becomes possible to detect an accurate operation input in consideration of the characteristics unique to each user.

  Furthermore, in the case of a touch panel device with a relatively large area, even if the user intends to perform exactly the same gesture operation, the degree of contact with each finger differs depending on the position of the touch surface, and the actual analysis pattern obtained varies. There are many. In order to deal with such a case, the touch surface of the touch panel device may be divided to define a plurality of areas, and individual sample patterns may be collected for each area. By doing so, it is possible to accurately detect an operation input by recognizing the main area where the gesture has been performed and comparing it with the sample pattern for the recognition area. When the touch panel device has a rectangular touch surface, it is possible to detect operation input more accurately by distinguishing whether the user is using the device in a portrait orientation or a landscape orientation. It becomes possible.

  In the analysis of the change pattern based on the temporal transition of the representative points, it is effective to recognize a stroke made up of a collection of a plurality of representative points showing the same finger movement. When a plurality of strokes are recognized, it can be recognized that the user is performing a gesture motion using a plurality of fingers. Further, if the stroke length L and the stroke curvature C are obtained for each stroke, each stroke can be divided into a point stroke, a straight stroke, and a curved stroke, and these different strokes or combinations thereof Since a pattern can be defined by, a plurality of patterns can be defined in a typical manner. Furthermore, if the stroke orientation B and the stroke position P are obtained, a pattern definition with more detailed type variations can be made.

  When there is no perfectly matched sample pattern when selecting a pattern, even if a slightly different gesture operation is performed by selecting an approximate sample pattern, the operation input is effective. Operation that can be detected as In this case, if the degree of approximation is determined with reference to the past history, a more appropriate determination can be made. Further, when the pattern does not match, by deleting the stroke having a short stroke length L, it becomes possible to handle the stroke ignoring the stroke caused by touching the finger by mistake.

It is a block diagram which shows the structure of the operation input detection apparatus using the touchscreen which concerns on fundamental embodiment of this invention. It is a top view which shows an example of the contact state of the finger | toe with respect to the touch surface of the touch panel apparatus 10 shown in FIG. 3 is a graph showing a detection signal obtained from the touch panel device 10 in the contact state shown in FIG. 2. It is a block diagram which shows the detailed structure of the pattern analysis part 120 of the operation input detection apparatus shown in FIG. It is a top view which shows the principle of the stroke recognition process by the stroke recognition part 122 shown in FIG. It is a top view which shows the stroke characteristic recognized by the characteristic recognition part 123 shown in FIG. It is a graph which shows the principle which calculates the stroke characteristic shown in FIG. It is a figure which shows the specific calculating formula which calculates the stroke characteristic shown in FIG. It is a flowchart which shows the procedure of the pattern determination process by the pattern determination part 124 shown in FIG. It is a top view which shows some examples of the analysis pattern determined by the pattern determination part 124 shown in FIG. It is a top view which illustrates the kind of gesture using two fingers. It is a top view which shows an example of the instruction | indication screen displayed on the display apparatus 20 of the operation input detection apparatus shown in FIG. It is a top view which shows an example of the sample pattern stored in the sample pattern storage part 150 of the operation input detection apparatus shown in FIG. It is a top view which shows an example of the sample pattern of another user different from the sample pattern shown in FIG. It is a block diagram which shows the structure of the operation input detection apparatus using the touchscreen which concerns on the modification of this invention. It is a top view which shows an example of the sample pattern for every user stored in the sample pattern storage part 150 of the operation input detection apparatus shown in FIG. It is a top view which shows the example which defined the some area | region on the touch surface and sampled the sample pattern for every area | region. It is a block diagram which shows the structure of the operation input detection apparatus using the touchscreen which concerns on another modification of this invention. It is a top view which shows an example of the instruction | indication screen displayed on the display apparatus 30 with a touch panel of the operation input detection apparatus shown in FIG. It is a top view which shows an example of the sample pattern for every area | region stored in the sample pattern storage part 150 of the operation input detection apparatus shown in FIG. It is a top view which shows the case where the display apparatus 30 with a touch panel of the operation input detection apparatus shown in FIG. 18 is used in a horizontally long state and in a vertically long state. It is a top view which shows the example which defined 36 types of linear strokes based on the stroke azimuth | direction B. FIG. It is a top view which shows the example which defined eight types of curve strokes based on the stroke azimuth | direction B. FIG. It is a top view which shows the example which defined 4 types of curve strokes based on the stroke curvature degree C. FIG. It is a top view which shows the variation of the gesture using two fingers. It is a block diagram which shows the structure of the operation input detection apparatus using the touchscreen which concerns on another modification of this invention. It is a table | surface which shows an example of the specific representative point data obtained when performing the gesture which slides two fingers down from the top. It is a top view which shows the pattern obtained by analyzing the representative point data shown in FIG. It is a table | surface which shows an example of the specific representative point data obtained when performing the gesture which slides two fingers from the left to the right. FIG. 30 is a plan view showing a pattern obtained by analyzing the representative point data shown in FIG. 29. It is a table | surface which shows an example of the specific representative point data obtained when performing clockwise gesture with two fingers. FIG. 32 is a plan view showing a pattern obtained by analyzing the representative point data shown in FIG. 31. It is a table | surface which shows an example of the specific representative point data obtained when performing the counterclockwise gesture with two fingers. It is a top view which shows the pattern obtained by analyzing the representative point data shown in FIG. It is a flowchart which shows the procedure of the operation input detection method using the touchscreen which concerns on this invention.

  Hereinafter, the present invention will be described based on the illustrated embodiments.

<<< §1. Basic Configuration of Operation Input Detection Device According to the Present Invention >>
FIG. 1 is a block diagram showing a configuration of an operation input detection device using a touch panel according to a basic embodiment of the present invention. As shown in the figure, this operation input detection device is composed of a touch panel device 10, a display device 20, and a digital processing unit 100. Based on a request from the external unit 200, an operation input performed by a user as a gesture operation with a finger. Is detected, and the detection result is returned to the external unit 200.

  Here, the touch panel device 10 is a device that detects a contact state of a user's finger with respect to the touch surface and outputs a detection signal. As such a touch panel device 10, a resistance film type device and a capacitance type device are mainly commercially available. The display device 20 is a general device that displays information to be presented to the user. In the present invention, the display device 20 is used to present an instruction to perform a predetermined gesture to the user in the preparation stage. .

  On the other hand, the digital processing unit 100 is composed of blocks in a region surrounded by a one-dot chain line in the figure. Specifically, the digital processing unit 100 includes a representative point recognition unit 110, a pattern analysis unit 120, a sample pattern collection unit 130, an operation input pattern collection unit 140, a sample pattern storage unit 150, a pattern selection unit 160, as illustrated. The detection result output unit 170 is configured. In the embodiment shown here, the digital processing unit 100 is realized by incorporating a dedicated program into a computer. Further, when a notebook computer is used as the computer, the touch pad provided in the notebook computer can be used as it is as the touch panel device 10, and the liquid crystal display or the like provided in the notebook computer can be used as it is. It can be used as the device 20.

  The external unit 200 is an apparatus that accepts an operation input from a user and executes various processes using the operation input detection device. In the embodiment shown here, the external unit 200 is also dedicated to a computer. It is realized by incorporating the program. Specifically, the external unit 200 can be configured by incorporating an OS program and an application program into a computer. When the OS program or application program requires an operation input from a user, the external unit 200 performs digital processing. A detection request signal is issued to the unit 100, and the detection result is returned from the digital processing unit 100 to the external unit 200.

  In short, the external unit 200 is a device having a function of giving a detection request signal to the operation input detection device and receiving an output of the detection result signal from the operation input detection device. It recognizes what operation input has been performed, and executes processing corresponding to the operation input. Eventually, an electronic device configured by adding an operation input detection device (touch panel device 10, display device 20, digital processing unit 100) to external unit 200 accepts a gesture operation by a user's finger as an operation input, It is a device that performs some kind of processing.

  In the illustrated embodiment, the touch panel device 10 and the display device 20 are configured to be shared by the digital processing unit 100 and the external unit 200. Of course, the external unit 200 has a separate user interface. When it is provided, it is not necessary for the external unit 200 to use the touch panel device 10 and the display device 20. However, for practical use, in order to efficiently use hardware, it is preferable to adopt a shared form as in the illustrated example. In particular, when a notebook computer is used, the touch panel device 10 and the display device 20 can be configured by a touch pad and a display incorporated in the notebook computer, and the digital processing unit 100 and the external unit 200 are connected to the notebook computer. It can be configured by a program installed in a personal computer.

  Next, functions of each component of the digital processing unit 100 will be described. First, the representative point recognizing unit 110 is a component having a function of recognizing a temporal transition of a representative point indicating a representative position of a contact area of a user's finger based on a detection signal given from the touch panel device 10. As will be described later, the touch panel device 10 is a device that outputs, as an electrical signal, finger contact states at a number of lattice point positions arranged vertically and horizontally, and a user's finger contact region is a set of a plurality of lattice points. Be recognized. The representative point recognizing unit 110 functions to recognize the representative position (for example, the center position) of the contact area as a representative point and to recognize its temporal transition.

  On the other hand, the pattern analysis unit 120 functions to obtain an analysis pattern by analyzing the temporal change state of the contact state based on the recognition result of the representative point recognition unit 110. That is, the representative point recognized by the representative point recognizing unit 110 is analyzed for how it moves in time, and a process for patterning the movement form is performed. For example, when the user performs a “gesture to slide one finger from top to bottom” on the touch surface, the operation is recognized as “a pattern in which one representative point moves in the vertical direction”. . Here, the pattern obtained by the analysis by the pattern analysis unit 120 is referred to as an “analysis pattern”. The analysis pattern thus obtained is used in the sample pattern collection unit 130 or the operation input pattern collection unit 140.

  The sample pattern collection unit 130 displays an instruction to perform a plurality of m gestures on the touch surface on the screen of the display device 20, and the pattern analysis unit 120 performs m gesture operations of the user based on the instructions. Each analysis pattern obtained functions as a sample pattern.

  In addition, the sample pattern storage unit 150 displays the analysis pattern collected when the sample pattern collection unit 130 displays an instruction to perform the k-th (1 ≦ k ≦ m) gesture as the k-th gesture. The function of storing as a sample pattern corresponding to is performed. For example, the analysis pattern obtained when instructing to perform a “gesture to slide one finger from top to bottom” is stored as a sample pattern corresponding to the gesture, and “two fingers from the left The analysis pattern obtained when instructing to perform the “slide to the right” gesture is stored as a sample pattern corresponding to the gesture.

  As described above, in order to use the operation input detection device according to the present invention, as a preparation stage, a sample pattern is collected for each of a plurality of m gesture operations and stored in the sample pattern storage unit 150 in advance. Need to do. When such a preparation stage is completed, the role of the sample pattern collection unit 130 is completed (of course, a new sample pattern can be collected for a new gesture operation as needed). After the preparation stage is completed, the actual detection stage can be performed using the function of each component described below.

  First, the operation input pattern collection unit 140 performs a function of collecting an analysis pattern obtained by the pattern analysis unit 120 as an operation input pattern after a detection request signal for requesting detection of an operation input is given from the external unit 200. . On the other hand, the pattern selection unit 160 compares the operation input pattern collected by the operation input pattern collection unit 140 with a plurality of m types of sample patterns stored in the sample pattern storage unit 150, and selects a matching sample pattern. Fulfills the function. The selection result is transmitted to the detection result output unit 170. The detection result output unit 170 functions to output information indicating a gesture corresponding to the sample pattern selected by the pattern selection unit 160 to the external unit 200 as a detection result signal indicating the detection result of the operation input.

  When the above processing is viewed from the external unit 200 side, when some operation input by the user is required, if a detection request signal is given to the operation input pattern collection unit 140, the detection result is displayed after the user finishes the gesture operation. A detection result signal is output from the output unit 170. Here, the detection result signal is a signal indicating which gesture operation performed by the user corresponds to a plurality of m gesture operations registered in advance (or a matching gesture operation is registered). Error signal).

  As described above, in the present invention, a plurality of m sample patterns registered in advance in the sample pattern storage unit 150 are used to select a pattern that matches the operation input pattern collected by the operation input pattern collection unit 140. Therefore, even if the user's gesture motion is far from the standard gesture motion, correct operation input can be detected based on the user's gesture motion.

  In general, a simple gesture operation in which one finger is slid vertically or horizontally will not make much difference between users. However, in the case of electronic devices that use a plurality of fingers, such as two fingers or three fingers, and perform various operation inputs by complex gestures that combine slides, rotations, taps, etc., instructions were given to perform the same gestures However, the gesture operation actually performed on the touch surface may vary among users, and different analysis patterns may be obtained by the pattern analysis unit 120.

  In the present invention, in the preparation stage, a plurality of m gesture operations are executed by the user himself, and the analysis patterns obtained at that time are respectively registered as sample patterns for the respective gestures. In the detection stage, the obtained operation input pattern is compared with a plurality of m sample patterns, and a specific gesture is selected. Therefore, even if there is a characteristic unique to the user in the gesture operation, it becomes possible to accurately detect the user's operation input using the characteristic.

<<< §2. Specific processing by each component >>>
Here, specific processing contents executed by individual elements constituting the digital processing unit 100 shown in FIG. 1 will be described. Of course, the specific processing described here shows one embodiment of the present invention, and the present invention is not limited to this embodiment.

<2-1: Specific Processing of Representative Point Recognition Unit>
As already described, the representative point recognition unit 110 is a component that recognizes the temporal transition of the representative point indicating the representative position of the contact area of the user's finger based on the detection signal given from the touch panel device 10. FIG. 2 is a plan view showing an example of a contact state of a finger with respect to the touch surface of the touch panel device 10. As shown in the figure, grid lines are defined vertically and horizontally on the touch surface of a general touch panel device 10, and finger contact states at a number of grid point positions arranged at intersections of the grid lines are detected as electrical signals. Is done. In the case of the resistive touch panel device, the change in the electrical resistance value at the lattice point position is taken out as an electrical signal, and in the case of the capacitive touch panel device, the change in the capacitance value at the lattice point location is taken out as an electrical signal. It is.

  Here, for convenience of explanation, a two-dimensional XY coordinate system as shown in the figure is defined on the touch surface, and the position of an arbitrary point on the touch surface is indicated by coordinate values (x, y). For example, the position of the lattice point H is represented by a coordinate value H (x, y). In the illustrated example, the vertical grid lines are arranged at a pitch dx in the X-axis direction, and the horizontal grid lines are arranged at a pitch dy in the Y-axis direction. Eventually, an electrical signal indicating the contact state of the finger is obtained as a discrete value for each of a large number of lattice points arranged with a pitch dx in the horizontal direction and a pitch dy in the vertical direction. In applications for detecting the contact state of a finger, it is sufficient that the pitches dx and dy are about 5 mm or less.

Areas A1 and A2 shown by hatching in FIG. 2 indicate contact areas where the finger contacts the touch surface. In this example, a state in which the user touches the touch surface with the tips of two fingers is shown. In this case, a significant signal indicating the contact state is obtained for the lattice points located in the contact areas A1 and A2. FIG. 3 is a graph showing detected values on the horizontal grid line indicated by the Y coordinate value y _k in the contact state shown in FIG. 2, and bars drawn with the pitch dx are obtained at individual grid points. The detected value is shown. Actually, since the lattice points are distributed on a two-dimensional plane, the detection signal given from the touch panel device 10 is expressed by a three-dimensional graph obtained by extending the graph shown in FIG. 3 in the depth direction of the drawing. . As described above, the actual detection values can be obtained only for the discrete lattice points arranged with the pitch dx in the horizontal direction of the figure and the pitch dy in the depth direction of the figure, but these discrete values are interpolated. Thus, the envelope surface E as shown can be obtained.

  As shown in FIG. 3, the portions corresponding to the contact areas A1 and A2 show a climax like a mountain. This is because the contact state with the finger is detected as a change amount of the resistance value or capacitance value at the position of each lattice point, that is, an analog amount. In the example shown here, this analog amount is converted into an 8-bit digital amount (a numerical value of 0 to 255) and is taken into the representative point recognition unit 110. That is, a signal indicating a contact state with a finger is not captured as a binary signal indicating contact / non-contact. The touch panel device 10 outputs a signal indicating a finger contact degree (degree of contact / finger pressing force) for each position on the touch surface, and the representative point recognition unit 110 indicates a stepwise contact degree. The detection value is captured. Of course, if the pressing force on the touch surface is large, a detection value that is larger is captured.

  Note that the resistance value of the resistive film and the capacitance value of the capacitive element also vary depending on the environment such as the temperature and humidity of the location where it is used. The detected value is output. In the graph shown in FIG. 3, the detected value of the part other than the contact areas A1 and A2 (mountain base) is not 0 because this noise component is detected. Therefore, the representative point recognition unit 110 performs a process of determining a predetermined reference value Th for the detection value indicating the contact degree, and recognizing the closed region on the touch surface where the detection value is equal to or greater than the reference value Th as the contact region. Do. In the case of the example shown in FIG. 3, the two peak portions whose detected values are equal to or greater than the reference value Th are recognized as the contact areas, and as a result, the contact areas A1 and A2 shown by hatching in FIG. Done.

  However, since the gesture action performed by the user is grasped as such a temporal transition pattern of the contact area, in performing the pattern analysis of the gesture action, the position of the area rather than the area information with the area is used. It is more convenient to handle the information converted to point information. Therefore, the representative point recognition unit 110 performs processing for obtaining the representative points Q1 and Q2 for the respective contact areas A1 and A2. These representative points can be obtained as position data in a format such as Q1 (x1, y1), Q2 (x2, y2) using coordinate values in the XY coordinate system.

Each representative point Q1, Q2 may be determined by any method as long as it is a point indicating the representative position of each contact area A1, A2. In the example shown in FIG. 3, the positions of the representative points Q1 and Q2 for each of the contact areas A1 and A2 are obtained as “a position where the contact degree reaches a peak in the contact area” (in short, the position of the peak of the mountain). . In FIG. 3, for convenience of explanation, the points Q1 and Q2 are plotted at the vertices of the mountain, but the actual representative points Q1 and Q2 are points defined on the touch surface (that is, the XY plane). Therefore, the points obtained by projecting the vertices of these peaks onto the XY plane are the representative points Q1 and Q2. Further, as shown in FIG. 2, the point Q1, Q2, so not in terms of the lateral direction of the grating lines for Y-coordinate value y _k, Q1 points shown in Figure 3, in fact, slightly above the plane of the graph The point Q2 shown in FIG. 3 which is the front point is actually a point slightly behind the paper surface of the graph.

  Another method for determining the representative point is the position of the center of gravity of the mountain that consists of the part of the reference value Th (or more accurately, in the coordinate system in which the touch surface is the horizontal plane and the contact degree is in the height direction) In this method, the position of the center of gravity of the mountain formed on the region is set as the representative point position. This method requires computation to determine the position of the center of gravity of the mountain. However, since the representative point can be determined in consideration of the volume distribution of the entire mountain, a more appropriate representative point can be obtained.

  When the user moves the finger on the touch surface and performs a gesture operation, the detection signal given from the touch panel device 10 changes from moment to moment, so of course, the position of the representative point Q also changes with time. . Therefore, the representative point recognizing unit 110 recognizes the coordinate value (x, y) of the representative point Q in the two-dimensional XY coordinate system defined on the touch surface for each contact area by the method described above, and at the current time t. And representative point data including values (x, y, t) is output at predetermined time intervals.

  For example, if the sampling period is set to 50 ms, the representative point recognizing unit 110 has the position coordinates (x, x, x) of all the representative points Q recognized based on the detection signal given from the touch panel device 10 every 50 ms. A process of outputting y) together with information indicating the time t at that time is performed. When the touch panel device 10 is an event-driven device, a detection signal is output only when a predetermined change or more occurs in the finger contact state. In such a case, the representative point recognizing unit 110 cannot obtain a detection signal at a fixed period, but by performing interpolation processing on the time axis, for example, the position coordinates (x, y for each sampling period of 50 ms). ). Of course, the sampling period does not necessarily have to be a constant cycle such as 50 ms. Therefore, every time a detection signal is given from the event-driven device, the position coordinate (x, y) of the representative point Q at that time and its position The time t at the time may be output.

  In the case of the embodiment described here, as shown in the graph of FIG. 3, the detection signal given from the touch panel device 10 is configured by an 8-bit detection value indicating the degree of contact. As described above, when the touch panel device 10 has a function of outputting a signal indicating the degree of finger contact for each position on the touch surface, not only the position coordinates (x, y) of the representative point Q but also the representative point. An intensity value a indicating the pressing force of the finger at the position can be obtained.

  This intensity value a can be defined as, for example, “a peak value of the degree of contact in the contact area”. In the case of the example shown in FIG. 3, the intensity value a1 for the contact area A1 is defined as the height (contact degree) of the top of the mountain shown as point Q1 in the figure, and the intensity value a2 for the contact area A2 is It is defined as the height (contact degree) of the peak of the mountain shown as point Q2. Alternatively, the intensity value a is defined as the volume of a mountain (in terms of accuracy, the volume of a mountain formed on the contact area in a coordinate system in which the touch surface is a horizontal plane and the degree of contact is in the height direction). You can also

  In the case of this embodiment, the representative point recognition unit 110 recognizes the intensity value a for each contact area, and representative point data Q (x, y, a, t) including the value (x, y, a, t). ) Is output at predetermined time intervals. Eventually, the representative point data Q (x, y, a, t) has a pressing force indicated by the intensity value a near the representative point Q indicated by the coordinate value (x, y) at time t. This is information indicating the fact that they have contacted. When a plurality of fingers touch the touch surface at the same time, a plurality of representative point data Q (x, y, a, t) having the same time t is output simultaneously.

<2-2: Specific Processing of Pattern Analysis Unit>
Subsequently, specific processing performed in the pattern analysis unit 120 will be described. As described above, the representative point data Q (x, y, a, t) is given from the representative point recognition unit 110 to the pattern analysis unit 120 at predetermined time intervals (for example, every 50 ms sample period). Based on the representative point data Q (x, y, a, t) given in this way, the pattern analysis unit 120 analyzes the temporal change state of the contact state of the finger, and forms one of the categorized patterns. Perform the fitting process. Here, the pattern thus applied is called an “analysis pattern”. Eventually, the pattern analysis unit 120 performs processing for obtaining a predetermined analysis pattern based on a group of representative point data Q (x, y, a, t) given during a certain period. Here, an example of a specific processing method until obtaining the analysis pattern will be described in detail.

  FIG. 4 is a block diagram illustrating a detailed configuration of the pattern analysis unit 120. As shown in the figure, the pattern analysis unit 120 includes components of a representative point data storage unit 121, a stroke recognition unit 122, a characteristic recognition unit 123, and a pattern determination unit 124. Actually, each of these components is realized by a computer in which a dedicated program is incorporated.

<2-2-1: Representative point data storage unit>
The representative point data storage unit 121 is a component that stores the representative point data given from the representative point recognition unit 110. As described above, in the embodiment shown here, the representative point data is data in the form of Q (x, y, a, t). The figure shows an example in which representative point data Q obtained at each time point of time t = t1, t2, t3,..., T9 is stored in the representative point data storage unit 121. The representative point data Q11 (x11, y11, a11, t1) stored in the first line in the figure is data at the time t1, and the representative point data Q12 (x12 stored in the second line in the figure). , Y12, a12, t2) are data at the time t2. The third line in the figure shows a state in which two sets of representative point data Q13 (x13, y13, a13, t3) and Q23 (x23, y23, a23, t3) are stored. The two sets of data are both data at the same time t3. Hereinafter, two sets of data are stored up to the row of time t8, and one set of representative point data Q29 (x29, y29, a29, t9) is stored in the last row of time t9.

  Here, it is assumed that there is a pause period in which no representative point data is given before time t1, and a pause period in which no representative point data is given after time t9. Such a pause period occurs when a detection signal from the touch panel device 10 cannot be obtained, and indicates a state in which the user lifts his / her finger from the touch surface. In other words, the period from the time t1 to the time t9 indicates a state in which the user makes a gesture operation with the finger touching the touch surface. In this way, the pattern analysis unit 120 performs an analysis process as described below on a group of representative point data Q (x, y, a, t) given during a certain continuous period. become.

<2-2-2: Stroke recognition unit>
The stroke recognizing unit 122 is a group of representative point data Q (x, y, a, t) to be analyzed stored in the representative point data storage unit 121, that is, a predetermined pause period exists before and after. Data including continuous time t is read out, and a process of recognizing a stroke composed of an aggregate of a plurality of representative points indicating the same finger movement is performed based on the read out representative point data.

  For example, 15 sets of representative point data Q11 (x11, y11, a11, t1) to Q29 (x29, y29, a29, t9) stored in the representative point data storage unit 121 of FIG. Let us consider a case where the results shown in FIG. 5 are obtained by plotting the representative points Q11 to Q29 on the XY plane based on the coordinate values. Here, for example, the representative point Q11 is a point plotted at the position indicated by the coordinate value (x11, y11).

  This result was obtained when the finger touched the position of the representative point Q11 at time t1, the finger touched the position of the representative point Q12 at time t2, and the representative at time t3. The finger is in contact with the positions of points Q13 and Q23, the finger is in contact with the positions of representative points Q14 and Q24 at time t4, and the finger is in contact with the positions of representative points Q15 and Q25 at time t5. At time t6, the finger is in contact with the representative points Q16 and Q26, at time t7, the finger is in contact with the representative points Q17 and Q27, and at time t8, the representative points Q18 and Q28 This is because the finger is in contact with the position and the finger is in contact with the position of the representative point Q29 at time t9.

  If such 15 representative points are analyzed in consideration of the time, the eight representative points Q11 to Q18 indicated by a broken line on the left side of the figure are a plurality of representative points indicating the movement of the same finger. A stroke S1 composed of a set of points is configured, and the seven representative points Q23 to Q29 indicated by a broken line on the right side of the figure are also a set of a plurality of representative points indicating the same finger movement. It can be recognized that the stroke S2 consisting of Eventually, the stroke recognizing unit 122 has 15 sets of representative point data Q11 (x11, y11, a11, t1) to Q29 (x29, y29, a29, t9) stored in the representative point data storage unit 121 of FIG. Based on the above, it is possible to recognize two sets of strokes S1 and S2 each indicating the same finger movement.

A specific stroke recognition algorithm by the stroke recognition unit 122 is, for example, “obtained at the i-th time t _i based on a group of representative point data output from the representative point recognition unit 110 in time. The two representative points whose distance between the representative point and the representative point obtained at the (i-1) -th time t _i-1 is equal to or less than a predetermined reference d are recognized as representative points belonging to the same stroke. Can be employed.

  If the above algorithm is applied to the example shown in FIG. 5, two sets of strokes S1 and S2 are recognized by the following process. Here, it is assumed that the length shown at the upper end of FIG. 5 is set in advance as the distance reference d. In this case, first, it is determined whether or not the representative point Q12 at time t2 belongs to the same stroke with reference to the representative point Q11 at time t1. In the case of the illustrated example, since the distance d1 <d between the two, the representative point Q12 is recognized as a point belonging to the same stroke as the representative point Q11.

  Next, the determination about the representative points Q13 and Q23 at time t3 is performed. In this case, since the distance d2 <d for the representative point Q13, the representative point Q13 is recognized as a point belonging to the same stroke as the representative point Q12. On the other hand, for the representative point Q23, since the distance d3> d, the representative point Q23 is recognized as a point belonging to a stroke different from the representative point Q12.

  Subsequently, the determination for the representative points Q14 and Q24 at time t4 is performed. In this case, for the representative point Q14, since the distance d4 <d and the distance d5> d, the representative point Q14 belongs to the same stroke as the representative point Q13, but a stroke different from the representative point Q23. It is recognized as a point belonging to. On the other hand, for the representative point Q24, since the distance d6 <d and the distance d7> d, the representative point Q24 belongs to the same stroke as the representative point Q23, but has a different stroke from the representative point Q13. Recognized as a belonging point. Thereafter, by repeating the same processing, the representative points Q11 to Q18 are eventually recognized as points belonging to the same stroke S1, and the representative points Q23 to Q29 are recognized as points belonging to another same stroke S2. Will be.

  The figure shows an example in which two sets of strokes are recognized. Of course, there may be a case where only one set of strokes is recognized (when the user performs a gesture action with one finger), three or more sets. May be recognized (when the user performs a gesture motion with three or more fingers). The distance reference value d is shorter than the distance between two representative points obtained when the user touches the touch surface with two fingers aligned, and is 1 when the user moves the finger on the touch surface. An appropriate value may be set to be longer than the moving distance of the sample period (even if the user's finger moves quickly, an appropriate reference value d can be set by setting a short one sample period) .

  In practice, when the pattern analysis unit 120 performs the analysis, the pattern is analyzed based only on the representative point data Q (x, y, a, t) whose value a is equal to or greater than a predetermined reference value. It is preferable to do so. Specifically, when the stroke recognition unit 122 reads a group of representative point data Q (x, y, a, t) to be analyzed stored in the representative point data storage unit 121, the intensity value a is The representative point data Q (x, y, a, t) that is less than the predetermined reference may not be read out.

  Since the strength value a is a parameter indicating the pressing force of the user's finger, the representative point Q at which the strength value a is less than the predetermined reference is generated due to the influence of some noise even though the user's finger is not touching. There is a high possibility that it is caused by an unrelated finger such as a little finger accidentally touching the touch surface, contrary to the intention of the user. Therefore, if such a representative point Q is not included in the analysis target, a more accurate analysis can be performed.

<2-2-3: Characteristic recognition unit>
The characteristic recognizing unit 123 performs processing for recognizing the characteristic of each stroke recognized by the stroke recognizing unit 122. One stroke indicates the trajectory of the movement of one finger of the user. By recognizing the trajectory, information useful for analyzing the characteristics of the gesture motion performed by the user can be obtained. .

  In the embodiment shown here, four parameters of stroke length L, stroke position P (xp, yp), stroke orientation B, and stroke curvature C are calculated as values indicating stroke characteristics for one stroke. . The stroke characteristics indicated by these four parameters will be described below with reference to FIG.

  FIG. 6 is a plan view for explaining the stroke characteristics recognized by the characteristic recognition unit 123. The stroke S1 shown in FIG. 5 is shown in the center of the figure. As described above, the stroke S1 is composed of eight representative points from the representative point Q11 (time t1) to the representative point Q18 (time t8), and indicates a sliding operation from the top to the bottom by the user's finger. Is.

  The stroke length L is a parameter indicating the total length of the stroke S1. Specifically, in the embodiment shown here, “a plurality of representative points Q11 to Q18 constituting one stroke S1 are arranged in time series. It is defined as “total length of connecting lines when connected in order”. Strictly speaking, the stroke length L based on such a definition is slightly different from the total length of the actual movement trajectory of the user's finger, but there is no problem in practical use. In the case of the illustrated example, the length of the line segment Q11-Q12, the length of the line segment Q12-Q13, the length of the line segment Q13-Q14, the length of the line segment Q14-Q15, the length of the line segment Q15-Q16 The sum of the lengths of the line segments Q16 to Q17 and the lengths of the line segments Q17 to Q18 becomes the scroll length L. The stroke length L is used to determine whether the stroke is a “point stroke” or a “line stroke”, as will be described later.

  On the other hand, the stroke position P (xp, yp) is a parameter indicating the position of the stroke S1 on the XY plane, and is represented by a coordinate value (xp, yp). Specifically, as shown, a point P indicating a representative position of the stroke S1 (for example, the gravity center position of all the representative points Q11 to Q18) is determined, and the coordinate value (xp, yp) of this point P is determined as the stroke position. And it is sufficient. This stroke position P (xp, yp) is used for grasping the mutual positional relationship when there are a plurality of strokes, as will be described later. Moreover, in the modification described in §3-2, it is used to determine the area where the gesture is performed.

  The stroke direction B is a parameter indicating the direction of the entire stroke S1 on the XY plane. As a rough concept, as shown in the figure, an orientation vector W indicating the overall direction of the stroke S1 is defined, and an angle formed by this orientation vector W and a predetermined coordinate axis (X axis in the illustrated example) is defined as a stroke orientation. B can be determined. A specific method for calculating the stroke direction B will be described later. This stroke direction B is used to determine the direction of movement of the user's finger.

  The stroke curvature C is literally a parameter indicating the degree of curvature of the stroke S1. In the case of the embodiment shown here, as will be described later, it is defined as a parameter that increases the degree of curvature of the stroke as the degree of curvature of the stroke increases. The stroke curvature C is used to determine whether the stroke is a “linear stroke” or a “curve stroke”. In the embodiment shown here, the stroke curvature C is determined so that the sign is different between when the stroke rotates clockwise and when it rotates counterclockwise. Therefore, the stroke determined as “curve stroke” can be further determined as “clockwise curve stroke” or “counterclockwise curve stroke” by the sign of stroke curvature C.

Next, an example of a specific method for calculating the above-described four parameters (stroke characteristics) will be described. FIG. 7 is a graph showing the principle of the stroke characteristic calculation method. One stroke recognized by the stroke recognizing unit 122 is an aggregate of a plurality of n representative points Q that are ordered based on time series. Therefore, as shown in FIG. 7 (a), the i-th representative point Q _i , the (i−1) -th representative point Q _i−1 immediately before it, and the (i−2) -th previous point (i−2). ) Consider the positional relationship of the first representative point Q _i-2 . Here, the coordinate value of the representative point Q _i is (x _i , y _i ), the coordinate value of the representative point Q _i−1 is (x _i−1 , y _i−1 ), and the representative point Q _i−2 The coordinate value is assumed to be (x _i-2 , y _i-2 ). A vector from the representative point Q _i-1 to the representative point Q _i is V _i , a vector from the representative point Q _i-2 to the representative point Q _i-1 is V _i-1 , and the vector V _i is a vector V _{i. An} angle formed with respect to ₋₁ is a displacement angle φ _i, and an angle formed by a vector V _i with respect to a predetermined reference coordinate axis (X axis in this example) is an azimuth angle θ _i .

Here, with respect to the angle, an example is shown in which the counterclockwise direction is positive and the clockwise direction is negative. Thus, in the example shown in FIG. 7 (a), the displacement angle phi _i is actually a negative value, displaced in the clockwise direction of the vector V _i is the direction of the vector V _i-1 Is shown. Thus, the displacement angle φ _i is defined as a value in the range of −180 ° <φ _i ≦ + 180 °. Similarly, in the example shown in FIG. 7A, the azimuth angle θ _i actually takes a negative value, and the direction of the vector V _i is clockwise with respect to the positive direction of the X axis (the right direction in the figure). It is shown that it is displaced. This azimuth angle θ _{i is} also defined as a value within the range of −180 ° <θ _i ≦ + 180 °. As shown in FIG. 7B, 0 ° is the right direction and + 90 ° is the upper direction on the XY coordinate system. Direction, −90 ° indicates the downward direction, and ± 180 ° indicates the left direction.

  Now, with such a definition, the parameter values indicating the four stroke characteristics can be calculated by a specific arithmetic expression as shown in FIG. Hereinafter, the arithmetic expressions shown in FIG. 8 will be described in order.

First, the stroke length L is
L = _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
It is calculated by the following arithmetic expression. This expression is an arithmetic expression for obtaining the sum of the lengths of the line segments connecting the representative points in order from the first representative point Q1 to the nth representative point Qn.

On the other hand, the stroke position P is indicated by coordinate values (xp, yp), and these coordinate values are
xp = _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 · x _i
/ _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
yp = Σ _{i = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 · y _i
/ _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
Is calculated by the following calculation. This equation is a weighted average value of the X coordinate value and the Y coordinate value of each representative point from the second representative point Q2 to the nth representative point Qn. That is, the term ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^{1/2 of} the numerator becomes the coordinate values x _i and y _i of the i th representative point Q _i. It is a weight to be multiplied and corresponds to the length of the vector V _i . Of course, the simple average value of the X coordinate value and the Y coordinate value of each representative point from the first representative point Q1 to the nth representative point Qn may be used as the coordinate value (xp, yp). However, if a weighted average value using the length of the vector V _i as a weight is used, a stroke position P more reflecting the position of the entire stroke can be obtained.

The stroke azimuth B can be obtained as an average value of the azimuth angles θ _i shown in FIG. Since the azimuth angle θ _i indicates the azimuth of the vector V _i , the azimuth of the entire stroke can be obtained by obtaining the average of these. However, in the embodiment shown here, as in the case of calculating the stroke position P, so that obtaining the weighted average value for the weight of the length of the vector V _i. That is, the stroke direction B is
B = _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 · θ _i
/ _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
Is calculated by the following calculation. Here too, the term ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 of the numerator is a weight multiplied by the azimuth angle θ _i of the vector V _i , This corresponds to the length of the vector V _i .

The stroke curvature C can be obtained as an average value of the displacement angles φ _i shown in FIG. The displacement angle φ _i is a parameter indicating how much the vector V _i is bent in which direction (counterclockwise or clockwise) with respect to the vector V _i−1 . Therefore, for example, if the average value is close to 0, the stroke is not bent in the whole counterclockwise or clockwise direction, indicating that it is close to a straight line. As the absolute value of the average value is larger, the curve is curved in either direction. In the embodiment shown here, also when obtaining the average value of the displacement angle φ _i , a weighted average value with the length of the vector V _i as a weight is obtained. That is, the stroke curvature C is
C = Σ _{i = 3 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 · φ _i
/ Σ _{i = 3 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
Is calculated by the following calculation. Again, the term ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 of the numerator is the weight multiplied by the displacement angle φ _i , and the vector V _i Corresponds to the length. Incidentally, the displacement angle phi _i, relative to the vector _{V i-1} extending from the representative point _{Q i-2} to the representative point _{Q i-1,} the vector _{V i} is formed extending from the representative point _{Q i-1} to the representative point _{Q i} Since it is an angle, it is an angle defined when i ≧ 3. For this reason, the denominator and numerator of the above calculation formula for obtaining the stroke curvature C is the sum of i = 3 to n.

<2-2-4: Pattern determination unit>
Based on the recognition results by the stroke recognition unit 122 and the characteristic recognition unit 123, the pattern determination unit 124 associates one stroke having a unique characteristic or a combination of a plurality of strokes with one pattern, thereby obtaining an analysis pattern. Perform the decision process. In the embodiment shown here, individual strokes are classified into a plurality of types based on their characteristics, and a process for determining an analysis pattern is performed by associating a specific analysis pattern with a specific type or combination thereof. It will be.

  The simplest stroke classification method is a method of classifying into “point stroke” and “line stroke” based on the stroke length L. That is, a stroke having a stroke length L less than a predetermined reference value Lth can be classified into a type “point stroke”, and a stroke having a stroke length L greater than the reference value Lth can be classified into a type “line stroke”. In other words, the “point stroke” is a stroke in which the user's finger hardly moves, and the “line stroke” can be said to be a stroke in which the user's finger moves to some extent.

  Further, when the stroke curvature C is combined with the stroke length L, the stroke length L can be classified into “point stroke”, “linear stroke”, and “curve stroke”. That is, of the strokes classified into the type of “line stroke” by the classification using the stroke length L described above, the strokes whose absolute value of the curvature C is less than the predetermined reference value Cth are classified into the type of “linear stroke”. It is only necessary to classify strokes whose absolute value of curvature C is equal to or greater than a reference value Cth into a type of “curve stroke”.

  Even if the user performs a gesture operation on the touch surface with the intention of moving the finger linearly, the movement trajectory rarely becomes a complete straight line. It will not be zero. Therefore, if an appropriate degree of curvature C, which is a limit that can be determined as linear movement, is set to the reference value Cth, whether the gesture action performed by the user is intended for a straight line or a curved line. Can be distinguished. In order to set a specific numerical value of the reference value Cth, for example, when some users are instructed to “perform a gesture for moving a finger linearly” and “move a finger in a curved line” A method of calculating the stroke curvature C based on the respective detection signals obtained when giving an instruction to “perform gesture” and setting an intermediate value between the two as a reference value Cth may be adopted.

  As described above, when the stroke length L and the stroke curvature C are used, the strokes can be classified into three types of “point stroke”, “linear stroke”, and “curve stroke”. Therefore, when it is assumed that the user performs a gesture operation with one finger, three analyzes of “point stroke” pattern, “linear stroke” pattern, and “curve stroke” pattern are provided for these three types, respectively. Patterns can be associated. It is also possible to add a gesture operation with two fingers. As for the gesture operation with two fingers, a specific analysis pattern can be associated with a combination of two stroke types. For example, a “point stroke + linear stroke” pattern, a “linear stroke + curve stroke” pattern, and so on. Of course, the same applies to gesture operations with three or more fingers.

  Further, for a pattern including a plurality of strokes, the mutual positional relationship can be grasped by referring to the stroke position P, and the pattern can be determined by distinguishing patterns having different mutual positional relationships. For example, a “left point stroke + right straight stroke” pattern, an “upper straight stroke + lower curve stroke” pattern, and so on.

  In the embodiment described here, in order to further increase the types of strokes, for the type of “curve stroke”, “clockwise curve stroke” and “counterclockwise curve stroke” are further based on the sign of stroke curvature C. It is handled by dividing it into two types. As described above, the stroke curvature C is defined so that the sign differs between when the stroke rotates clockwise and when it rotates counterclockwise, so refer to the sign of the stroke curvature C. For example, it is possible to classify “clockwise curved stroke” and “counterclockwise curved stroke”.

  On the other hand, for the type “straight stroke”, the type “straight stroke” is further divided into four types having different directions based on the stroke direction B. As shown in FIG. 7B, the stroke direction B is a parameter indicating a numerical value of −180 ° to + 180 °, and plays a role of indicating a specific direction on the XY plane. In the case of the embodiment described here, “downward linear stroke” when −135 ° <B ≦ −45 °, “rightward linear stroke” when −45 ° <B ≦ + 45 °, + 45 ° < When B ≦ + 135 °, “upward linear stroke” is classified, and when + 135 ° <B or −135 ° ≧ B, “leftward linear stroke” is classified.

  After all, in the embodiment described here, the pattern determination unit 124 converts each stroke recognized by the stroke recognition unit 122 into three stroke characteristics recognized by the characteristic recognition unit 123 (stroke length L, stroke curvature C). , Stroke direction B), “point stroke”, “clockwise curved stroke”, “counterclockwise curved stroke”, “upward linear stroke”, “downward linear stroke”, “leftward linear stroke” ”And“ Right straight stroke ”are classified into any of the seven types.

  And when there is only one stroke recognized by the stroke recognition part 122, the process which determines an analysis pattern based on the said specific classification is performed. For example, an analysis pattern called a “clockwise curved stroke” pattern is associated with a gesture action including only a single stroke of the type “clockwise curved stroke”. Eventually, any of the seven analysis patterns is associated with a gesture operation including only a single stroke.

  On the other hand, when there are a plurality of strokes recognized by the stroke recognition unit 122, a process of associating a specific analysis pattern with a combination of individual stroke types is performed. At this time, by referring to the stroke position P, the mutual positional relationship is grasped for a plurality of strokes, and patterns having different mutual positional relationships are distinguished and determined. For example, for a gesture motion that includes two strokes that are in the left-right positional relationship, the pattern of “clockwise curved stroke on the left side + downward straight stroke on the right side” and “clockwise curved stroke on the right side + downward on the left side” The analysis pattern is different from the “linear stroke” pattern.

  FIG. 9 is a flowchart showing a procedure of pattern determination processing performed by the pattern determination unit 124 based on the above-described policy. First, in step S11, the parameter j indicating the stroke number is set to an initial value 1, and in step S12, the characteristic calculated by the characteristic recognition unit 123 for the jth stroke Sj, that is, the stroke length L and the stroke position. Four parameter values, P, stroke orientation B, and stroke curvature C, are extracted. In step S13, the stroke length L is compared with a predetermined reference value Lth. If L <Lth, the stroke is recognized as a “point stroke” in step S14.

  On the other hand, if L ≧ Lth in step S13, it is recognized as “line stroke”, and in step S15, the absolute value of stroke curvature C is compared with a predetermined reference value Cth. If | C | ≧ Cth, the stroke is recognized as a “curve stroke” in step S16. If | C | <Cth, the stroke is determined as “straight stroke” in step S18. Is recognized.

  If it is recognized in step S16 that “curve stroke”, the bending direction is determined by the sign of the degree of curvature C in step S17. That is, when the degree of curvature C is positive, it is recognized as “counterclockwise curved stroke”, and when the degree of curvature C is negative, it is recognized as “clockwise curved stroke”. On the other hand, when the recognition of “straight stroke” is made in step S18, the determination of up, down, left and right is made based on the stroke direction B in step S19 as described above.

  In any case, in step S20, any of the seven types is associated with the jth stroke Sj. That is, a specific type is determined for the stroke Sj. The symbols by the white figure shown in the block of step S20 in FIG. 9 are the seven types, that is, “point stroke”, “clockwise curve stroke”, “counterclockwise curve stroke”, “upper”. "Directional linear stroke", "Downward linear stroke", "Left linear stroke", "Right straight stroke". Hereinafter, in the drawings of the present application, these seven types of strokes are indicated by symbols by these white figures.

  Thus, when a predetermined type is determined for the jth stroke Sj, the process proceeds to step S22 via step S21, the parameter j indicating the stroke number is updated by 1, and the processing from step S12 is repeatedly executed again. Is done. When such processing is completed for all strokes recognized by the stroke recognition unit 122, the process proceeds to step S23 through step S21, and a predetermined analysis pattern is determined based on the type of stroke and the stroke position P. .

  As described above, when there is only one stroke recognized by the stroke recognition unit 122, an analysis pattern corresponding to the type of the stroke is associated, but when there are a plurality of recognized strokes, their mutual positions One analysis pattern is associated with a combination of types considering the relationship.

  FIG. 10 is a plan view showing some examples of analysis patterns determined by the pattern determination unit 124. The patterns PT (A) and PT (B) are both analysis patterns when there is only one stroke recognized by the stroke recognition unit 122, and are obtained when the user performs a gesture operation with one finger. Analysis pattern. That is, the pattern PT (A) is a pattern obtained by analyzing a gesture motion including only a single stroke of the type “clockwise curved stroke”, and the pattern PT (B) is “downward linear stroke”. This is a pattern obtained by analyzing a gesture motion including only a single stroke of a certain type.

  On the other hand, the patterns PT (C) to PT (E) are analysis patterns when there are two strokes recognized by the stroke recognition unit 122, and when the user performs a gesture operation with two fingers. This is an analysis pattern obtained. As described above, when a plurality of strokes are recognized, one analysis pattern is associated with a combination of types in consideration of the mutual positional relationship. Accordingly, the patterns PT (C) to PT (E) are patterns that are combinations of “clockwise curved strokes” and “downward linear strokes”, but are different analysis patterns. That is, the pattern PT (C) is a “left clockwise curved stroke + right downward linear stroke” pattern, and the pattern PT (D) is “left downward linear stroke + right clockwise curved stroke”. The pattern PT (E) is a “upper downward linear stroke + lower clockwise curve stroke” pattern.

  What is necessary is just to determine the mutual positional relationship of two strokes based on each stroke position P (xp, yp). In the embodiment shown here, the mutual positional relationship is determined to be either “left-right relationship” or “up-down relationship”. Specifically, the position of the first stroke S1 is P1 (xp1, yp1), the position of the second stroke S2 is P2 (xp2, yp2), and the positional deviations Δx = | xp1-xp2 |, Δy = | yp1-yp2 | is determined, and when [Delta] x> [Delta] y, it is determined as "right and left relation", and when [Delta] x <[Delta] y, it is determined as "upper and lower relation". When the “right / left relation” is determined, it is determined which stroke is on the left side and which stroke is on the right side depending on the magnitude of the X coordinate value of the stroke position P. ”Is determined based on the magnitude of the Y coordinate value of the stroke position P, which stroke is the upper side and which stroke is the lower side.

<2-3: Specific Processing of Sample Pattern Collection Unit and Storage Unit>
Next, specific processing performed by the sample pattern collection unit 130 and the sample pattern storage unit 150 in the apparatus shown in FIG. 1 will be described. The sample pattern collection unit 130 displays an instruction to “perform a plurality of m gestures on the touch surface” on the screen of the display device 20, and a pattern analysis unit for the m gesture operations of the user based on the instruction Each analysis pattern obtained by 120 is collected as a sample pattern.

  As the “plural m-number of gestures” instructed to the user, any gesture may be defined as long as it is a gesture using a finger. In particular, there are various variations in gestures using a plurality of fingers as well as one finger. FIG. 11 is a plan view illustrating types of gestures using two fingers. The illustrated black arrow indicates the direction of movement of the two fingers on the touch surface, and shows six different gestures G1 to G6.

  Gesture G1 is a “up scroll” gesture that slides two fingers from bottom to top, and gesture G2 is a “down scroll” gesture that slides two fingers from top to bottom, The gesture G3 is a “left scroll” gesture that slides two fingers from right to left, and the gesture G4 is a “right scroll” gesture that slides two fingers from left to right. The gesture G5 is a “clockwise” gesture that rotates two fingers in a clockwise direction, and the gesture G6 is a “counterclockwise rotation that rotates two fingers in a counterclockwise direction. ”Gesture.

  Here, for convenience of explanation, an example will be described in which processing for registering a sample pattern is performed for each of the six gesture operations shown in FIG. In practice, various gesture operations can be registered as necessary. Of course, a gesture operation using one finger and a gesture operation using three or more fingers can also be registered.

  In order to perform such registration, the user first instructs the sample pattern collection unit 130 to perform initial setting. Actually, an operation for starting a program for causing a computer to function as the sample pattern collection unit 130 may be performed. Then, the sample pattern collection unit 130 displays a screen for instructing gesture registration in order for each of the six gesture operations on the display device 20, and the pattern analysis unit 120 obtains the user's gesture operation based on the instruction. The analysis pattern is collected as a sample pattern for the gesture.

  FIG. 12 is a plan view showing an example of an instruction screen displayed on the display device 20 by the function of the sample pattern collection unit 130. This instruction screen is an instruction screen for registering the gesture G1 “up scroll” shown in FIG. 11, and instructs the user to perform a gesture for moving two fingers in parallel upward on the touch surface. It is a screen. The user who sees this instruction performs the instructed gesture operation on the touch surface of the touch panel device 10. The gesture motion performed by the user is analyzed by the pattern analysis unit 120, and a predetermined analysis pattern is associated with the gesture motion. The sample pattern collection unit 130 collects the analysis pattern thus obtained as a sample pattern for the gesture.

  The sample pattern storage unit 150 functions to store each sample pattern thus collected. That is, the analysis pattern collected when the sample pattern collection unit 130 displays an instruction to perform the kth (1 ≦ k ≦ m) gesture is stored as a sample pattern corresponding to the kth gesture. To do.

  FIG. 13 is a plan view showing an example of the sample pattern stored in the sample pattern storage unit 150 in this way. The six sample patterns SP (G1) to SP (G6) shown here are sample patterns corresponding to the six types of gestures G1 to G6 shown in FIG. 11, respectively.

  Here, the sample patterns SP (G1), SP (G3), and SP (G5) are standard sample patterns, and the types of strokes constituting the pattern (indicated by white graphics) are shown in FIG. It corresponds to the type of black arrow of the gestures G1, G3, G5 shown. However, the sample patterns SP (G2), SP (G4), and SP (G6) slightly deviate from the standard pattern.

  For example, since the sample pattern SP (G2) corresponds to the gesture G2 “down scroll” shown in FIG. 11, both the left and right strokes are originally “down straight strokes” (downward white arrows). In fact, the left stroke is a “clockwise curve stroke”. The reason why such a result was obtained is that this user himself intends to perform a “gesture that moves two fingers in parallel downward”, but in actuality, the movement of the left finger is a clock. It was because it was rotating around. That is, when the user moves the two fingers in parallel downward, the user has a peculiar habit that the movement of the left finger rotates clockwise.

  Similarly, the sample pattern SP (G4) is unique in that when the user is caused to move two fingers in parallel in the right direction, the movement of the lower finger rotates clockwise. It shows that there is a habit. The sample pattern SP (G6) also shows that this user has a peculiar habit that the right finger hardly moves when both fingers are rotated counterclockwise. ing. An important feature of the present invention is that a user's unique habit is registered in advance as a sample pattern for each gesture by the gesture registration work performed in the preparation stage. As described above, if the sample pattern is registered in advance in the preparation stage, the operation input performed by the user can be accurately detected by referring to the sample pattern in the detection stage.

  Of course, since each user has different gesture habits, if another user performs a registration operation, the sample patterns stored in the sample pattern storage unit 150 also differ. FIG. 14 is a plan view showing sample patterns stored in the sample pattern storage unit 150 by another user registration operation. The example shown in FIG. 14 is also a sample pattern obtained for the six kinds of gestures shown in FIG. 11, but it can be seen that patterns having different contents are registered as compared with the example shown in FIG.

<2-4: Processing of Operation Input Pattern Collection Unit / Pattern Selection Unit / Detection Result Output Unit>
The operation input pattern collection unit 140 performs a process of collecting an analysis pattern obtained by the pattern analysis unit 120 as an operation input pattern after a detection request signal for requesting detection of an operation input is given from the external unit 200. Then, the pattern selection unit 160 compares the operation input pattern thus collected with a plurality of m types of sample patterns stored in the sample pattern storage unit 150, and performs a process of selecting a matching sample pattern. Finally, the detection result output unit 170 performs processing to output information indicating a gesture corresponding to the sample pattern selected by the pattern selection unit 160 to the external unit 200 as a detection result signal indicating the detection result of the operation input.

  As described in §1, in the external unit 200, a program for performing some processing is executed, and when the program requires some operation input from the user, the external unit 200 transmits to the digital processing unit 100. A detection request signal is issued. In this case, the program executed in the external unit 200 is designed to continue processing by recognizing, for example, the six gestures G1 to G6 shown in FIG. 11 as specific operation inputs, respectively.

  Here, as an example, after the detection request signal from the external unit 200 is given, the user's gesture operation is performed on the touch panel device 10, and the pattern analysis unit 120 analyzes the gesture operation, so that FIG. Assume that the pattern PT (C) shown in FIG. In this case, the operation input pattern collection unit 140 collects the pattern PT (C) shown in FIG. 10 as an operation input pattern, and performs a process of delivering it to the pattern selection unit 160. The pattern selection unit 160 compares the delivered pattern PT (C) with the six sample patterns SP (G1) to SP (G6) stored in the sample pattern storage unit 150 shown in FIG. The selected sample pattern SP (G2) is selected. As a result, the detection result output unit 170 outputs information indicating the gesture G2 corresponding to the sample pattern SP (G2) to the external unit 200 as a detection result signal. If no matching sample pattern is found, for example, an error message may be returned as a detection result signal.

  Thus, a detection result signal indicating that an operation input corresponding to the gesture G2 has been made by the user is returned to the external unit 200. The program being executed in the external unit 200 will continue the predetermined process in response to the operation input. For example, when there is an operation input corresponding to the gesture G1 shown in FIG. 11, the screen is scrolled upward, and when there is an operation input corresponding to the gesture G5, the object is rotated clockwise. Such processing is performed.

  In the above example, although the user intends to move the two fingers downward in parallel with respect to the gesture G2 “down scroll with two fingers”, the movement of the left finger is actually clockwise. It has a unique habit of rotating. In the conventional operation input detection device, when the gesture G2 “down scrolling with two fingers” is performed without considering such a peculiar wrinkle, two “downward linear strokes” arranged side by side are arranged. "Is obtained on the premise that" "is obtained. Therefore, in the above example, correct detection cannot be performed. In the present invention, such a unique wrinkle for a user is registered as a sample pattern, so that an accurate detection process considering the unique wrinkle is possible.

<<< §3. Various modifications >>
The configuration of the basic embodiment of the operation input detection device according to the present invention has been described in Section 1 and the specific processing by each component has been described in Section 2. Therefore, here, various modifications of the present invention will be described.

<3-1: Modification Example for Specifying User>
FIG. 15 is a block diagram illustrating a configuration of an operation input detection device using a touch panel according to a modification of the present invention. The difference from the basic embodiment shown in FIG. 1 is that a digital processing unit 100 A is used instead of the digital processing unit 100. A major feature of the digital processing unit 100A is that a user specifying unit 180 is added to the digital processing unit 100.

  The user specifying unit 180 is a component having a function of inputting information for specifying a user currently in use and storing it. In the case of the example shown here, the user identification code U is given from the external unit 200 to the user identification unit 180, and the user identification unit 180 inputs this user identification code U as information for identifying the user currently in use. And save. This is because, in the case of a general electronic device, it can be considered that the user who is using the external unit 200 is the same as the user who is using the operation input detection device according to the present invention.

  For example, when all the blocks shown in FIG. 15 are configured by a personal computer incorporating a predetermined program, the user logs in to the personal computer using a predetermined user identification code U (account) and uses it. It will be. In this case, since the user identification code U is stored in the external unit 200, the user specifying unit 180 captures the user identification code U stored in the external unit 200, thereby identifying the user currently in use. Can be identified.

  On the other hand, the function of the sample pattern collection unit 130A is substantially the same as the function of the sample pattern collection unit 130 in the apparatus shown in FIG. However, the process which collects a sample pattern independently for every user specified by the user specific | specification part 180 is performed. Similarly, the function of the sample pattern storage unit 150A is substantially the same as the function of the sample pattern storage unit 150 in the apparatus shown in FIG. However, the process which stores a sample pattern independently for every user specified by the user specific | specification part 180 is performed. As can be seen by comparing FIG. 13 and FIG. 14, even if sample patterns are collected for exactly the same gesture, the user may have unique habits, so the obtained sample patterns may vary from user to user. . In the modification shown in FIG. 15, the sample pattern for each user is stored in the sample pattern storage unit 150A.

  FIG. 16 is a plan view showing an example of a sample pattern for each user stored in the sample pattern storage unit 150A of the operation input detection device shown in FIG. In this example, independent sample patterns SP (Ua), SP (Ub), and SP (Uc) are stored for three users A, B, and C having user identification codes Ua, Ub, and Uc, respectively. The state is shown. Here, each sample pattern SP (Ua), SP (Ub), SP (Uc) is configured by six sample patterns as shown in FIG. 13, for example.

  The function of the pattern selection unit 160A is substantially the same as the function of the pattern selection unit 160 in the apparatus shown in FIG. However, when performing pattern selection, pattern selection is performed by comparing with the sample pattern for the user specified by the user specifying unit 180. For example, when the user identification code Ua is stored in the user specifying unit 180 and it is specified that the currently used user is the user A, the pattern selecting unit 160A displays the user in the sample pattern storing unit 150A. A pattern selection is performed by making a comparison using the sample pattern SP (Ua) for A.

  As described above, according to the modification shown in FIG. 15, even when use by a plurality of users is planned, it is possible to detect an accurate operation input in consideration of features unique to each user.

<3-2: Modified example for specifying region>
In FIG. 1 and FIG. 15, for convenience of explanation, the touch panel device 10 and the display device 20 are shown as separate blocks. However, the touch panel device 10 and the display device 20 are each a display device with a touch panel whose display screen forms a touch surface. It is also possible to configure by. Recently, tablet-type electronic devices, multi-function mobile phones, and portable game machines have become widespread, and display devices with a touch panel are usually used in these devices.

  In an electronic device in which such a display device with a touch panel is incorporated, the user can touch an arbitrary position on the display screen. Therefore, the larger the display screen, the wider the touch surface. However, when the touch surface becomes wide, there is a possibility that a difference occurs in the obtained analysis pattern depending on the place where the gesture is performed. Recently, electronic devices incorporating a display device with a touch panel having a display screen of about 17 inches are also commercially available. Consider a case where such an electronic device is held with the left hand in a horizontally long state and various gesture operations are performed on the screen (touch surface) using the finger of the right hand. In this case, when the gesture is performed at the left end of the screen and when the gesture is performed at the right end, even if the user intends to perform the same gesture, the analysis pattern obtained from the pattern analysis unit 120 is different. There is a possibility. This tendency becomes more prominent when a plurality of display devices with a touch panel are arranged and used as a larger display device with a touch panel.

  In order to deal with such a case, the touch surface of the touch panel device may be divided to define a plurality of areas, and individual sample patterns may be collected for each area. By doing so, it is possible to accurately detect an operation input by recognizing the main area where the gesture has been performed and comparing it with the sample pattern for the recognition area.

  FIG. 17 is a plan view showing an example in which the touch surface (display surface) of the display device with a touch panel is divided into four regions T1 to T4, and sample patterns are separately collected for each region. In the lower column of the figure, the analysis pattern obtained from the pattern analysis unit 120 when the user is instructed to perform the gesture G5 (clockwise gesture with two fingers) shown in FIG. The example taken as a sample pattern of each region is shown.

  Looking at the results in the lower column of FIG. 17, the sample pattern SP (G5T1) for the region T1 and the sample pattern SP (G5T2) for the region T2 are both “clockwise curved stroke on the left side + clockwise curved stroke on the right side. The sample pattern SP (G5T3) for the region T3 is a “left dot stroke + right clockwise curve stroke” pattern, and the sample pattern SP (G5T4) for the region T4 is “ "Downward straight stroke + right clockwise stroke" pattern. These four sample patterns are all the sample patterns SP (G5) for the gesture G5, but their contents differ depending on the region. This is thought to be a phenomenon caused by biological structures such as the human skeleton and muscles.

  In this way, the present invention is applied to a touch panel device having a relatively large area in which the analysis pattern obtained for each region may be different even though the user intentionally performs the same gesture. If a sample pattern corresponding to an operation input pattern by a specific gesture is selected, a sample for the region where the gesture is performed is collected. It is preferable to make a comparison with the pattern.

  FIG. 18 is a block diagram showing a configuration of an operation input detection device according to a modified example having such a function. The difference from the basic embodiment shown in FIG. 1 is that the touch panel device 10 and the display device 20 are replaced by a display device 30 with a touch panel (as will be described later, in this embodiment, a gravitational acceleration sensor is incorporated). And a digital processing unit 100B is used instead of the digital processing unit 100. As described above, the touch panel-equipped display device 30 is a device in which a touch panel is provided on the display screen, and the touch panel device 10 and the display device 20 can be used together.

  The sample pattern collection unit 130B in the digital processing unit 100B is a component having substantially the same function as the sample pattern collection unit 130 in the apparatus shown in FIG. However, for a plurality of areas T1 to T4 obtained by dividing the touch surface of the touch panel, an instruction to perform a gesture separately is displayed, and individual sample patterns are collected for each of the areas T1 to T4. Perform the process.

  FIG. 19 is a plan view showing an example of an instruction screen displayed on the touch panel-equipped display device 30 by the sample pattern collection unit 130B. In this example, an area T1 on the screen is hatched, and an instruction is given to perform a gesture (gesture G1 shown in FIG. 11) for moving two fingers in parallel in the hatched area. . When the user who sees this instruction performs the gesture within the hatched area on the screen, the analysis pattern obtained by analyzing this is the sample pattern SP for the area T1 of the gesture G1. (G1T1) is stored in the sample pattern storage unit 150B.

  Similarly, the sample pattern collection unit 130B issues an instruction to perform the gesture G1 for the regions T2, T3, and T4, and the analysis patterns obtained at that time are obtained for the regions T2, T3, and T4 of the gesture G1, respectively. Are stored in the sample pattern storage unit 150B as sample patterns SP (G1T2), SP (G1T3), and SP (G1T4). Of course, for other gestures G2 to G6, sample patterns are collected in the same manner for the regions T1 to T4, respectively.

  The sample pattern storage unit 150B in the digital processing unit 100B is a component having substantially the same function as the sample pattern collection unit 130 in the apparatus shown in FIG. However, each sample pattern collected for each of the regions T1 to T4 is stored separately and independently. FIG. 20 is a plan view illustrating an example of a sample pattern for each region stored in the sample pattern storage unit 150B. Here, the illustrated sample pattern SP (T1) indicates sample patterns SP (G1T1) to SP (G6T1) collected for the six gestures G1 to G6 performed in the region T1 shown in FIG. Similarly, sample patterns SP (T2), SP (T3), and SP (T4) are patterns obtained for six gestures G1 to G6 performed in the regions T2, T3, and T4 shown in FIG. .

  On the other hand, the pattern selection unit 160B in the digital processing unit 100B is a component having substantially the same function as the pattern selection unit 160 in the apparatus shown in FIG. However, since different sample patterns are stored for each region in the sample pattern storage unit 150B, the pattern selection unit 160B displays the main region where the gesture based on the collected operation input pattern is performed. The pattern is selected by recognizing and comparing with the sample pattern for the recognition area.

  In order to recognize the main area where the gesture is performed, the analysis result of the pattern analysis unit 120 may be used. As already described, in the analysis processing process of the pattern analysis unit 120, the stroke is recognized from the detection signal indicating the temporal transition of the representative point, and the stroke position P (xp, yp) is calculated for each stroke. Therefore, the pattern selection unit 160B can recognize the main region where the gesture is performed based on the stroke position P (xp, yp).

  For example, when the operation input pattern includes only a single stroke, it is determined to which of the regions T1 to T4 the stroke position P (xp, yp) for the stroke belongs, and the stroke position P ( The region including xp, yp) may be recognized as the main region where the gesture is performed. In addition, when the operation input pattern includes a plurality of strokes, for example, the center of gravity position of the stroke position P (xp, yp) of the plurality of strokes is calculated, and the gesture including the region including the center of gravity position is performed. What is necessary is just to certify as a main field.

  When the gesture is performed near the boundary of the region, for example, a situation may occur in which the left stroke is included in the region T1 and the right stroke is included in the region T2. Even in such a case, if the region including the center of gravity position of both stroke positions is recognized as the main region where the gesture is performed and the operation input pattern is compared with the sample pattern for the recognized region, the pattern selection unit 160B The processing by will not be delayed. However, if a more flexible response is to be taken, it is preferable to perform selection processing with reference to the sample patterns in both areas where the gesture has been performed. In that case, first, a process for comparing with the sample pattern for the region including the center of gravity position is performed. As a result, if there is no matching sample pattern, secondly, any stroke is detected. The selection procedure of performing the process of comparing with the sample pattern for the region including the stroke position P may be adopted.

  In FIG. 17, the touch surface is divided into four in the horizontal direction to define four rectangular regions T1 to T4. However, the touch surface division method and the number of divisions are arbitrary. Therefore, it may be divided in the vertical direction or may be divided in both vertical and horizontal directions. Of course, the shape of each region obtained by the division is not limited to a rectangle. For example, the region may be divided into concentric circles to form an arcuate region.

  Also, as in the example shown in the figure, when the touch panel device has a rectangular touch surface, it is possible to distinguish whether the user is using in a vertically long state or in a horizontally long state. This makes it possible to detect operation inputs more accurately. For example, in the case of a device having a rectangular touch surface as shown in FIG. 17, even if the regions T1 to T4 are defined in the same division mode, the device is used in a horizontally long state as shown in FIG. As shown in FIG. 21 (b), there is a high possibility that the analysis patterns obtained for the same gesture are different from those in the case of using in the vertically long state. For example, even if the same region T1 is used in the horizontally long state as shown in FIG. 21 (a), it is the leftmost region, but if it is used in the vertically long state as shown in FIG. Since the region is located, even if the user performs a gesture in the same region T1, the gesture is substantially performed in a different region depending on whether the usage state is horizontally long or vertically long.

  Therefore, when the touch panel device has a rectangular touch surface, the sample pattern collection unit 130B is independent of the horizontal mode in which the touch surface is used in the horizontal state and the vertical mode in which the touch surface is used in the vertical state. The sample pattern may be collected, and the sample pattern storage unit 150B may store the sample pattern separately for the landscape mode and the portrait mode.

  Specifically, in the case of the landscape mode, four regions of a landscape T1 region, a landscape T2 region, a landscape T3 region, and a landscape T4 region are defined in order from the left. In the portrait mode, the portrait T1 is defined in order from the top. Four regions may be defined: a region, a vertically long T2 region, a vertically long T3 region, and a vertically long T4 region. In this case, substantially, the sample patterns are collected independently for each of the eight patterns.

  In order to take appropriate measures when the left and right sides of the device are switched in landscape mode or when the top and bottom sides of the device are switched in portrait mode, each area is not set to a specific physical part of the touch panel. Instead, it is preferable that the setting is made based on the spatial positional relationship at that time. For example, the “laterally long T1 region” does not mean a specific part unique to the touch panel, but means a region on the leftmost side when the touch panel is held horizontally.

  In this case, the pattern selection unit 160B has a function of recognizing the current mode (horizontal mode or vertical mode), and performs pattern selection by comparing with the sample pattern for the recognition mode. Be able to do that. Although the mode recognition can be performed by allowing the user to input which mode is being used, in the embodiment shown here, the display device with a touch panel 30 is a sensor for detecting the gravitational acceleration (gravity of gravity). The pattern selection unit 160B can recognize the current posture of the device based on the output signal of the sensor and is being used in the landscape mode. Or in use in portrait mode.

  As described above, the modification example for specifying the region has been described with reference to FIG. 18. Of course, the modification example (FIG. 15) for specifying the user described in §3-1 can be combined with this modification example. is there. In this case, the sample pattern is collected independently for each individual user and for each region, and stored in the sample pattern storage unit 150B. The pattern selection unit 160B compares the operation input pattern with a sample pattern for a specific area of a specific user, and selects a corresponding pattern.

<3-3: Diversification of analysis patterns>
In the basic embodiment described above, as described in §2-2-4, the pattern determination unit 124 assigns each stroke to “point stroke”, “clockwise curve stroke”, “counterclockwise curve stroke”, Processes are classified into one of the seven types of "upward linear stroke", "downward linear stroke", "leftward linear stroke", and "rightward linear stroke", and one analysis is performed based on the combination. Although the process of applying a pattern has been performed, the stroke classification types are not necessarily limited to the above seven types, and finer classification can also be performed.

  For example, in the embodiment described in §2-2-4, based on the stroke direction B, “straight stroke” is handled by dividing it into four types of up, down, left, and right. However, it is not always necessary to divide into four types of top, bottom, left, and right, and in general, it is possible to handle it by dividing it into a plurality of W types having different orientations.

  FIG. 22 is a plan view showing an example in which W = 36 is set and “straight stroke” is classified into 36 types based on the stroke direction B. According to this classification, the “straight line stroke” is subdivided according to a fine direction in units of 10 °. The white arrows in the figure are symbols indicating these 36 types. Since the stroke direction B is a parameter having a value of −180 ° to + 180 °, the angle type closest to the angle indicated by the stroke direction B of “straight stroke” is selected from the 36 angle types shown in FIG. By selecting, the “linear stroke” can be classified into 36 subdivided types.

  Of course, the “curve stroke” can be subdivided. In the embodiment described in §2-2-4, it is classified as either “clockwise curved stroke” or “counterclockwise curved stroke” with reference to only the sign of stroke curvature C. By combining the stroke orientation B, subdivision can be performed. FIG. 23 is a plan view showing an example in which “curve strokes” are classified into eight types in total by referring to the stroke direction B. FIG. The four types shown in the upper part of the figure are all “clockwise curved strokes”, but depending on the stroke direction B, “right clockwise curved stroke α1”, “downward clockwise curved stroke α2”, “left It is subdivided into “clockwise curved stroke α3” and “upward clockwise curved stroke α4”. Similarly, the four types shown in the lower part of the figure are all “counterclockwise curve stroke”, but depending on the stroke direction B, “left counterclockwise curve stroke β1”, “upward counterclockwise curve” It is subdivided into “stroke β2”, “right counterclockwise curved stroke β3”, and “downward counterclockwise curved stroke β4”.

  FIG. 24 is an example in which subdivision is further performed in consideration of the absolute value of the stroke curvature C. That is, the types α1-1, α1-2, α1-3, and α1-4 shown in the figure all belong to the “rightward clockwise curve stroke α1” shown in FIG. Are different.

  Thus, if the types of individual strokes are subdivided, the analysis patterns defined by these strokes will naturally be diversified. In particular, an analysis pattern for a gesture using multiple fingers is defined by a combination of multiple strokes. Therefore, if the type of each stroke increases, the variation of the analysis pattern defined by these combinations is an exponent. Increase. Therefore, it is possible to detect operation input with higher accuracy.

  However, when the analysis patterns are diversified, there are many cases where a sample pattern that matches a specific operation input pattern is not found, and there are many cases where the pattern selection unit 160 returns an error detection result signal. For example, in the basic embodiment described in §2-2-4, “straight stroke” is classified into the type “straight stroke in the right direction” regardless of whether the stroke direction is B = 10 ° or B = −20 °. However, when segmentation as shown in FIG. 22 is performed, the strokes are classified into different types of linear strokes.

  For example, when a sample pattern of a certain gesture G is collected, the movement of the finger moved from the left to the right is recognized as “a linear stroke with a stroke direction B = 10 °”. Consider a case where it is registered as a “sample pattern”. Subsequently, in order to perform an operation input of the same gesture G, it is assumed that an operation input pattern including “stroke direction B = −20 ° linear stroke” is recognized as a result of performing an operation of moving a finger from left to right. . In this case, in the basic embodiment described in §2-2-4, since both straight strokes are classified into the type of “right straight stroke”, both patterns match and the operation input is a problem. And “gesture G” is detected.

  However, when the subdivision shown in FIG. 22 is performed, the “stroke direction B = 10 ° linear stroke” and the “stroke direction B = −20 ° linear stroke” are different types and therefore match. Sample pattern not found. Therefore, the pattern selection unit 160 cannot select a matching pattern, and is forced to output a detection result signal indicating an error. In fact, as shown in FIG. 22, when the stroke direction B is subdivided in units of 10 ° and the type is determined, a detection result signal indicating an error will be output with a considerable frequency.

  Therefore, when performing operation to subdivide the stroke characteristics, the pattern selection unit 160 includes a stroke included in the operation input pattern when there is no sample pattern that matches the collected operation input pattern in the sample pattern storage unit. An approximate sample pattern including a stroke having a characteristic approximate to the above characteristic may be selected.

  For example, in the case of the above example, when there is no sample pattern including “straight stroke of azimuth B = −20 °”, first, the azimuth B is increased / decreased by 10 ° and “azimuth B = −10 ° or −30”. A sample pattern including “linear stroke” is selected as an approximate sample pattern. When such an approximate sample pattern does not exist, the azimuth B is increased or decreased by 20 ° and “azimuth B = 0 ° or −40”. A sample pattern including “linear stroke” is selected as an approximate sample pattern. When such an approximate sample pattern does not exist, the direction B is increased or decreased by 30 ° and “direction B = 10 ° or −50”. Approximately expanded the approximate range, such as selecting a sample pattern that includes `` ° linear stroke '' as the approximate sample pattern It is sufficient to perform constant. Then, if there is no matching sample pattern, a sample pattern having the highest possible approximation is selected as the approximate sample pattern. Of course, the limit of the approximate range is set to ± 30 °, for example, and if an approximate sample pattern within the range is not found, an error signal may be returned with no approximate sample pattern.

  In this way, when there is no completely matched sample pattern at the time of pattern selection, by selecting an approximate sample pattern, even when a gesture operation slightly different from that at the time of sample pattern collection is performed, Operations that can be detected as valid operation inputs can be performed.

<3-4: Diversification of gesture actions>
In the basic embodiment described above, as described in §2-3, the example in which the sample pattern collection unit 130 performs the process of registering the sample pattern for each of the six gesture operations illustrated in FIG. 11 has been described. Of course, in practice, various other gesture operations can be registered as necessary. Of course, it is also possible to register a gesture using one finger or a gesture using three or more fingers.

  Gestures G11 to G18 shown in FIG. 25 are variations of gestures using two fingers. Gestures G11 to G14 are gestures that move two fingers away from each other, and can be used as, for example, a gesture that indicates an operation input for enlarging and displaying an object. On the other hand, the gestures G15 to G18 are gestures that move two fingers so as to approach each other, and can be used as, for example, a gesture indicating an operation input for reducing and displaying an object. Gestures G11 to G14 and gestures G15 to G18 are variations with different finger movement directions, and different operation inputs can be performed depending on variations in the movement directions.

<3-5: Modification Considering Selection History>
FIG. 26 is a block diagram showing a configuration of an operation input detection device using a touch panel according to still another modified example of the present invention. The difference from the basic embodiment shown in FIG. 1 is that a digital processing unit 100 C is used instead of the digital processing unit 100. A major feature of the digital processing unit 100C is that a history storage unit 190 is added to the digital processing unit 100.

  The history storage unit 190 is a component having a function of storing a history of detection results output in the past by the detection result output unit 170. The detection result output unit 170 has a function of outputting information indicating a gesture corresponding to the sample pattern selected by the pattern selection unit 160C to the outside as a detection result signal indicating the detection result of the operation input. Therefore, the history storage unit 190 performs processing for storing the history every time the detection result output unit 170 outputs a detection result signal.

  Such a history can be used when the pattern selection unit 160C selects an approximate sample pattern as described in the modification example of diversifying the analysis pattern of §3-3. That is, if the sample pattern storage unit 150 does not store a matching sample pattern because the analysis patterns are diversified, the pattern selection unit 160C has characteristics that approximate the characteristics of the stroke included in the operation input pattern. An approximate sample pattern including a stroke can be selected. At this time, referring to the history stored in the history storage unit 190, a sample pattern having a higher past selection frequency is set to a more approximate sample pattern. Judgment can be made.

  For example, in §3-3, when an analysis pattern including “straight stroke with stroke orientation B = 10 °” is registered as “sample pattern of gesture G”, “straight stroke with orientation B = −20 °”. The selection method when the operation input pattern including is given was described. In this case, for example, a linear stroke in which the azimuth B is within ± 30 ° is defined as an approximate stroke with respect to the “linear stroke of the direction B = −20 °”, and among sample patterns including such an approximate stroke, A sample pattern having the highest past selection frequency may be handled as a sample pattern having the highest degree of approximation, and processing for selecting this as an approximate sample pattern may be performed.

  In general, the user tends to repeatedly give the same operation input. Therefore, if the degree of approximation is determined with reference to the past history, a more appropriate judgment can be made.

<3-6: Modified example ignoring short stroke>
In the basic embodiment described so far, as described in §2-2-2, the same is based on a group of representative point data to be analyzed stored in the representative point data storage unit 121 of FIG. A stroke composed of an aggregate of a plurality of representative points indicating the movement of the finger is recognized. However, the stroke recognized by such a method is not necessarily a correct stroke indicating the locus of the user's finger.

  As shown in the graph of FIG. 3, for the detection value whose contact degree is less than the predetermined reference value Th, the representative point Q is not recognized, but some noise component happens to be detected above the reference value Th. When a value is generated or an irrelevant finger such as a little finger accidentally touches the touch surface against the user's intention, an unnecessary representative point Q is recognized, and based on such a representative point Q Unnecessary strokes will be recognized.

  When such a phenomenon occurs, for example, although the user is performing a two-finger gesture, three strokes are recognized, and the collected operation input pattern is treated as a pattern including three strokes. Become. As a result, the pattern selection unit cannot select a sample pattern that matches or approximates the collected operation input pattern.

  Therefore, when the pattern selection unit cannot select a sample pattern that matches or approximates the collected operation input pattern, the stroke length L is the shortest among a plurality of strokes included in the operation input pattern. A corrective operation input pattern in which the stroke is deleted may be created, and a sample pattern that matches the corrective operation input pattern may be selected. A stroke with a short stroke length L is likely to be a useless stroke caused by noise or an erroneous operation by the user. Therefore, by deleting a stroke with a short stroke length L, it can be handled ignoring the useless stroke. Become.

<<< §4. Specific data analysis example >>
Here, an example of measurement of representative point data obtained by actually performing a gesture operation by the user and an example of a pattern obtained by analyzing the representative point data are shown.

  FIG. 27 is a table showing an example of specific representative point data obtained when a gesture of sliding two fingers from top to bottom is performed. This representative point data corresponds to actual data stored in the representative point data storage unit 121 shown in FIG. The data shown in the four columns on the left side of this table are the data of the representative points that constitute the first stroke S1, and the data shown in the four columns on the right side are the data of the representative points that constitute the second stroke S2. . Specifically, the data in the time column indicates time t (unit: ms), and the data in the X, Y, and Amp columns indicate the X coordinate value, Y coordinate value, and intensity value of the representative point, respectively. Show. In this table, the data of the first stroke S1 (left half) and the data of the second stroke S2 (right half) are listed at the same time t. Therefore, the representative point data described in the same row of the table indicates finger contact information at the same time.

  FIG. 28 is a plan view showing an analysis pattern obtained by analyzing the representative point data shown in FIG. As shown in the figure, this analysis pattern is a pattern in which the first stroke S1 from the top to the bottom is arranged on the left side and the second stroke S2 from the top to the bottom is arranged on the right side. As described above, the pattern analysis unit 120 calculates parameter values indicating characteristics for the strokes S1 and S2, and classifies the strokes into several types based on the parameter values. The lower column of the table of FIG. 27 shows specific numerical values of the stroke length L, the stroke orientation B, and the stroke curvature C calculated for the strokes S1 and S2. These numerical values are obtained by performing calculations based on the calculation formula shown in FIG. 8 using the representative point data listed in the table.

  Here, the reference value Lth of the stroke length L is set to Lth = 1, and the reference value Cth of the stroke curvature is set to Cth = 15. Therefore, in the pattern determination process shown in FIG. 9, in step S13, when L <1, it is recognized as a point stroke. If it is recognized that the stroke is a line stroke, in step S15, if | C | <15, the stroke is recognized as a linear stroke. In the example shown in FIG. 27, the strokes S1 and S2 are both recognized as line strokes because L> 1, and are both recognized as linear strokes because | C | <15. In step S19, “downward” is determined based on the value of the azimuth B. In step S20, the type of “downward linear stroke” is determined for any stroke. Thus, finally, in step S23, an analysis pattern (sample pattern SP (G2) in FIG. 14) of “left downward linear stroke + right downward linear stroke” is determined.

  On the other hand, FIG. 29 is a table showing an example of specific representative point data obtained when a gesture of sliding two fingers from left to right is performed, and FIG. 30 shows representative point data shown in FIG. It is a top view which shows the analysis pattern obtained by analyzing. As shown in the figure, this analysis pattern is a pattern in which the first stroke S1 from the left to the right is arranged on the lower side and the second stroke S2 from the left to the right is arranged on the upper side. Again, the lower column of the table of FIG. 29 shows the parameter values calculated for these strokes.

  Also in this example, the strokes S1 and S2 are both recognized as line strokes because L> 1, and are both recognized as linear strokes because | C | <15. In step S19, “right direction” is determined based on the value of the azimuth B. In step S20, the type of “right linear stroke” is determined for any stroke. Thus, finally, in step S23, an analysis pattern (sample pattern SP (G4) in FIG. 14) of “upper right linear stroke + lower right linear stroke” is determined.

  FIG. 31 is a table showing an example of specific representative point data obtained when a gesture is performed in which both fingers rotate clockwise. FIG. 32 shows the representative point data shown in FIG. It is a top view which shows the analysis pattern obtained by analyzing. As shown in the figure, this analysis pattern is a pattern in which the first stroke S1 from the top to the bottom is arranged on the right side and the second stroke S2 from the bottom to the top is arranged on the left side. Again, the lower column of the table of FIG. 31 shows the parameter values calculated for these strokes.

  Also in this example, the strokes S1 and S2 are both recognized as line strokes because L> 1. However, regarding the degree of curvature C, since | C |> 15 in all cases, it is recognized as a curved stroke. In step S17, since the degree of curvature C shows a negative value, the curve direction is determined to be clockwise, and in step S20, a "clockwise curve stroke" is obtained for any stroke. The type is determined. Thus, finally, in step S23, an analysis pattern (sample pattern SP (G5) in FIG. 13) of “left clockwise curve stroke + right clockwise curve stroke” is determined.

  FIG. 33 is a table showing an example of specific representative point data obtained when a gesture is performed in which two fingers are rotated counterclockwise. FIG. 34 is a table showing representative points shown in FIG. It is a top view which shows the analysis pattern obtained by analyzing data. As shown in the figure, this analysis pattern is a pattern in which the first stroke S1 that stays at substantially the same position is arranged on the right side, and the second stroke S2 going from top to bottom is arranged on the left side. Again, the lower column of the table in FIG. 33 shows the parameter values calculated for these strokes.

  In this example, the stroke S2 is recognized as a line stroke because L> 1, but the stroke S1 is recognized as a point stroke because L <1. Further, since the curvature C of the stroke S2 recognized as a line stroke is | C |> 15, it is recognized as a curved stroke, and the curvature C shows a positive value, so the bending direction is counterclockwise. The decision is made. Therefore, in step S20, the type “point stroke” is determined for the stroke S1, and the type “counterclockwise curve stroke” is determined for the stroke S2. Thus, finally, in step S23, an analysis pattern (sample pattern SP (G6) in FIG. 13) “left counterclockwise curved stroke + right point stroke” is determined.

<<< §5. Basic procedure of operation input detection method according to the present invention >>>
Finally, the basic procedure of the operation input detection method according to the present invention will be described. In this method, a touch panel device that detects a contact state of a user's finger with respect to a touch surface and a display device that displays information to be presented to the user are used to detect an operation input to the touch panel device by the computer. Therefore, the operation input detection device according to the present invention described so far is used.

  FIG. 35 is a flowchart showing the procedure of this operation input detection method. As shown in the figure, this procedure includes a preparation stage (steps S31 to S33) for registering a user's sample operation in advance and a detection stage (steps S34 to S38) for receiving and detecting a user's actual operation input. The

  Here, in the preparation stage, the computer displays an instruction to perform a predetermined gesture on the touch surface on the screen of the display device, and based on a detection signal obtained from the touch panel device by a user's gesture operation based on the instruction. The step of recognizing the temporal transition of the representative point indicating the representative position of the contact area of the user's finger (step S31), and the computer changes the temporal change state of the contact state based on the temporal transition of the representative point And the result is collected as a sample pattern corresponding to the gesture (step S32), and each of these steps is executed for a plurality of m types of gestures (step S33), corresponding to each gesture. Processing for collecting the sample pattern to be performed is performed.

  On the other hand, at the detection stage, when the actual operation input needs to be detected, the computer recognizes the temporal transition of the representative point indicating the representative position of the contact area of the user's finger based on the detection signal given from the touch panel device. And a stage in which the computer analyzes the temporal change state of the contact state based on the temporal transition of the representative point and collects the result as an operation input pattern corresponding to the actual operation input (step S34). Step S35), the computer compares the collected operation input pattern with a plurality of m sample patterns collected in the preparation step, and selects a sample pattern that matches or approximates (Step S36). A step of detecting that an operation input corresponding to the selected sample pattern has been made (step S37). If necessary, these steps are repeated (step S38).

  As described in §3-1 as a modification example for specifying a user, in the preparation stage, the computer performs processing for collecting individual and independent sample patterns for a plurality of users, and in the detection stage, the computer If a user in use is specified and pattern selection is performed by comparing the user with a sample pattern, it is possible to detect an accurate operation input that takes into account the features unique to the user.

  In addition, as described in §3-2 as a modification example for specifying an area, in the preparation stage, the computer performs a gesture separately for each of a plurality of areas obtained by dividing the touch surface of the touch panel device. Displays instructions, performs processing to collect sample patterns for each area, and at the detection stage, the computer recognizes the main area where the gesture was performed and compares it with the sample pattern for that recognition area Thus, if the pattern is selected, it is possible to detect an accurate operation input in consideration of the area where the gesture is performed.

10: Touch panel device 20: Display device 30: Display device with touch panel 100, 100A, 100B, 100C: Digital processing unit 110: Representative point recognition unit 120: Pattern analysis unit 121: Representative point data storage unit 122: Stroke recognition unit 123: Characteristic recognition unit 124: pattern determination unit 130, 130A, 130B: sample pattern collection unit 140: operation input pattern exploitation unit 150, 150A, 150B: sample pattern storage unit 160, 160A, 160B, 160C: pattern selection unit 170: detection result Output unit 180: User specifying unit 190: History storage unit 200: External units A1, A2: Contact areas a11 to a29: Intensity value B: Stroke orientation C: Stroke curvature Cth: Stroke curvature reference value d: Distance reference Values d1-d : Distance between two points dx, dy: Grid line pitch E: Envelope surfaces G1 to G18: Gesture H (x, y): Grid point i, j: Parameter indicating number L: Stroke length Lth: Reference value of stroke length n: number of representative points included in one stroke O: origin of coordinate system P: stroke positions PT (A) to PT (E): analysis patterns Q1, Q2,..., Q29: representative point Q _i-2 , Q _i−1 , Q _i : representative points S 1, S 2, S _j : strokes S 11 to S 38: each step SP of the flowchart: sample patterns T 1 to T 4: region
Th: Contact degree reference values t1 to t9: Times U, Ua, Ub, Uc: User identification codes V _i−1 , V _i : Vector W: Direction vector X: Coordinate axes x1, x2,..., X29: Representative X coordinate value x _i-2 , x _i−1 , x _{i of} point: X coordinate value of representative point xp: X coordinate value of stroke position P Y: Coordinate axes y1, y2,..., Y29: Y of representative point Coordinate values y _i-2 , y _i-1 , y _i : Y coordinate value y _k of representative point y coordinate value yp of grid line yp: Y coordinate value α1 to α4 of stroke position P: Curve stroke type α1-1 ~α1-4: the type of the curve stroke β1~β4: type of curve stroke θ, θ _i: azimuth angle φ, φ _i: displacement angle

Claims (31)

A touch panel device that detects a contact state of the user's finger with respect to the touch surface, a display device that displays information to be presented to the user, and a digital processing unit,
The digital processing unit is
Based on a detection signal given from the touch panel device, a representative point recognition unit that recognizes a temporal transition of a representative point indicating a representative position of a contact area of a user's finger,
Based on the recognition result of the representative point recognition unit, a pattern analysis unit that analyzes the temporal change state of the contact state and obtains an analysis pattern;
An instruction to perform a plurality of m gestures on the touch surface is displayed on the screen of the display device, and each analysis pattern obtained by the pattern analysis unit for m gesture operations of the user based on the instruction is displayed. , A sample pattern collection unit for collecting each as a sample pattern,
A sample pattern storage unit that stores an analysis pattern collected when an instruction to perform the k-th (1 ≦ k ≦ m) gesture is displayed as a sample pattern corresponding to the k-th gesture;
An operation input pattern collection unit that collects an analysis pattern obtained by the pattern analysis unit as an operation input pattern after a detection request signal for requesting detection of an operation input from the outside is provided;
A pattern selection unit that compares the sampled operation input pattern with a plurality of m sample patterns stored in the sample pattern storage unit, and selects a sample pattern that matches or approximates;
A detection result output unit that outputs information indicating a gesture corresponding to the sample pattern selected by the pattern selection unit to the outside as a detection result signal indicating a detection result of the operation input;
An operation input detection device using a touch panel. The operation input detection device according to claim 1,
The digital processing unit further includes a user specifying unit that inputs information for specifying a user currently in use and stores the information.
The sample pattern collection unit collects a sample pattern independently for each user specified by the user specifying unit,
The sample pattern storage unit stores the sample pattern independently for each user specified by the user specifying unit,
An operation input detection apparatus using a touch panel, wherein a pattern selection unit performs pattern selection by comparing with a sample pattern for a user specified by the user specifying unit. The operation input detection device according to claim 1 or 2,
The sample pattern collecting unit displays an instruction to perform a gesture separately for each of a plurality of areas obtained by dividing the touch surface of the touch panel device, and collects a sample pattern for each area,
The sample pattern storage unit stores independent and independent sample patterns for each of the individual regions,
An operation input detection device using a touch panel, wherein a pattern selection unit recognizes a main area where a gesture is performed and compares the recognition area with a sample pattern for the recognition area. The operation input detection device according to claim 3,
The touch panel device has a rectangular touch surface,
Sample pattern collecting unit, for the landscape mode that uses the touch surface in a landscape state and the portrait mode that uses the touch surface in a portrait state, respectively, to collect a separate independent sample pattern,
The sample pattern storage unit stores the sample pattern separately and independently for the landscape mode and the portrait mode,
An operation input detection apparatus using a touch panel, wherein a pattern selection unit has a function of recognizing a current mode and performs pattern selection by comparing with a sample pattern for the recognition mode. The operation input detection device according to claim 4,
The touch panel device has a sensor for detecting gravitational acceleration,
An operation input detection device using a touch panel, wherein the pattern selection unit has a function of recognizing a current mode based on an output signal of the sensor. In the operation input detection device according to any one of claims 1 to 5,
The touch panel device outputs a signal indicating the degree of finger contact for each position on the touch surface,
The representative point recognizing unit recognizes the closed region on the touch surface where the degree of contact is greater than or equal to a predetermined reference value as a contact region, and “a position where the degree of contact reaches a peak in the contact region” or “touch” An operation input detection device using a touch panel, characterized in that, in a coordinate system in which a surface is a horizontal plane and a degree of contact is in a height direction, a “centroid position of a mountain formed on a contact area” is recognized as a representative point. In the operation input detection device according to any one of claims 1 to 6,
The representative point recognition unit recognizes the coordinate value (x, y) and the current time t of the representative point in the two-dimensional XY coordinate system defined on the touch surface for each contact area, and the value (x, y, t). The representative point data including is output at a predetermined time interval,
An operation input detection device using a touch panel, wherein a pattern analysis unit obtains an analysis pattern based on the representative point data. The operation input detection device according to claim 7,
The touch panel device outputs a signal indicating the degree of finger contact for each position on the touch surface,
For each contact area, a representative point recognition unit is further formed on the contact area in a coordinate system in which the “contact level peak value in the contact area” or “the touch surface is a horizontal plane and the contact level is in the height direction”. The volume of the mountain is recognized as the strength value a of the contact area, and representative point data Q (x, y, a, t) including the value (x, y, a, t) is output at a predetermined time interval. ,
An operation input detection device using a touch panel, wherein a pattern analysis unit obtains an analysis pattern based on the representative point data Q (x, y, a, t). The operation input detection device according to claim 8,
An operation input detection device using a touch panel, wherein the pattern analysis unit analyzes a pattern based only on representative point data Q (x, y, a, t) having a value a greater than or equal to a predetermined reference value. In the operation input detection device according to any one of claims 7 to 9,
The pattern analyzer
A representative point data storage unit for storing representative point data;
Based on the representative point data, a stroke recognition unit for recognizing a stroke composed of an aggregate of a plurality of representative points indicating the same finger movement;
A characteristic recognition unit for recognizing the characteristics of individual strokes recognized by the stroke recognition unit;
A pattern determining unit that determines an analysis pattern by associating one stroke having a unique characteristic or a combination of a plurality of strokes with one pattern based on the recognition result by the stroke recognizing unit and the characteristic recognizing unit; ,
An operation input detection device using a touch panel characterized by comprising: The operation input detection device according to claim 10,
Stroke identification unit, based on the set of representative point data are temporally continuous output from the representative point recognition unit, a representative point obtained to the i-th time t _i (i-1) th time A touch panel that recognizes a stroke by recognizing two representative points whose distance from the representative point obtained at t _i-1 is equal to or less than a predetermined reference as a representative point belonging to the same stroke. Operation input detection device using The operation input detection device according to claim 10 or 11,
The characteristic recognition unit indicates at least the stroke length L indicating the sum of the lengths of the connecting lines when a plurality of representative points constituting one stroke are connected in order along the time series, and the degree of curvature of the stroke. An operation input detection device using a touch panel, wherein the stroke curvature C is calculated as a value indicating the characteristic of the stroke. The operation input detection device according to claim 12,
The characteristic recognizing unit sets the coordinate value of the i-th representative point Q _i to (x _i , y _i ) and (i−1) -th representative point for a stroke composed of a total of n representative points. The coordinate value of Q _i-1 is (x _i-1 , y _i-1 ), the vector from the representative point Q _{i -1} to the representative point Q _i is V _i , and the (i-2) -th representative point Q _{When a} vector from _i-2 to the representative point Q _i-1 is V _i-1 and an angle formed by the vector V _i with respect to the vector V _i-1 is a displacement angle φ _i ,
L = _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
C = Σ _{i = 3 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 · φ _i
/ Σ _{i = 3 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
The operation input detection device using the touch panel, wherein the stroke length L and the stroke curvature C are calculated by the following calculation. In the operation input detection device according to claim 12 or 13,
The pattern determining unit classifies each stroke into a plurality of types based on the stroke length L and the stroke curvature C, and determines an analysis pattern by associating a specific analysis pattern with a specific type or a combination thereof. The operation input detection device using the touch panel characterized by this. The operation input detection device according to claim 14,
The pattern determining unit classifies strokes having a stroke length L less than a predetermined reference value Lth into a type of “point stroke”, and sets strokes having a stroke length L equal to or greater than the reference value Lth as “line strokes”. Among them, strokes whose absolute value of curvature C is less than a predetermined reference value Cth are classified into types of “straight stroke”, and among “line strokes”, strokes whose absolute value of curvature C is equal to or greater than the reference value Cth. Are classified into the type of “curve stroke”, and the analysis pattern is determined by associating a specific analysis pattern with each type of “point stroke”, “straight stroke”, “curve stroke”, or a combination thereof. An operation input detection device using a touch panel characterized by performing. The operation input detection device according to claim 15,
The stroke curvature C is determined so that the sign differs between when the stroke rotates clockwise and when it rotates counterclockwise,
The pattern determining unit handles the type of “curve stroke” based on the sign of stroke curvature C, and further classifies it into two types of “clockwise curved stroke” and “counterclockwise curved stroke” An operation input detection device using a touch panel, characterized in that determination is made. The operation input detection device according to claim 15 or 16,
An operation input detection device using a touch panel, wherein the characteristic recognition unit further calculates a stroke direction B indicating the direction of the entire stroke on the XY plane as a value indicating the characteristic of the stroke. The operation input detection device according to claim 17,
The characteristic recognizing unit has a predetermined vector V _i from the (i−1) -th representative point Q _i−1 to the i-th representative point Q _i for a stroke composed of a total of n representative points. When the angle formed with respect to the reference coordinate axis is azimuth angle θ _i ,
B = _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 · θ _i
/ _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
The operation input detection device using the touch panel, wherein the stroke direction B is calculated by the following calculation. The operation input detection device according to claim 17 or 18,
The operation using the touch panel, wherein the pattern determination unit handles the type of “straight stroke” based on the stroke direction B, further divided into a plurality of W types having different directions, and performs pattern determination Input detection device. In the operation input detection device according to any one of claims 15 to 19,
An operation input detection device using a touch panel, wherein the characteristic recognition unit further calculates a stroke position P indicating the position of the stroke on the XY plane as a value indicating the characteristic of the stroke. The operation input detection device according to claim 20,
The characteristic recognizing unit sets the coordinate value of the i-th representative point Q _i to (x _i , y _i ) and (i−1) -th representative point for a stroke composed of a total of n representative points. When the coordinate value of Q _i-1 is (x _i-1 , y _i-1 ),
xp = _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 · x _i
/ _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
yp = Σ _{i = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2 · y _i
/ _{Σi = 2 to n} ((x _i −x _i−1 ) ² + (y _i −y _i−1 ) ² ) ^1/2
An operation input detection device using a touch panel, wherein the coordinate value of the stroke position P (xp, yp) is calculated by the following calculation. The operation input detection device according to claim 20 or 21,
The pattern determining unit, for a pattern including a plurality of strokes, grasps the mutual positional relationship by referring to the stroke position P, and determines a pattern by distinguishing patterns having different mutual positional relationships. Operation input detection device using a touch panel. In the operation input detection device according to any one of claims 10 to 22,
The pattern selection unit selects an approximate sample pattern including a stroke having characteristics similar to those of the stroke included in the operation input pattern when there is no sample pattern that matches the collected operation input pattern. An operation input detection device using a featured touch panel. The operation input detection device according to claim 23,
The digital processing unit further includes a history storage unit that stores a history of detection results output in the past by the detection result output unit,
An operation using a touch panel, wherein the pattern selection unit refers to the history when selecting an approximate sample pattern, and determines that a sample pattern having a higher past selection frequency is a more approximate sample pattern. Input detection device. In the operation input detection device according to any one of claims 12 to 22,
A correction operation in which the pattern selection unit deletes the stroke with the shortest stroke length L from the plurality of strokes included in the operation input pattern when there is no sample pattern that matches or approximates the collected operation input pattern An operation input detection device using a touch panel, wherein an input pattern is created and a sample pattern that matches the correction operation input pattern is selected. In the operation input detection device according to any one of claims 1 to 25,
An operation input detection device using a touch panel, wherein the touch panel device and the display device are configured by a display device with a touch panel whose display screen forms a touch surface. The operation input detection device according to any one of claims 1 to 26;
An external unit having a function of receiving a detection request signal from the operation input detection device and receiving a detection result signal from the operation input detection device;
An electronic device comprising:   A program that causes a computer to function as a digital processing unit in the operation input detection device according to any one of claims 1 to 25. A method of detecting an operation input of a user to the touch panel device by a computer connected to a touch panel device that detects a contact state of a user's finger with respect to a touch surface and a display device that displays information to be presented to the user. ,
A preparatory stage for registering a user's sample operation in advance, and a detection stage for receiving and detecting the user's actual operation input,
In the preparation stage,
The computer displays an instruction to perform a predetermined gesture on the touch surface on the screen of the display device, and based on a detection signal obtained from the touch panel device by the user's gesture operation based on the instruction, Recognizing the temporal transition of the representative point indicating the representative position of the finger contact area;
The computer analyzes the temporal change mode of the contact state based on the temporal transition of the representative point, and collects the result as a sample pattern corresponding to the gesture;
Is executed for a plurality of m types of gestures, and a process of collecting sample patterns corresponding to individual gestures is performed.
In the detection step,
When the computer needs to detect an actual operation input, based on a detection signal given from the touch panel device, recognizing the temporal transition of the representative point indicating the representative position of the contact area of the user's finger;
The computer analyzes the temporal change mode of the contact state based on the temporal transition of the representative point, and collects the result as an operation input pattern corresponding to the actual operation input;
The computer compares the collected operation input pattern with a plurality of m sample patterns collected in the preparation step, and selects a sample pattern that matches or approximates;
A computer detecting that an operation input corresponding to the selected sample pattern has been made;
An operation input detection method using a touch panel, wherein: The operation input detection device according to claim 29,
In the preparation stage, the computer performs a process of collecting independent sample patterns for a plurality of users,
An operation input detection method using a touch panel, wherein a computer selects a pattern by specifying a user currently in use at a detection stage and comparing it with a sample pattern for the user. The operation input detection device according to claim 29 or 30,
In the preparatory stage, the computer displays an instruction to perform a gesture separately for each of a plurality of areas obtained by dividing the touch surface of the touch panel device, and collects a sample pattern for each area. Done
An operation input detection method using a touch panel, wherein a computer recognizes a main area where a gesture is performed at a detection stage and performs pattern selection by comparing with a sample pattern for the recognition area.

Images