JP2011135226A - Device and method for recognizing gesture, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique for improving operability in gesture operation. <P>SOLUTION: An operation screen comprising a hierarchical structure is displayed on a display screen. This device for recognizing gesture includes: an input means to input coordinates on a designated display screen; a storage means to store a gesture dictionary including the shapes of gestures and processing contents corresponding to the shapes; a recognition means to recognize, when continuing input coordinates are input by the input means, the processing content corresponding to the input based on the gesture dictionary; a determination means to determine whether the input is performed by a first input method or a second input method; and an execution means to execute a transition process to a screen corresponding to the processing content recognized by the recognition means when the input is determined to be performed by the first input method, and to execute a transition process to a screen of an upper hierarchy relative to the screen corresponding to the processing content recognized by the recognition means when the input is determined to be performed by the second input method. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a gesture recognition device, a recognition method thereof, and a program.

  A technique for calling a function of a device using a gesture is known. Japanese Patent Application Laid-Open No. 2004-151561 discloses a technique for instructing print settings of an image forming apparatus using a gesture. In this apparatus, a preview image for visually expressing the set function is displayed on the operation panel. For example, in order to call a punch hole function, the user inputs two circles by touch operation at a position where a punch hole is desired on the preview image.

  A general apparatus using a hierarchy screen / hierarchy menu has to follow a hierarchy one by one in order to call a desired function. In contrast, an apparatus using a gesture has an advantage that a desired function can be directly called. Therefore, fewer steps are required until a desired function is called than when a hierarchical screen / hierarchy menu is used.

JP 2006-41947 A

  In the technique described in Patent Literature 1 described above, for example, when the corresponding gesture is not known, the user cannot call a desired function. Therefore, it was not easy to use.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide a technique that improves operability during gesture operation.

  In order to solve the above-described problem, an aspect of the present invention is a gesture recognition device that displays an operation screen having a hierarchical structure on a display screen, and an input unit that inputs coordinates on the display screen instructed by an indicator And a storage means for storing a gesture dictionary including the shape of the gesture and the content of the process corresponding to the shape, and when the input coordinates that are continuous are input by the input means, the continuous When the input coordinates and the shape of the gesture are compared, the recognition means for recognizing the content of the processing corresponding to the input, and the continuous input coordinates are input by the input means, the input is based on the first input method. Or a determination unit that determines whether the input is based on the second input method, and the determination unit determines that the input is based on the first input method In this case, a transition process to a screen corresponding to the content of the process recognized by the recognition unit is performed, and when the determination unit determines that the input is based on the second input method, the recognition unit recognizes it. And executing means for executing a transition process to a screen higher than the screen corresponding to the contents of the processing.

  According to the present invention, the operability during gesture operation can be improved.

2 is a diagram illustrating an example of a hardware configuration of the image forming apparatus 10. FIG. The figure which shows an example of the hierarchical structure of an operation screen. The figure which shows an example of an operation screen. 2 is a diagram illustrating an example of a functional configuration realized on the image forming apparatus 10. FIG. The figure which shows an example of a gesture dictionary. 2 is a flowchart illustrating an example of a processing flow of the image forming apparatus 10 illustrated in FIG. 1. 2 is a flowchart illustrating an example of a processing flow of the image forming apparatus 10 illustrated in FIG. 1. 2 is a flowchart illustrating an example of a processing flow of the image forming apparatus 10 illustrated in FIG. 1. The figure explaining an example of the operation | movement at the time of the gesture input corresponding to a 2nd input start. 9 is a flowchart illustrating an example of a processing flow of the image forming apparatus 10 according to the second embodiment. The figure which shows an example of a modification. The figure which shows an example of a modification.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the embodiment described below, a case where the gesture recognition apparatus is applied (built in) to the image forming apparatus will be described as an example. Therefore, the function that the user calls by gesture is print setting. Of course, the gesture recognition apparatus is not limited to the image forming apparatus, and can be applied to other apparatuses. For example, it may be applied to a digital camera.

(Embodiment 1)
FIG. 1 is a diagram illustrating an example of a hardware configuration of an image forming apparatus 10 to which a gesture recognition apparatus according to an embodiment of the present invention is applied.

  Here, the image forming apparatus 10 includes a CPU 11, a ROM 12, a RAM 13, a storage unit 14, a touch panel 15, a display unit 16, and an image forming unit 17.

  A CPU (Central Processing Unit) 11 controls the overall operation of the apparatus. A ROM (Read Only Memory) 12 stores programs and various data. A RAM (Random Access Memory) 13 is used as a work area when the CPU 11 executes a program, and temporarily stores calculation results, data, and the like. The storage unit 14 is realized by, for example, an SD (Secure Digital) memory card, an HDD (Hard Disk Drive), or the like, and stores various data.

  The display unit 16 displays various operation screens. The touch panel 15 functions as a coordinate input device, detects the coordinate position of an indicator (finger, pen, etc.) pointing on the display screen (display unit 16), and inputs the detected coordinates into the device. Here, the display unit 16 and the touch panel 15 are integrally configured. That is, the touch panel 15 is provided so as to overlap the display unit 16. An area where touch input is possible on the touch panel 15 is referred to as an input enabled area. It is desirable that the input enabled area covers the entire display unit 16.

  The image forming unit 17 forms an image on a recording medium such as paper and performs printing. The method of the image forming unit 17 is not particularly limited. For example, an electrophotographic method or an ink jet method may be employed.

  Here, the image forming apparatus 10 has a plurality of operation screens for performing various print settings. The plurality of operation screens are configured to have a hierarchical structure. Hereinafter, an operation for performing desired print settings and the like by sequentially tracing the hierarchically configured operation screens is referred to as a hierarchical screen operation. On the other hand, an operation for jumping to a desired print setting using a gesture and performing the desired print setting or the like is called a gesture operation. Here, a case will be described in which desired print settings and the like are performed by combining gesture operation and hierarchical screen operation.

  Here, an example of a hierarchical structure on the operation screen will be described with reference to FIG. The hierarchical structure of the operation screen is roughly divided into a root node screen, an intermediate node screen, and a leaf node screen. In this case, the copy basic screen corresponds to the root node screen, the application mode screen, the copy-forgery-inhibited pattern printing screen, and the like correspond to the intermediate node screen, and the copy screen and the secret screen correspond to the leaf node screen. Here, in the hierarchical structure of the operation screen, the direction toward the root node is expressed as up, and the direction toward the leaf node is expressed as down. For example, the screen one level higher than the copy screen is a stamp screen, and the screen one level lower than the stamp screen is a copy screen, a confidential screen, a confidential screen, or the like.

  In the hierarchy screen operation, the root node screen becomes the start screen. The user sequentially selects items from the root node screen and follows the hierarchical structure of the operation screen. The leaf node screen that arrives last is a screen on which print settings can be made. For example, when setting a stamp at the time of copying, the hierarchical structure of the operation screen is followed in the order of the copy basic screen, application mode screen, copy-forgery-inhibited pattern printing screen, stamp screen, and copy screen. The gesture input screen is a screen on which gesture input is possible. This is a start screen for gesture operation, and this operation screen does not belong to the above-described hierarchical structure and is independent.

  Here, FIG. 3 shows an example of an operation screen displayed on the display unit 16. 3A corresponds to the gesture input screen shown in FIG. 2, and FIG. 3B corresponds to the stamp screen shown in FIG. On the stamp screen, print settings such as copy, secret, and internal secret can be selected. FIG. 3C corresponds to the confidential screen shown in FIG.

  A paper preview area 41 is provided at the center of the gesture input screen as shown in FIG. The paper preview area 41 represents a print finished image, and serves as a position guide when inputting a gesture. For example, when it is desired to watermark the “confidential” stamp at the center of the paper, the user inputs a gesture corresponding to “confidential” at the central part of the paper preview area 41. Further, for example, when the upper left corner of the paper is to be stapled, the user inputs a gesture corresponding to “upper left staple” in the upper left area in the paper preview area 41.

  Next, an example of a functional configuration realized on the image forming apparatus 10 will be described with reference to FIG. 4 is realized by the CPU 11 executing a program stored in the ROM 12 or the storage unit 14, for example.

  The image forming apparatus 10 includes an input unit 21, a region management unit 22, a determination unit 23, a recognition unit 24, a dictionary information storage unit 25, an execution unit 26, and a screen information storage unit 27. Is done.

  The screen information storage unit 27 is realized by, for example, the ROM 12, the storage unit 14, and the like, and stores a plurality of operation screens. That is, the screen information storage unit 27 stores a screen having a hierarchical structure shown in FIG. 2 and an independent screen (gesture input screen).

  The input unit 21 is realized by, for example, the touch panel 15, the display unit 16, and the like, and inputs a gesture (continuous input coordinates) into the apparatus. Note that the input unit 21 performs not only gesture input but also input processing using buttons displayed on the screen.

  The area management unit 22 is realized by, for example, the CPU 11 or the like, and manages an input possible area on the display screen by dividing the area into a first input area (direct input area) and a second input area (indirect input area). . For example, in the case of the gesture input screen, the inside of the paper preview area 41 is the first input area, and the outside is the second input area.

  The determination unit 23 is realized by, for example, the CPU 11 and determines the input method of the gesture input by the input unit 21. That is, it is determined whether the input gesture is input by the first input method (direct input method) or the second input method (indirect input method). In this determination, it is determined that the gesture input in the first input area is input by the first input method, and the gesture input in the other area (second input area) is determined by the second input method. It is determined that the input is from. Here, the first input method means that a corresponding gesture is directly input. When the user knows a desired print setting gesture, the gesture input is performed by this input method. The second input method means that a gesture having a print setting similar to the desired print setting is input. This is an input method that enables gesture input even when the user does not know the desired print setting gesture. Similar print settings are determined based on the hierarchical structure of the operation screen. For example, referring to FIG. 2, a stamp screen is provided in a layer one level higher than the copy screen, the confidential screen, and the confidential screen. Therefore, here, the functions provided by the copy screen, the confidential screen, and the confidential screen are defined as similar print settings.

  The dictionary information storage unit 25 stores a gesture dictionary. In the gesture dictionary, as shown in FIG. 5, each entry has a shape, a positional relationship with the paper preview area, and print setting items (processing contents). “Shape” is information indicating the shape of a gesture (continuous input coordinates). For example, the shape of the gesture corresponding to “2 in 1” is composed of one horizontal stroke. The “input start by the second input method (hereinafter abbreviated as second input start)” will be described in the second embodiment.

  The “positional relationship with the paper preview area” is information indicating the positional relation between the input gesture and the paper preview area 41 (see FIG. 2). For example, it is stipulated that the gesture corresponding to “staple upper left” needs to be input small in the upper left inside the paper preview area 41. Further, for example, it is stipulated that a gesture corresponding to “confidentiality” needs to be largely input to the central portion of the paper preview area 41. It should be noted that “staple upper left” and “staple lower left” have the same gesture shape, but both are identified using information defined in “positional relationship with paper preview area”.

  The recognition unit 24 recognizes the gesture input by the input unit 21. That is, the input gesture is recognized based on the gesture dictionary. When a gesture is input by the first input method, the gesture is input inside the paper preview area 41. In this case, the recognizing unit 24 performs recognition using both the “positional relationship with the paper preview area” and the “shape”. On the other hand, when a gesture is input by the second input method, the gesture is input outside the paper preview area 41. In this case, since the recognition using the comparison result between the “positional relationship with the paper preview area” and the input gesture cannot be performed, the recognition unit 24 performs the recognition using only the information defined in the “shape”.

  The execution unit 26 executes processing corresponding to the input gesture based on the determination result of the determination unit 23 and the recognition result of the recognition unit 24. Even if the same gesture is input, the execution unit 26 performs different processing depending on the input method (first input method, second input method). When the gesture is input by the first input method, the execution unit 26 causes the display unit 16 to display a screen corresponding to the recognition gesture. For example, when a gesture of “confidentiality” is recognized, a confidential screen is displayed on the display unit 16.

  On the other hand, when a gesture is input by the second input method, a screen that is one level higher than the screen corresponding to the recognition gesture is displayed on the display unit 16. For example, when a gesture of “confidentiality” is recognized, a stamp screen corresponding to a layer one level higher than the confidential screen is displayed on the display unit 16. In this case, the user reaches the desired print setting by using both the hierarchical screen operation and the gesture operation. In this embodiment, it is important how much the user knows similar print settings. However, as the user uses both the hierarchical screen operation and the gesture operation, the user uses similar print settings. The setting will be remembered naturally.

  Next, an example of the flow of processing in the image forming apparatus 10 shown in FIG. 1 will be described with reference to FIGS. Here, the flow of processing when performing gesture input on the gesture input screen will be described first with reference to FIG.

  The user inputs a desired gesture using a pen, a finger, or the like. Then, the image forming apparatus inputs the input gesture into the apparatus at the input unit 21 (S101), and determines the input method of the input gesture at the determination unit 23 (S102). That is, it is determined whether or not the input gesture has been input into the paper preview area 41. This determination is made based on the rectangular area circumscribing the input gesture. Here, for example, an area of a rectangle circumscribing the input gesture is S1, and an area of a region where the rectangle circumscribing the input gesture and the paper preview area 41 overlap is S2. Then, X = S2 ÷ S1 is calculated. For example, if X is 0.9 or more, it is determined that an input gesture has been input inside the paper preview area 41 (first input method). If X is less than 0.9, it is determined that the input gesture has been input outside the paper preview area 41 (second input method).

  As a result of the determination, when it is determined that a gesture has been input by the first input method (YES in S103), the image forming apparatus 10 causes the recognition unit 24 to determine “positional relationship with the sheet preview area” and “shape”. Is used to recognize the input gesture (S104). And in the execution part 26, it changes to the screen corresponding to an input gesture (S105). Thereafter, although details will be described later, processing such as print setting is performed (S109).

  On the other hand, when it is determined that the gesture is input by the second input method as a result of the determination (NO in S103), the image forming apparatus 10 uses the “shape” to recognize the input gesture in the recognition unit 24. Recognize (S106). Then, the execution unit 26 transitions to a screen that is higher than the screen corresponding to the input gesture (one higher in the present embodiment) (S107). Thereafter, although details will be described later, after a transition process to a desired screen or the like is performed (S108), a process such as print setting is performed (S109).

  Next, an example of the flow of processing in S108 shown in FIG. 6 (transition processing to a desired screen) will be described using FIG. A flow of processing from a state in which a screen (intermediate node screen) higher than the screen corresponding to the input gesture is displayed by the processing of S107 will be described. Here, for convenience of explanation, it is assumed that the stamp screen shown in FIG. 3B is displayed as an example of the intermediate node screen. The stamp screen is provided with six setting buttons of copy, secret, confidential, copy prohibited, invalid, and confidential, a setting cancel button, and a return button. When one of the setting buttons is pressed, the screen transitions to a screen corresponding to the button, and the corresponding print setting is performed.

  Here, the user presses one of the buttons arranged on the stamp screen using, for example, a finger or a pen. Then, the image forming apparatus 10 uses the input unit 21 to input the button pressing information into the apparatus (S201). Here, the image forming apparatus 10 determines whether or not the pressed button is a setting button.

  As a result of the determination, if it is a setting button (YES in S202), the image forming apparatus 10 performs a transition process to a screen corresponding to the pressed setting button in the execution unit 26 (S205). For example, when the user presses the copy button, a copy screen is displayed on the display unit 16.

  On the other hand, as a result of the determination in S202, when a button other than the setting button is pressed (NO in S202), the image forming apparatus 10 determines whether the return button is pressed. If the return button is pressed (YES in S203), the image forming apparatus 10 performs a transition process to the original screen (gesture input screen) in the execution unit 26 (S206). Note that returning to the gesture input screen is at the time of a gesture operation, and returning to the screen at the next higher level at the time of the hierarchy screen operation. When the hierarchy screen is operated, for example, when a return button is pressed on the stamp screen, the screen returns to the tint block print screen (see FIG. 2).

  If the cancel button is pressed (NO in S203, YES in S204), the image forming apparatus 10 cancels the process in the execution unit 26 and then returns to the original screen (gesture input screen). A transition process is performed (S206). The return screen is the same as when the return button is pressed. If the button press information in S201 is not any of the above buttons (NO in S204), the image forming apparatus 10 enters a button press wait state and returns to the process of S201 again.

  Next, an example of the flow of processing in S109 shown in FIG. 6 (processing for performing print settings) will be described with reference to FIG. A flow of processing from a state in which a screen (leaf node screen) that can be set for printing by the processing of S105 or S108 is displayed will be described. Here, for convenience of explanation, it is assumed that the confidential screen shown in FIG. 3C is displayed as an example of the leaf node screen. The secret screen is provided with a determination button and a cancel button. In the confidential screen, a “confidential” stamp is displayed on the paper preview in order to represent the print image when the “confidential” print setting is performed.

  Here, the user presses one of the buttons arranged on the confidential screen using, for example, a finger or a pen. Then, the image forming apparatus 10 uses the input unit 21 to input the button pressing information into the apparatus (S301). Here, the image forming apparatus 10 determines the pressed button.

  As a result of the determination, if it is a decision button (YES in S302), the image forming apparatus 10 performs print settings corresponding to the pressed button in the execution unit 26 (S304). In this case, for example, a confidential stamp is set. Thereafter, the original screen (gesture input screen) is displayed on the display unit 16 (S305).

  On the other hand, if the pressed button is a cancel button (NO in S302, YES in S303), the image forming apparatus 10 cancels the execution of the process in the execution unit 26. Thereafter, the original screen (gesture input screen) is displayed on the display unit 16 (S305). If the button press information in S301 is not any of the above buttons (NO in S303), the image forming apparatus 10 enters a button press wait state and returns to the process of S301 again.

  The corresponding print settings may be set when the process of S105 in FIG. 6 is performed. In this case, the print setting is made together with the process of S105, and then the original screen (for example, the gesture input screen) is displayed.

  As described above, according to the first embodiment, it is possible to call a desired function without knowing a corresponding gesture, so that operability at the time of inputting a gesture can be improved. For example, even if the user does not know the gesture corresponding to the copy, the user can perform the print setting if he / she knows the gesture of the print setting similar to the copy such as the secret.

(Embodiment 2)
Next, Embodiment 2 will be described. In the above-described processing, when input is performed by the second input method, the recognition processing may be performed using only “shape”, and thus the recognition rate may be reduced. For example, in the case of the gesture dictionary shown in FIG. 5, the shapes corresponding to “2 in 1” and “staple upper left” are similar in shape. Therefore, when the recognition is performed without using the “positional relationship with the paper preview area”, there is a possibility that these gestures cannot be accurately identified. For example, even if the user inputs a gesture intended for “2 in 1”, it may be erroneously recognized as “staple upper left”. When such a misrecognition occurs, for example, when the screen transitions to a screen that has nothing to do with the hierarchical structure, it is necessary to reach the desired screen by operating the hierarchical screen from that position. Is overburdened.

  Therefore, in the second embodiment, a configuration will be described in which a decrease in the recognition rate during the second input method is avoided with a few procedures. As shown in FIG. 5, the gesture corresponding to the second input start has a rectangular shape representing the paper. That is, when inputting by the second input method, the user inputs the paper shape. The gesture corresponding to the second input start is not a gesture for performing print settings alone, but the user continuously inputs a desired gesture after inputting a shape corresponding to the paper.

  Here, an example of an operation at the time of a gesture input corresponding to the second input start will be described with reference to FIG.

  First, as shown in FIG. 9A, the user inputs a gesture corresponding to the second input start to the second input area. Then, the image forming apparatus 10 recognizes the gesture corresponding to the second input start in the recognition unit 24. Here, as shown in FIG. 9B, the image forming apparatus 10 displays the shape corresponding to the input gesture (hereinafter referred to as the second paper preview area 42) and displays the next gesture. Wait for input.

  Subsequently, as shown in FIG. 9C, the user inputs a gesture corresponding to, for example, 2 in 1 inside the second paper preview area 42. Here, in the image forming apparatus 10, the recognition unit 24 recognizes the input gesture using the second paper preview area 42. Specifically, using the second paper preview area 42, recognition using the “positional relationship with the paper preview area” is performed. Thereby, the fall of a recognition rate can be prevented.

  Note that a gesture corresponding to the second input start may be input to the first input area. In this case, for example, as shown in FIG. 9D, the second paper preview area 42 hides the paper preview area 41. When the input of the gesture corresponding to the second input start is small and the paper preview area 41 cannot be hidden, the second preview area 42 is enlarged and displayed so that the paper preview area 41 is hidden. In this case, the area management unit 22 changes the entire inputable area on the display screen to the second input area. That is, when a gesture corresponding to the second input start is input to the first input area, the image forming apparatus 10 treats all subsequent inputs as inputs by the second input method. In the case of FIG. 9B, when the touch panel 15 is small, the area in which the second paper preview area 42 can be input becomes small and it is difficult to input. However, if the method shown in FIG. Such a problem disappears.

  Next, an example of a processing flow in the image forming apparatus 10 according to the second embodiment will be described with reference to FIG. Here, the flow of processing when performing gesture input on the gesture input screen will be described.

  The user inputs a desired gesture using a pen, a finger, or the like. Then, the image forming apparatus inputs the input gesture into the apparatus at the input unit 21 (S401), and determines whether or not the input gesture is a gesture corresponding to the second input start at the recognition unit 24. judge. In the gesture recognition process here, a simpler process than the gesture recognition performed in S406 described later is performed. Specifically, in S406 and the like, it is necessary to recognize all gestures included in the gesture dictionary, but the gesture recognition performed here is only for the gesture corresponding to the second input start. . That is, the likelihood is obtained for the gesture corresponding to the second input start, and if the likelihood is equal to or greater than a predetermined value, it is determined that the gesture corresponding to the second input start is input.

  As a result of the determination, if the gesture corresponds to the start of the second input (YES in S402), the image forming apparatus 10 displays the second paper preview area 42 corresponding to the shape of the input gesture on the display unit 16. (S403). Thereafter, the process returns to S401 again and waits for a gesture input. When the gesture corresponding to the second input start is input to the first input area, the size of the second paper preview area 42 is changed and displayed in the center of the screen as described above.

  On the other hand, if it is determined in S402 that the input gesture is not a gesture corresponding to the second input start (NO in S402), the image forming apparatus 10 determines that the input gesture is the second paper preview area 42 in the determination unit 23. It is determined whether or not it has been done. This determination is performed based on the result of the arithmetic processing described in S102 of FIG. If the second paper preview area 42 is not displayed, this determination is not performed and the process proceeds to S405 as it is.

  As a result of the determination, when the input gesture is input outside the second paper preview area 42 (NO in S404), the image forming apparatus 10 determines the input method of the input gesture in the determination unit 23 (S405). ). That is, it is determined whether or not the input gesture has been input into the paper preview area 41. Note that the processing of S405 to S412 is the same as the processing of S102 to S109 in FIG. 6 describing the first embodiment, and thus the description thereof is omitted here.

  If it is determined in step S404 that an input gesture has been input inside the second paper preview area 42 (YES in step S404), the image forming apparatus 10 recognizes the input gesture in the recognition unit 24. . Specifically, the input gesture is recognized using “positional relationship with the paper preview area” and “shape” (S413). Thereafter, the image forming apparatus 10 deletes the display of the second paper preview area 42 from the gesture input screen (S414). Then, the execution unit 26 transitions to a screen that is higher than the screen corresponding to the input gesture (one higher in the present embodiment) (S415). Since the subsequent processing is the same as the processing of S108 and S109 in FIG. 6 describing the first embodiment, the description thereof is omitted here.

  As described above, according to the second embodiment, the same effects as those of the first embodiment can be obtained, and the recognition rate at the time of input by the second input method can be improved.

  The above is an example of a typical embodiment of the present invention, but the present invention is not limited to the embodiment described above and shown in the drawings, and can be appropriately modified and implemented without departing from the scope of the present invention. . Hereinafter, some modifications of the present embodiment will be listed.

(Modification 1)
Instead of the second input start gesture described in the second embodiment, for example, an input change button for alternately changing the first input method and the second input method may be provided. In this case, the paper preview area 41 is displayed during the first input method, and the second paper preview area 42 is displayed during the second input method. The paper preview area 41 and the second paper preview area 42 are displayed in an identifiable form. For example, the display of the paper preview area 41 is shown in FIG. 3A, and the display of the second paper preview area 42 is shown in FIG. 9D. Further, as shown in FIG. 11, a paper preview area 41 and a second paper preview area 42 may be displayed simultaneously. In this case, in the image forming apparatus 10, the gesture input inside the paper preview area 41 is determined to be input by the first input method, and the gesture input in the second paper preview area 42 is determined as the second input method. The input is determined to be input.

(Modification 2)
The layout of the operation screen described in the above embodiment may be changed as appropriate. For example, as shown in FIG. 12A, the stamp screen may be provided with a button that associates a selectable print setting with a preview of a corresponding gesture. For example, the copy button displays a preview of a copy gesture.

  The stamp screen illustrated in FIG. 12A is displayed, for example, when the screen transitions to a screen one level higher than the screen corresponding to the input gesture (for example, after the processing of S410 and S415 illustrated in FIG. 10). . That is, it is displayed when an input is made by the second input method. In general, in many cases, the user does not know a gesture corresponding to a desired print setting. Therefore, by displaying such a gesture preview, the user can easily learn the gesture.

  Note that when the screen is changed to the stamp screen by the hierarchy screen operation, the stamp screen shown in FIG. 3B may be displayed. This is because the user can assume that he / she does not want to know the corresponding gesture when performing a hierarchical screen operation, and avoids stress due to extra information.

(Modification 3)
In S201 of FIG. 7, the user selects a desired print setting, and during the transition to the corresponding screen in S205 of FIG. 7, for example, as shown in FIG. A preview of the corresponding gesture may be displayed. Thereby, the user can easily learn a gesture.

(Modification 4)
In the above description, when an input is performed using the second input method, an example in which a screen one level higher than the screen corresponding to the input gesture is displayed has been described. Not limited to. For example, the transition destination screen may be dynamically generated based on the operation history of the image forming apparatus 10. For example, print settings used in association with the print settings corresponding to the input gesture are taken out, and a screen in which these print settings are arranged is dynamically generated.

  For example, consider the case of “confidentiality”. The print setting used with “confidential” is X, and the print setting used with X is Y. In this case, Y is a print setting used instead of “confidentiality”. Specifically, in the operation history, there are many combinations of print settings using “2 in 1” and “confidential” in association with each other and combinations of print settings using “2 in 1” and “copy” in association with each other. To do. In this case, “copy” is the print setting used in connection with “confidentiality”. As a result, for example, if the print setting used in connection with confidentiality is copy or confidential, the image forming apparatus 10 generates an operation screen including “confidential”, “copy”, and “confidential”. , Transition to the operation screen.

(Modification 5)
You may make it determine a 1st input system and a 2nd input system using the input speed at the time of gesture input. The determination is made by comparing the input speed with a predetermined value (for example, an average value). For example, if the input speed of the gesture is lower than a predetermined value, it is determined that the input is based on the first input method, and if it is equal to or higher than the predetermined value, it is determined that the input is based on the second input method. In this case, the image forming apparatus 10 is provided with an input speed calculation unit that calculates the input speed instead of the area management unit 22.

  Moreover, you may make it determine a 1st input system and a 2nd input system using the pressing force on the display screen at the time of gesture input. Determination is made by comparing the pressing force with a predetermined value (average value). For example, if the gesture pressing force is equal to or greater than a predetermined value, it is determined that the input is based on the first input method, and if it is smaller than the predetermined value, it is determined that the input is based on the second input method. In this case, the image forming apparatus 10 is provided with a pressing force calculation unit that calculates the pressing force, instead of the area management unit 22.

  The above is the description of the modified example. It should be noted that the present invention can also take the form of, for example, a system, apparatus, method, program, or storage medium. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (12)

A gesture recognition device for displaying an operation screen having a hierarchical structure on a display screen,
An input means for inputting coordinates on the display screen instructed by the indicator;
Storage means for storing a gesture dictionary including the shape of the gesture and the content of the processing corresponding to the shape;
Recognizing means for comparing the continuous input coordinates with the shape of the gesture based on the gesture dictionary and recognizing the content of the processing corresponding to the input when continuous input coordinates are input by the input means; ,
A determination unit that determines whether the input is performed by the first input method or the second input method when continuous input coordinates are input by the input unit;
When the determination unit determines that the input is based on the first input method, a transition process to a screen corresponding to the content of the process recognized by the recognition unit is performed, and the determination unit performs the second process. And executing means for executing a transition process to a screen higher than the screen corresponding to the content of the process recognized by the recognition means when it is determined that the input is based on the input method. Gesture recognition device. On the upper layer screen,
A button for instructing transition to a screen corresponding to the content of the recognized process is provided,
The buttons include
The gesture recognition apparatus according to claim 1, wherein a preview of a shape of a gesture used when transitioning to a screen corresponding to the content of the recognized process by input using the first input method is displayed. On the upper layer screen,
A button for instructing transition to a screen corresponding to the content of the recognized process is provided,
On the display screen,
While transitioning to a screen corresponding to the content of the recognized process as the button is pressed, a preview of the shape of the gesture used when transitioning to the transition destination screen by input using the first input method The gesture recognition apparatus according to claim 1, wherein: is displayed. Further comprising area management means for managing the display screen separately into a first input area and a second input area;
The determination means includes
When the continuous input coordinates are input to the first input area, it is determined that the input is based on the first input method, and when the continuous input coordinates are input to the second input area, The gesture recognition apparatus according to claim 1, wherein the gesture recognition apparatus determines that the input is based on the second input method. The input means includes
Enter the coordinates indicated on the operation screen including the paper preview area representing the finished image of the print,
The area management means includes:
Managing the inside of the paper preview area as a first input area, managing the outside of the paper preview area as a second input area,
The gesture dictionary
Define print settings as the content of the process,
The execution means includes
When the determination unit determines that the input is based on the first input method, a transition process to a screen capable of selecting the print setting recognized by the recognition unit is performed, and the determination unit performs the second process. The gesture according to claim 4, wherein when it is determined that the input is based on an input method, a transition process to a screen of a higher hierarchy than a screen capable of selecting the print setting recognized by the recognition unit is performed. Recognition device. In the gesture dictionary,
The shape of the gesture, the positional relationship between the shape and the paper preview area, and the print settings corresponding to the shape are defined in association with each other,
The recognition means is
When the continuous input coordinates are input inside the paper preview area, based on the comparison results of the continuous input coordinates and the shape of the gesture and the positional relationship between the continuous input coordinates and the paper preview area, The gesture recognition apparatus according to claim 5, wherein a print setting corresponding to the input is recognized. The gesture dictionary
Including a shape of a gesture for instructing start of input by the second input method,
The determination means includes
When the recognition means recognizes that the continuous input coordinates are the shape of a gesture instructing the start of input by the second input method, the continuous input coordinates input next are input to the second input The gesture recognition device according to claim 1, wherein the gesture recognition device determines that the input is based on a method. The shape of the gesture for instructing the start of input by the second input method is a shape of paper,
In the gesture dictionary,
The gesture shape, the positional relationship between the shape of the gesture and the area where the shape of the paper is input, and the print setting corresponding to the shape of the gesture are defined in association with each other,
The recognition means is
Based on the comparison result of the next input coordinate and the shape of the gesture that are input next to the shape of the paper, and the positional relationship between the input region of the continuous input coordinate that is input next and the shape of the paper The gesture recognition apparatus according to claim 7, wherein a print setting corresponding to the input is recognized. An input speed calculating means for calculating an input speed when continuous input coordinates are input by the input means;
The determination means includes
Determining whether the input is based on the first input method or the second input method based on a comparison between the input speed calculated by the input speed calculation means and a predetermined value. The gesture recognition apparatus according to claim 1. A pressing force calculating means for calculating a pressing force on the display screen when continuous input coordinates are input by the input means;
The determination means includes
Determining whether the input is based on the first input method or the second input method based on a comparison between the pressing force calculated by the pressing force calculating means and a predetermined value. The gesture recognition device according to claim 1, wherein: A gesture recognition device recognition method for displaying a hierarchical operation screen on a display screen,
An input means for inputting coordinates on the display screen indicated by the indicator;
Storing a gesture dictionary including the shape of the gesture and the content of the process corresponding to the shape;
When continuous input coordinates are input by the input unit, the recognition unit compares the continuous input coordinates with the shape of the gesture based on the gesture dictionary, and recognizes the contents of processing corresponding to the input. And a process of
A step of determining whether a determination unit is an input by a first input method or an input by a second input method when continuous input coordinates are input by the input unit;
When the execution unit determines that the input by the first input method is performed by the determination unit, a transition process to a screen corresponding to the content of the process recognized by the recognition unit is performed, and the determination unit A step of performing a transition process to a screen higher than the screen corresponding to the content of the process recognized by the recognition unit when it is determined that the input is based on the second input method. A recognition method for a gesture recognition device. A computer built in a gesture recognition device that displays a hierarchical operation screen on a display screen,
An input means for inputting coordinates on the display screen instructed by the indicator;
Storage means for storing a gesture dictionary including the shape of a gesture and the content of processing corresponding to the shape;
When continuous input coordinates are input by the input means, a recognition means for comparing the continuous input coordinates with the shape of the gesture based on the gesture dictionary and recognizing the content of processing corresponding to the input;
A determination unit that determines whether the input is performed by the first input method or the second input method when continuous input coordinates are input by the input unit;
When the determination unit determines that the input is based on the first input method, a transition process to a screen corresponding to the content of the process recognized by the recognition unit is performed, and the determination unit performs the second process. A program for functioning as an execution means for executing a transition process to a screen higher than the screen corresponding to the content of the process recognized by the recognition means when it is determined that the input is based on the input method.

Images