JP2015035103A - Information processing device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To discriminate and instruct a plurality of different operations with the same gesture for an information processing device.SOLUTION: A motion sensor 16 acquires a pattern of a gesture which a finger tip 100 of a user performs and its performance position. An information processing device recognizes a gesture of dragging the same finger tip 100 as "pen tool" operation if it is performed in a range closest to a screen 15 of a display device 14, recognizes as "palm tool" operation if it is performed in an intermediate range, and recognizes as "page feeding" operation if it is performed in a range farthest from the screen 15.

Description

  The present invention relates to an information processing apparatus and a program.

  There is a user interface system that controls processing executed by a program according to a user's gesture. For example, Leap Motion Controller from Leap Motion, Kinect (registered trademark) from Microsoft, etc. are examples.

  In Japanese Patent Laid-Open No. 2004-260, a hand movement input device for inputting information by inputting a locus of a hand inputs a hand motion with a CCD camera or the like, detects a position coordinate of the hand from the input result, and detects a hand from the input result. A method is disclosed in which the shape and motion of the detection means are discriminated and the detection input position coordinates of the detection means are input by an input mode corresponding to the discrimination result.

  The apparatus disclosed in Patent Literature 2 detects a movement of a recognition target object in a recognition area set as a camera photographing area, for example, using a human hand as a recognition target object, and determines a movement category based on the movement. Then, a command associated with the determined motion category is selected and input to an information processing apparatus such as a PC to execute data processing corresponding to the command. In this apparatus, a plurality of recognition areas can be set in a two-dimensional imaging area, and a command is identified from a combination of movements of recognition target objects in each recognition area. For example, if the hand passes through the first recognition area from bottom to top, the second recognition area from left to right, and the third recognition area from top to bottom, For example, it is recognized that the movement of the hand indicates a command to turn the page and advance to the next page.

  In the technique disclosed in Patent Document 3, a system for controlling a device includes a gesture recognition unit that recognizes a gesture, an attribute recognition unit that recognizes an attribute of a background object of the recognized gesture, a recognized gesture, and its background. Command sending means for generating a control command based on the combination with the target attribute and sending it to the device. In this apparatus, the same gesture can be associated with another control by changing the background.

JP-A-8-211979 JP 2007-34525 A JP 2010-204730 A

  An object of the present invention is to enable an information processing apparatus to distinguish and instruct a plurality of different operations using the same gesture.

  The invention according to claim 1 is a gesture performed by a user from a storage unit that stores an operation corresponding to a combination of a gesture pattern and a position, and a gesture recognition device that recognizes a gesture and a three-dimensional position where the gesture is performed. An acquisition unit that acquires information on the position where the pattern and the gesture are performed, and an operation corresponding to the combination of the pattern and position of the gesture acquired by the acquisition unit are obtained from the storage unit, and the obtained operation is executed. And an information processing apparatus having control means for performing control.

  According to a second aspect of the present invention, the storage means stores an operation corresponding to a combination of a gesture pattern and a position range, and the control means obtains a gesture start position from the gesture recognition device. The information processing apparatus according to claim 1, wherein the range to which the start position belongs is extended.

  The invention according to claim 3 is characterized in that the control means performs control so that information indicating the obtained type of operation is displayed on a display screen of the information processing apparatus. The information processing apparatus described.

  The invention according to claim 4 is the information processing apparatus according to any one of claims 1 to 3, wherein the position is a distance from a display screen of the information processing apparatus.

  The invention according to claim 5 is characterized in that the storage means stores a relationship for increasing the operation amount for the same type of operation as the distance increases. The information processing apparatus according to claim 1.

  According to a sixth aspect of the present invention, a user performs a computer from a storage unit that stores an operation corresponding to a combination of a gesture pattern and a position, a gesture, and a gesture recognition device that recognizes a three-dimensional position where the gesture is performed. An acquisition unit that acquires information on a gesture pattern and a position where the gesture is performed, an operation corresponding to a combination of the gesture pattern and the position acquired by the acquisition unit is obtained from the storage unit, and the obtained operation is executed. It is a program for functioning as control means for performing control.

  According to the first and sixth aspects of the invention, it is possible to distinguish and instruct a plurality of different operations with the same gesture to the information processing apparatus.

  According to the second aspect of the present invention, even if the position of a series of gestures deviates from the initial range to some extent, it can be recognized that the initially recognized operation is continuing.

  According to the third aspect of the present invention, the user can recognize the operation instructed by the gesture.

  According to the invention which concerns on Claim 4, it can distinguish and instruct | indicate the operation relevant to a screen display by the easy-to-understand standard of perspective with respect to a screen.

  According to the fifth aspect of the present invention, it is possible to provide an operation instruction in accordance with an intuitively easy-to-understand standard that a gesture at a position farther from the screen represents a rough operation.

It is a figure for demonstrating the concept of embodiment. It is a functional block diagram of the apparatus of an example of an embodiment. It is a figure which shows an example of the definition information of the operation instruction hold | maintained at a gesture pattern management part. It is a figure which shows the example of the position range divided by the distance from a screen. It is a figure which shows an example of the process sequence of embodiment. It is a figure which shows another example of the definition information of an operation instruction. It is a figure which shows an example of the relationship between the distance from a screen, and the number of pages turned by a gesture. It is a figure which shows another example of the relationship between the distance from a screen, and the number of pages turned by gesture. It is a figure which shows the example of the display image for every range of the distance in which the gesture was performed. It is a figure which shows another example of the definition information of an operation instruction. It is a figure which shows the example of the icon display which shows the operation instruct | indicated by gesture. It is a figure which shows the process sequence of the example which expands the position range corresponding to the recognized operation.

  With reference to FIG. 1, the concept of the gesture input in this embodiment is demonstrated. As shown in FIG. 1, in the present embodiment, the meaning represented by a gesture is switched according to the position of the gesture performed by the user with the fingertip 100 of the hand or the like in the distance direction from the screen of the display device 14. In the example of FIG. 1, the information processing device (not shown) connected to the display device 14 performs a gesture of moving the fingertip 100 when the position of the gesture is within the first range closest to the screen 15. This is interpreted as a “pen tool” operation. Further, when the position of the gesture is within the second range farther from the screen 15 than the first range, it is interpreted as a “hand tool” operation, and the third farthest from the screen than the second range. If it is within the range, it is interpreted as a “page feed” operation. The illustrated display images 15a, 15b, and 15c are examples of images displayed on the screen 15 when each operation is performed.

  Here, the “pen tool” operation is an operation of drawing a curve corresponding to the movement locus of the fingertip 100 on the document represented by the display image 15a. The “hand tool” operation is an operation for scrolling the screen 15. That is, when the fingertip 100 is moved almost in parallel with the screen 15 within the second range corresponding to the “hand tool”, the portion of the document displayed as the display image 15b on the screen 15 is the direction in which the fingertip 100 is moved. Scroll to The “page advance” operation is an operation for moving the page displayed on the screen 15 forward or forward. For example, when the fingertip 100 is moved to the left side substantially parallel to the screen 15 within the third range, the page displayed as the display image 15c on the screen 15 returns to the smaller page number, and is displayed when moved to the right side. Page advances in the direction of increasing page number.

  The motion sensor 16 detects the gesture. As the motion sensor 16, a conventional sensor such as a Leap Motion Controller or Kinect (registered trademark) may be used.

  As described above, in the present embodiment, the same gesture (for example, a hand movement that moves the fingertip 100 substantially parallel to the screen 15) is performed in accordance with the position at which the gesture is performed (distance from the screen 15 in the illustrated example). , To be interpreted as different meanings (ie operations).

  Next, an example of a functional configuration of the information processing apparatus according to the present embodiment will be described with reference to FIG. As shown in FIG. 2, the information processing apparatus according to the present embodiment includes an OS / application 10, a display driver 12, a display device 14, a motion sensor 16, a sensor driver 18, a gesture encoding unit 20, a gesture pattern control unit 22, and a gesture pattern. A management unit 24 is included. Of these, the display device 14 and the motion sensor 16 are hardware, and the others are typically software. The display device 14 and the motion sensor 16 may be built in the housing of the information processing device or may be a device externally attached to the information processing device.

  The OS / application 10 is an operating system (OS) that is basic software of the information processing apparatus or an application program executed on the OS. In relation to the present embodiment, the OS / application 10 is an input destination of an operation instruction input by an instruction input system using a gesture described below. That is, the operation instruction input by the gesture is input to the OS or any application program. The input destination of the operation instruction is an active state (that is, a state selected by the user as an operation input target) at the time of the input of the OS and the application program.

  The display driver 12 is a device driver that controls the display operation of the display device 14. The display device 14 is a device such as a liquid crystal display or an organic EL display that displays a display image generated by the OS / application 10.

  The motion sensor 16 is a device that detects a user gesture for inputting an instruction to the information processing device. The user's gesture is performed using the user's hand or finger, or an instruction bar held by the user. Hands, fingers, pointers, and the like that the user moves to input instructions are hereinafter referred to as “instruction units”. The motion sensor 16 is a distance image sensor using, for example, a triangulation method, a TOF (Time Of Flight) method, or the like, and obtains a distance image in the field of view including an instruction unit to be moved by a user at predetermined time intervals. Output. The distance image includes information on the three-dimensional position of each point on the surface of each object within the measurement target range. Particularly in connection with this embodiment, information on the three-dimensional position of each point on the surface of the pointing unit is obtained from the information on the distance image.

  The motion sensor 16 may output a feature amount obtained by analyzing the distance image instead of outputting the distance image. For example, by analyzing the distance image, the tip of the fingertip or the pointing rod may be extracted, and the extracted three-dimensional position of the tip may be output as a feature amount every moment. When a plurality of fingertips are detected, a set of fingertip positions may be output as a feature amount. Further, a plane that approximates the surface of the hand may be obtained, and a feature amount representing the plane may be output.

  The sensor driver 18 is a device driver that controls the operation of the motion sensor 16. The sensor driver 18 receives a detection signal (distance image or a feature amount obtained from the distance image) output from the motion sensor 16 and passes it to the gesture encoding unit 20.

  The motion sensor 16 is not limited to the one that generates a distance image, and may be of any type as long as it can detect the position and movement of the user's instruction unit (hand or pointer).

  The gesture encoding unit 20 sends an operation instruction to the sensor driver 18 and obtains a gesture pattern and a gesture execution position from a signal input from the motion sensor 16 via the sensor driver 18 according to the operation instruction. A signal input from the motion sensor 16 is a time series of raw detection information (for example, a distance image) of the motion sensor 16 or a feature amount (for example, a position coordinate of each fingertip of the hand) obtained by processing the detection information. Series.

  Gesture patterns (types) include, for example, “pinch-in” that brings the fingertips of multiple fingers closer together, “pinch-out” that spreads the fingertips, “swipe” that moves in one direction to sweep hands and fingers, etc. One hand (such as an index finger), a "grab" action, a "grasping" hand spread (ie, a "grabbed" object) "release" action, an open hand "twist" action, an index finger, etc. Alternatively, various operations such as “pointing” with several aligned fingers) can be considered. In addition, the state of the hand such as whether the finger of the hand is open, whether the hand is in a clenched fist, or in the state where only one finger is extended like a pointing finger, is determined from the distance image information of the motion sensor 16. It is also possible to discriminate, and even an operation of moving the same hand may be distinguished from a different gesture pattern depending on the state of the hand at that time. It is also possible to define a combination of a plurality of basic operations as one gesture, such as defining a “drag” operation to move a hand in a “grabbed” state after a “grab” operation. The gesture pattern recognition process performed by the gesture encoding unit 20 may be the same as that used in the prior art such as the Leap Motion Controller.

  The execution position of the gesture is a position (a three-dimensional position in this example) where the gesture is performed. Since the input information from the motion sensor 16 includes position information of an instruction unit (such as a fingertip), the gesture encoding unit 20 may recognize the position of the gesture from the position information. In one example, the position of the tip of the instruction unit is recognized as the gesture position. For example, in the case of a “pointing” gesture, the position information of the tip of the finger input from the motion sensor 16 may be recognized as the execution position of the gesture. In addition, in the case of an instruction unit having a somewhat large volume, such as a clenched fist, a representative position such as the center of gravity or the tip (tip along the direction in which the upper arm extends) may be used as the gesture position by the instruction unit. When there are a plurality of tips of the instruction unit (for example, when the tips of a plurality of fingers are detected by the motion sensor 16), for example, the position of the center of gravity of the positions of the plurality of tips may be used as the gesture position. A set of tip positions (for example, a set of tip positions of each finger) may be used as the gesture position.

  Gestures are movements and have a time range. The execution position of the gesture is obtained based on the position of the instruction unit at one time point or a plurality of time points in the time width.

  In one example, the start position of the gesture is recognized as the execution position of the gesture. For example, if the pointing unit (such as a hand) starts to move suddenly (with some large acceleration) from a state where it is almost stopped (or a state where it moves at a low speed that is not recognized as a gesture), The gesture start time may be set, and the position of the instruction unit at that time may be set as the gesture start position.

  In another example, the representative position is different from the start position, such as the average position of the pointing unit at each time point constituting the recognized gesture, or the position of the pointing unit at the central point in time of the gesture. The position may be the execution position of the gesture.

  In addition, all the positions of the instruction unit at each time point may be regarded as gesture execution positions. For example, in the case of an operation for drawing a line drawing, the position of the fingertip 100 at each time point corresponds to the position of the pen tip at that time point in the “drawing line drawing” gesture.

  The gesture pattern control unit 22 sends an operation instruction to the gesture encoding unit 20 and also receives information on a gesture pattern and an execution position input from the gesture encoding unit 20, and receives a user operation instruction from the combination of the pattern and the execution position. Determine. In this determination, the operation instruction definition information managed by the gesture pattern management unit 24 is referred to.

  FIG. 3 shows an example of operation instruction definition information managed by the gesture pattern management unit 24. Each row of the table shown in FIG. 3 is one piece of operation instruction definition information. Each row has items of “ID”, “gesture”, “application”, and “corresponding operation”. “ID” is identification information of an entry (row) of the definition information. “Corresponding operation” is identification information of an operation instruction corresponding to the ID (an operation instructed by a user by a gesture, in other words, “meaning” of the gesture). The meaning of the operation of the “pen tool” shown in the figure has already been described.

  “Position” represents the position range of the gesture corresponding to the operation instruction. In other words, in order for the user's gesture to be recognized as indicating the operation instruction, the execution position of the gesture needs to be within the range indicated by the “position”. In the illustrated example, the position range is represented by orthogonal three-dimensional coordinates of X, Y, and Z. Here, the Z coordinate is a coordinate in the Z-axis direction perpendicular to the screen 15 of the display device 14, the point on the screen 15 is Z = 0, and the front direction of the screen 15 is the positive direction of the Z-axis. is there. The X and Y coordinates are coordinate axes extending in the direction of two orthogonal sides of the rectangular shape of the screen 15.

  “Gesture” represents a pattern (type) of a gesture corresponding to the operation instruction. In addition, the gesture “drag” illustrated in the figure is an operation of moving the instruction unit from the start point specified by tapping (an operation to move the instruction unit such as a fingertip). Similar to the mouse drag operation, For example, this represents an operation of moving the cursor (according to the movement locus of the fingertip) from the start point specified on the screen 15.

  “Application” is identification information of an application program or OS that is the target of the operation instruction. The “document viewer” illustrated in the figure is an application used for browsing or editing an electronic document in any document format such as PDF (Portable Document File). Different applications can be interpreted to indicate different operations with the same gesture.

  In the example of FIG. 3, the same type of gesture “drag” for the same application “document viewer” is classified into three cases (ID = 1, 2, 3) having different position ranges. The position ranges of these cases are the same range in the X and Y directions (that is, the rectangular range of the screen 15), but are different in the Z direction. The position ranges of these cases are shown in FIG. As shown in FIG. 4, the position range corresponding to ID = 1 is a rectangular parallelepiped region having a range of 0 to 150 mm in the Z direction shown in (1) of FIG. In addition, the position range corresponding to ID = 2 is a rectangular parallelepiped region in the range of 150 to 300 mm in the Z direction shown in (2), and the position range corresponding to ID = 3 is in the Z direction shown in (3). This is a rectangular parallelepiped region in the range of 300 to 450 mm.

  When the definition information illustrated in FIG. 3 is used, the same gesture “drag” is interpreted as a “pen tool” in the range closest to the screen 15 (ID = 1), and in the range one step further than this, the “open palm tool”. It is interpreted as “page feed” in the far range.

  Here, when comparing the “Pen tool”, “Tenoko tool”, and “Page feed”, it is sufficient to indicate whether to return or advance the page in “Page feed”, whereas in the “Tenoko tool” the display image is displayed. It is necessary to indicate in which direction and how much to scroll. Therefore, a more precise gesture is required in order to convey the user's operation intention in the case of the “hand tool” than in the case of “page feed”. In addition, in order to draw a line drawing intended by the user with the “pen tool”, a more precise gesture is required than the screen scrolling with the “tent tool”.

  In the definition information illustrated in FIG. 3, it can be seen that an operation requiring a more precise gesture sets the position range of the gesture corresponding to the operation to a range closer to the screen 15. This setting is easy to understand intuitively for the user.

  3, the X coordinate of 0 to 500 mm is the range of the horizontal width of the screen 15, and the Y coordinate of 0 to 400 mm is the range of the vertical height of the screen 15. In the example of FIG. 3 (and FIG. 4), the width of each position range in the X and Y directions is the same, but this is only an example. Instead, for example, the width of the position range in the X and Y directions may be increased as the distance from the screen 15 increases.

  Although only one application is shown in FIG. 3, there may be a case where a plurality of applications to which an operation input by a gesture can be applied are installed in the information processing apparatus. In such a case, even if the combination of the gesture pattern and the position range is the same, the “corresponding operation” may be different if the application to be applied is different.

  The gesture pattern control unit 22 specifies an operation instruction (“corresponding operation”) corresponding to the combination of the gesture pattern input from the gesture encoding unit 20 and the execution position with reference to the definition information illustrated in FIG. To do. Further, when there are a plurality of applications to which gesture input is applied, an operation instruction corresponding to a combination of an application currently active in the information processing apparatus, a gesture pattern input from the gesture encoding unit 20 and an execution position, It is specified from the definition information as exemplified in FIG. When the OS is in an active state (in other words, when all activated applications are not in an active state), an operation instruction corresponding to the combination of the gesture pattern, the execution position, and the OS is displayed from the definition information. It is specified as the meaning indicated by the gesture.

  Then, the gesture pattern control unit 22 inputs the meaning of the gesture thus identified, that is, an operation instruction, to the currently active application or OS (OS / application 10). Upon receiving this input, the application or OS executes processing according to the operation instruction.

  The definition information managed by the gesture pattern management unit 24 can be customized. For example, deleting an existing operation instruction definition entry, changing any item in the existing operation instruction definition entry, position range, gesture type (pattern), application, and operation instruction ( A new definition entry consisting of a set of “operations”) can be created and added to the definition information.

  FIG. 5 shows an example of a processing procedure performed by the gesture encoding unit 20, the gesture pattern control unit 22, and the gesture pattern management unit 24 of the present embodiment. In this procedure, first, the gesture encoding unit 20 encodes (analyzes) the detection signal of the motion sensor 16 input from the sensor driver 18 (S10), and the detection signal is obtained from the information on the feature amount as a result of the encoding. The pattern and the execution position of the gesture to be shown are extracted (S12). Next, the gesture pattern control unit 22 outputs an operation instruction corresponding to the combination of the application or OS currently in the active state, the extracted gesture pattern and the execution position, and the definition information in the gesture pattern management unit 24. It specifies with reference (S14). If no operation corresponding to the gesture is found in S14 (the determination result in S16 is N), the gesture is invalidated in the illustrated example, and the process returns to S10 to recognize the next gesture. If an operation instruction corresponding to the gesture is found in S14 (the determination result in S16 is Y), the operation instruction is input to the application or OS (S18). In response to this input, the application or OS executes an operation instructed by the user through the gesture.

  Now, the three-dimensional position of the pointing unit (such as a hand) represented by the detection signal of the motion sensor 16 is the position of the motion sensor 16 in the coordinate system. On the other hand, the “position” in the definition information of the operation instruction illustrated in FIG. 3 is a position in a coordinate system based on the screen 15 of the display device 14 connected to the information processing apparatus. Therefore, in order to realize the processing of the present embodiment, it is necessary to coordinate-convert the position obtained from the detection signal of the motion sensor 16 into a position in a coordinate system with the screen 15 as a reference.

  Here, if the positional relationship between the screen 15 and the motion sensor 16 is always constant, the coordinate transformation is fixed. In this case, information on the fixed coordinate conversion may be obtained in advance, and the gesture encoding unit 20 or the like may perform coordinate conversion using the information. A device with a built-in motion sensor 16 in a housing with a built-in screen, for example, a device having a built-in motion sensor 16 for a tablet terminal, or a digital multifunction device (printer, scanner, copier, facsimile machine, etc.) An example of this is a display having a motion sensor 16 built in a display of a multi-function device.

  On the other hand, the positional relationship between the screen 15 and the motion sensor 16 may be variable. For example, a typical example is a case where the motion sensor 16 is connected to the information processing apparatus by a cable or wirelessly. Even when the information processing apparatus is a notebook PC (personal computer), when the motion sensor 16 is built in the main body (a casing provided with a keyboard), the opening angle of the display is variable. The positional relationship with the motion sensor 16 is variable.

  As described above, when the positional relationship between the screen 15 and the motion sensor 16 is variable, for example, when the motion sensor 16 is activated, a calibration process for obtaining the coordinate conversion is performed. In this calibration process, the user taps the four corners of the screen 15 (the operation of tapping with a fingertip), and the tapped positions are obtained by the motion sensor 16. The positions of the four corner points obtained at this time are positions in the coordinate system of the motion sensor 16. For example, one of the four corner points is the origin of the coordinate system based on the screen 15, and two of the other three points closer to the origin are in the X-axis and Y-axis directions. Is specified. The direction perpendicular to the plane (screen 15) defined by the four corner points is the Z-axis direction. From this relationship, coordinate conversion from the coordinate system based on the motion sensor 16 to the coordinate system based on the screen 15 may be calculated.

  When performing the calibration process, for example, the gesture pattern control unit 22 may display a message to tap the four corners of the screen on the screen 15 of the information processing apparatus.

  Next, a first modification will be described. This modification is different from the above-described embodiment in the definition information of the operation instruction. An example of the operation instruction definition information used in the first modification is shown in FIG.

  As can be seen in comparison with FIG. 3, the definition information in FIG. 6 assumes a plurality of applications.

  In this example, the “send fingertip to the right” gesture for the “document viewer” is associated with the “page break” (page feed in the direction of increasing page numbers) operation. It is subdivided into two according to the position of the gesture. That is, when the gesture is performed in a position range (ID = 1) where the distance from the screen 15 is 0 to 100 mm, the gesture is interpreted as a page break for each page. On the other hand, when the gesture is performed in a position range (ID = 2) where the distance from the screen 15 is 100 to 500 mm, the number of pages turned by one gesture depends on the distance from the screen 15. It is interpreted as a type of page break that increases linearly.

  FIG. 7 shows how the meaning of the “send fingertip to the right” gesture changes according to the distance from the screen of the gesture execution position. As shown in FIG. 7, when the execution position of the “send fingertip to the right” gesture is within the range of 0 to 100 mm distance (Z coordinate) from the screen, the gesture is displayed on the screen 15 to the document viewer. This represents an operation instruction for advancing the page to be advanced by one page. If the gesture is performed at a distance of 500 mm from the screen, the gesture is interpreted as an operation instruction for turning the total number of pages of the document being displayed, and the document viewer determines the document according to the gesture. Display the last page. In addition, when the distance from the screen where the gesture is performed is within a range of 100 mm to 500 mm, the number of pages advanced by the gesture is the number of pages that linearly increases from one page to the total number of pages of the document. The number of pages depends on the distance. If “page breaks” with different numbers of page turning per time are interpreted as different operations, gestures performed within the Z coordinate range of 100 mm to 500 mm are different for each Z coordinate at the execution position. It can also be understood as an instruction for operation.

  In the example of FIGS. 6 and 7, the number of pages turned according to the gesture increases continuously with the increase of the distance from the screen 15 beyond 100 mm from the screen 15, but instead, for example, as shown in FIG. In this way, the number of pages to be turned may be increased stepwise as the distance increases. The relationship between the gesture execution position and the number of pages shown in FIG. 7 and FIG. 8 is registered in the gesture pattern management unit 24 as information defining the content of the operation instruction corresponding to the gesture.

  Next, a second modification will be described with reference to FIGS. In this modification, the OS / application 10 displays a GUI (graphical user interface) corresponding to the operation instruction on the screen in response to the operation instruction by the gesture. FIG. 9 is a diagram illustrating an example of a GUI according to an operation instruction, and FIG. 10 is an example of operation instruction definition information in this modification.

  As shown in FIG. 10, in this example, a gesture for moving the fingertip to the left or right at a position within a range of 0 to 150 mm from the screen 15 is interpreted as an enlargement / reduction operation of the image displayed on the screen 15. For example, a gesture for moving the fingertip to the left is a reduction instruction, and a gesture for moving to the right is an enlargement instruction. Further, even a gesture within a distance of 0 to 150 mm from the same screen 15 is interpreted as a drag operation within the page. By dragging within the page, for example, range designation within the page is performed. In this example, even if the hand moves to the right or left within the same distance range of 0 to 150 mm, the meaning of the gesture differs depending on whether it is moved while holding the hand or the fingertip is moved by extending one finger. .

  Further, in the example of FIG. 10, the operation of moving the hand to the left or right within the range of the distance from the screen 15 of 150 to 300 mm is interpreted as a page turning operation for each page. For example, a gesture that moves the hand to the left indicates an operation of returning to the previous page, and a gesture that moves to the right indicates an operation of moving backward by one page. Further, the operation of moving the hand to the right or left within a range of 300 to 450 mm from the screen 15 is interpreted as an operation of sending a plurality of pages with one gesture. Also in this case, for example, a gesture for moving the hand to the left indicates an operation for returning to the previous page, and a gesture for moving to the right indicates an operation for proceeding to the next page.

  The screen display of the information processing apparatus when the gesture interpretation is performed according to the definition information of FIG. 10 is as illustrated in FIG. For example, when a gesture (“enlargement / reduction”) for moving the fingertip to the right or left within a range of 0 to 150 mm from the screen 15 is performed, the display image 15 a is displayed on the screen 15. In the display image 15a, a GUI 152 for instructing input of a display enlargement / reduction ratio is displayed. When the slider 153 on the GUI 152 is selected with, for example, a mouse and dragged to the right or left, the display image can be enlarged or reduced. In this embodiment, instead of the operation by the mouse, a gesture for moving the fingertip to the right or left within the range of the distance of 0 to 150 mm from the screen 15 is used.

  When a gesture for moving the hand to the right or left within the range of a distance of 150 to 300 mm from the screen 15 (“page advance”) is performed, a display image 15 b is displayed on the screen 15. The display image 15b displays a GUI 154 including a display indicating “currently displayed page number / total number of pages” and a GUI button for instructing page feed one page at a time in the front and rear directions. In addition to this gesture, the page to be displayed can be returned or advanced by clicking the page feed button of the GUI 154 with a mouse or the like.

  Further, when a gesture of moving the hand to the right or left within the range of 300 to 450 mm from the screen 15 (multiple page feed) is performed, the display image 15c is displayed on the screen 15. The display image 15c displays a GUI 156 in which thumbnail images of each page are arranged in the page order. In this GUI 156, the thumbnail corresponding to the page displayed in the main display column is highlighted. If you move your hand to the right, pages that are advanced by the number of pages corresponding to the distance you moved are displayed in the main display area.

  In this example, as illustrated in FIG. 9, GUIs 152, 154, and 156 for operation are displayed on the screen 15, so that the user can select the operation mode selected with the current gesture from this display as “enlargement / reduction”, It recognizes whether it is “1 page feed” or “multiple page feed”.

  Instead of displaying the operation GUIs 152, 154, and 156 corresponding to the operation instructed by the gesture as in the example of FIG. 9, icons 158a and 158b representing the instructed operation are displayed as shown in FIG. May be displayed on the screen 15.

  In the example of FIG. 11A, a gesture of moving a finger at a position away from the screen 15 is recognized as a “hand tool” operation by the gesture pattern control unit 22. In accordance with the recognition result, the gesture pattern control unit 22 instructs the OS / application 10 (“document viewer”) to execute the “terahira tool” operation and operates the “document viewer” displayed on the screen 15. An instruction is given to select (active) the icon 158a of the “hand tool” from the mode selection icon group. Thereby, the icon 158a of the “terahira tool” becomes a display form indicating the selected state, and the user knows from this display that the currently selected operation mode is the “terahira tool”.

  Similarly, in the example of FIG. 11B, a gesture of moving a finger at a position close to the screen 15 is recognized as a “pen tool” operation by the gesture pattern control unit 22. The gesture pattern control unit 22 instructs the OS / application 10 (“document viewer”) to execute the “pen tool” operation according to the recognition result, and also operates the operation mode of the “document viewer” displayed on the screen 15. An instruction is given to select the “pen tool” icon 158b in the selection icon group. As a result, the “Pen tool” icon 158b is in a display form indicating the selected state, and the user knows from this display that the currently selected operation mode is “Pen tool”.

  Next, a third modification will be described with reference to FIGS. As described above, in an example of the control of the present embodiment, the meaning of the same gesture is changed according to which range the gesture execution position (for example, the distance from the screen 15) is. Here, when the position of an instruction unit (such as a fingertip) at each time point while a gesture is being performed is regarded as the execution position of the gesture, a gesture that spans a plurality of position ranges becomes a problem. For example, in the example using the definition information in FIG. 3, a case where a line drawing is started on the screen by the “pen tool” by performing a drag gesture with a fingertip within a position range of ID = 1 (0 to 150 mm from the screen). Think. In the drag gesture, it is possible that the fingertip moves in a direction away from the screen and enters the position range of ID = 2 (“Tenoko Tool”). In such a case, as an example, a method may be adopted in which the gesture is considered to be completed when the instruction unit (such as a fingertip) deviates from the position range where the gesture is first recognized. However, for the user, the continuous movement of the pointing unit is likely to be the same operation even if the distance from the screen changes to some extent. Is good.

  Therefore, in this modification, when the meaning of the gesture, that is, “corresponding operation” is recognized from the combination of the gesture pattern and the execution position, the position range corresponding to the operation is expanded. The position range after expansion (enlargement) corresponding to the operation includes the position range at normal time (before expansion) and is larger than the position range at normal time. As a result of this extension, even if the pointing unit such as the fingertip moves closer or further away from the screen within the gesture, it is recognized that the same gesture is continuous unless it deviates from the extended position range. The

  FIG. 12 shows an example of the processing procedure of this modification. In this procedure, first, the gesture encoding unit 20 encodes the detection signal of the motion sensor 16 (S20), and extracts a gesture pattern and an execution position indicated by the detection signal from the result of the encoding (S22). Next, the gesture pattern control unit 22 determines whether or not a gesture is started based on the extraction result of S22 (S24). In S24, for example, when an instruction unit (such as a fingertip) enters the detection area of the motion sensor 16, or when the instruction unit is in an operation state that does not correspond to any operation (for example, almost stopped). ), When the instruction unit indicates a specific movement indicating the start of a gesture, when the instruction unit detects a movement of the instruction unit after indicating a specific movement indicating the end of the gesture, It is determined that the gesture has been started when, for example, indicates an operation of touching a predetermined specific coordinate in the detection area. If the start of the gesture is not detected, the process returns to S20.

When the start of the gesture is detected in S24, the gesture pattern control unit 22 performs an operation instruction corresponding to the combination of the currently active application or OS and the extracted gesture pattern and execution position with the gesture pattern management. It specifies with reference to the part 24 (S26). If the operation corresponding to the gesture is not found in S26 (the determination result in S28 is N), in the illustrated example, the gesture is invalidated, and the process returns to S20 to recognize the next gesture. If an operation instruction corresponding to the gesture is found in S26 (the determination result in S28 is Y), the position range corresponding to the operation instruction is expanded (S39),
The operation instruction is input to the application or OS (S32). In response to this input, the application or OS executes an operation instructed by the user through the gesture. The process returns to S20, and the gesture recognition process is continued. When the position range corresponding to the operation instruction is expanded, the position range of another operation instruction adjacent to the position range is reduced by the amount corresponding to the expanded position range corresponding to the operation instruction.

  When it is determined in S24 that the gesture is not started, the gesture pattern control unit 22 determines whether or not the gesture pattern and the execution position extracted in S20 and S22 are consistent with the currently selected operation ( S34). That is, when the gesture starts (when the determination result in S24 is Y), the operation selected by the user by the gesture is specified in S26. Whether the gesture following the start matches the selected operation is determined. Judgment is made. This determination is performed based on the combination of the gesture pattern and the execution position as in S26. However, in this determination, as the operation instruction definition information (see, for example, FIG. 3), the position range corresponding to the operation specified at the start of the gesture is expanded, and the position range corresponding to another operation adjacent to this position range is displayed. The one reduced by the enlargement is used. If the gesture pattern extracted in S22 corresponds to the gesture pattern corresponding to the currently selected operation, and the gesture execution position extracted in S22 belongs to the expanded position range corresponding to the operation. , It is determined that the gesture is consistent with the current operation. If it is determined in S34 that it is “consistent”, the gesture pattern control unit 22 transmits an operation parameter corresponding to the gesture to the OS / application 10 as a parameter of the currently selected operation (S36). For example, if it is determined that the gesture is a “pen tool” operation at the start of the gesture, and the position of the fingertip that continues to move is within the position range corresponding to the “pen tool” (though enlarged in S30), Information on the coordinates on the screen 15 corresponding to the X and Y coordinates of the fingertip is notified from the gesture pattern control unit 22 to the OS / application as a parameter indicating the coordinates of the pentip of the “pen tool”. If it is determined in S34 that it is “not consistent”, the gesture pattern control unit 22 instructs the OS or application to cancel the selection of the currently selected operation (S38), and that in the operation instruction definition information. The position range corresponding to the operation is reset to the range before enlargement (normal time).

  In the above, embodiment and the modification of this invention were demonstrated. In the example described above, the case where the position range divided in the direction perpendicular to the screen of the information processing apparatus is used as the position range of the gesture corresponding to the operation instruction is described as an example. A position range divided in a direction parallel to the screen or a position range divided in both a direction perpendicular to the screen and a direction parallel to the screen may be used.

  In the above example, an example of gesture input for operations related to display on the screen 15 (drawing, scrolling, page turning, etc.) has been shown. However, the control processing of the present embodiment is an operation not related to screen display. Is also applicable. For example, the mechanism of the present embodiment can be applied to input of operation instructions for an electronic musical instrument. For example, even for a gesture indicating the same “vibrato”, the amount of vibrato (pitch amplitude) is changed in accordance with the distance from the electronic musical instrument at the position where the gesture is performed.

  The processing of the embodiment described above is implemented as a program executed on a computer, such as a computer such as a personal computer or a tablet terminal, or a computer built in an apparatus such as a digital multifunction peripheral. The computer referred to here includes, for example, a microprocessor such as a CPU, a memory (primary storage) such as a random access memory (RAM) and a read only memory (ROM), a HDD (hard disk drive), a flash memory, and the like as hardware. A circuit configuration in which a secondary storage controller that controls secondary storage, various I / O (input / output) interfaces, a network interface that performs control for connection to a wireless or wired network, and the like are connected via a bus, for example. Have. In addition, various standards such as a disk drive and a flash memory for reading and / or writing to a portable disk recording medium such as a CD, a DVD, and a Blu-ray disk are connected to the bus via, for example, an I / O interface. A memory reader / writer for reading from and / or writing to the nonvolatile recording medium may be connected. A program in which the processing content of each functional module exemplified above is described is stored in a secondary storage device such as a flash memory via a recording medium such as a CD or DVD, or via a communication means such as a network, and the computer To be installed. The program stored in the secondary storage device is read into the RAM and executed by a microprocessor such as a CPU, thereby realizing the functional module group exemplified above.

10 OS / application, 12 display driver, 14 display device, 15 screen, 15a, 15b, 15c display image, 16 motion sensor, 18 sensor driver, 20 gesture encoding unit, 22 gesture pattern control unit, 24 gesture pattern management unit.

Claims (6)

Storage means for storing operations corresponding to combinations of gesture patterns and positions;
An acquisition means for acquiring a gesture pattern performed by a user and information on a position where the gesture is performed from a gesture recognition device that recognizes the gesture and the three-dimensional position where the gesture is performed;
Control means for obtaining an operation corresponding to the combination of the pattern and position of the gesture acquired by the acquisition means from the storage means, and performing control so that the obtained operation is executed;
An information processing apparatus. The storage means stores an operation corresponding to a combination of a gesture pattern and a position range,
The control means expands the range to which the start position belongs when acquiring the start position of the gesture from the gesture recognition device.
The information processing apparatus according to claim 1.   The information processing apparatus according to claim 1, wherein the control unit performs control so that information indicating the type of operation obtained is displayed on a display screen of the information processing apparatus.   The information processing apparatus according to claim 1, wherein the position is a distance from a display screen of the information processing apparatus.   5. The information processing according to claim 1, wherein the storage unit stores a relationship in which an operation amount for the same type of operation increases as the distance increases. 6. apparatus. Computer
Storage means for storing operations corresponding to combinations of gesture patterns and positions;
An acquisition means for acquiring a gesture pattern performed by a user and information on a position where the gesture is performed from a gesture recognition device that recognizes the gesture and a three-dimensional position where the gesture is performed;
Control means for obtaining an operation corresponding to the combination of the pattern and position of the gesture obtained by the obtaining means from the storage means and performing control so that the obtained operation is executed;
Program to function as.