JP2017049661A - Information processing apparatus, control method therefor, program and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem that when recognizing a touch operation based on a proximity state of an object and a target surface in the prior arts, an operation to determine an operation class after a state shift from a touched state to a non-touched state may be recognized erroneously.SOLUTION: In order to solve the problem, the information processing apparatus comprises: an image acquisition part 120 for acquiring an input image obtained by imaging a space in which a person inserts an arm and performs an operation; an instruction point detection part 122 for detecting an instruction point corresponding to a fingertip from a region where the arm of the person is photographed, in the input image; a reference determination part 123 for determining a position different from the fingertip in the region of the input image as a position of a reference point; and an operation recognition part 124 by which, when a touched state with respect to a predetermined surface included in the space is shifted to a non-touched state for the fingertip, it is determined whether a predetermined touch operation is inputted by using movements of positions corresponding to the fingertip detected from a plurality of the input images acquired within the same period and a movement of the position of the reference point.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for recognizing a touch operation based on a proximity state between a recognition target and a touch target surface.

  In recent years, user interfaces (hereinafter referred to as UI) for performing projector display have been developed. In the above-described UI, a user interface component (hereinafter referred to as UI component) such as a button is projected onto a table, a wall, a document, or an object, and the user touches the UI component. The above-described UI is advantageous in that it can be carried and rich interaction combining a physical medium and an electronic medium becomes possible. In such a scene, a distance image captured by a stereo camera, a distance image sensor, or the like is used, and the distance between the target surface and a part of a predetermined object used for the operation (for example, the fingertip of the user's hand) In many cases, the contact between the two is detected based on the size of the contact. Specifically, a predetermined threshold is provided for the distance between the target surface and the user's fingertip. If the distance is smaller than the threshold, it is determined that the “touch state” indicates that the target surface is touched, and if the distance is greater than the threshold, the “non-touch state” indicates that the target surface is not touched. . Different threshold values may be used for the two types of determination. The action of moving an object to make a transition from a touched state to a non-touched state is called a release, and the non-touched state is sometimes called a released state.

  In recognition of touch operations using a stereo camera, a distance image sensor, etc., in the same way as recognition using a touch panel, instructions for touch operations such as taps and flicks are based on the time from touch to release and the trajectory of the pointing points during that time. Often recognizes the content. A tap is an operation of touching one point and releasing it without moving it. The flick is an operation of releasing while moving at high speed so as to pay (or play) on the target surface. The pinch is an operation for enlarging or reducing the distance between the designated points in a state where two points are designated. In order to be able to recognize various touch operations, it is necessary to accurately distinguish and recognize them.

  Patent document 1 discloses recognizing a flick when the speed at which an object such as a finger is released from a touch panel is faster than a set threshold value. In Patent Document 2, in order to distinguish flicking with two fingers from a pinch operation, not only changes in the distance between two points while touching the touch panel, but also the comparison is made by dividing the moving direction of the two points into components. Is disclosed.

JP 2012-128830 A JP 2014-130385 A

  In the case of a system that recognizes touch / non-touch using the distance image for the size of the distance to the touch target surface, depending on the threshold, even if the object is not actually touching the touch target surface, While touching, a touch state is detected. Therefore, even if the user intends to move the finger after moving it away from the touch target surface according to the information recognized by the user using tactile sensation or vision, the system says that the finger was moved during the touch state. It may be recognized. Further, an error may occur in the process of acquiring distance and position information from the distance image. Depending on the magnitude of the error and the timing at which it occurs, a transition between the touch state and the non-touch state is detected against the user's intention. According to these factors, even if the user only intends to input a tap, the device may be recognized as a flick, and a situation that may be regarded as a malfunction of the device from the user's point of view may occur. In particular, operations such as tapping and flicking that determine the operation type after transitioning to the non-touch state are easily confused by being recognized as movement after transition to the non-touch state and movement during touch. In Patent Documents 1 and 2, no consideration is given to such a problem that is likely to occur in a system that does not have a sensor function that directly detects contact between a touch target surface and an object.

  The present invention has been made in view of the above problems, and even when the touch operation is recognized based on the proximity state between the object and the target surface, the operation recognition accuracy in which the operation type is determined after the state transition from the touch state to the non-touch state is achieved. It aims at improving.

  In order to solve the above-described problem, the present invention provides an image acquisition unit that acquires an input image obtained by imaging a space in which a person inserts an arm and performs an operation, and an area in which the human arm is captured as a subject in the input image. Detecting means corresponding to a fingertip of the person's hand, and a position of the person's arm at a position different from the fingertip in a region of the input image where the person's hand is captured as a subject. Reference determining means for determining the position of a reference point serving as a reference, and when the touch state of the fingertip with respect to a predetermined surface included in the space is changed to a non-touch state, the image acquisition means during the same period A predetermined touch using a movement of a position corresponding to the fingertip detected by the detection means and a movement of the position of the reference point determined by the reference determination means from the plurality of acquired input images. operation And an operation recognition means for determining whether the inputted.

  According to the present invention, even when a touch operation is recognized based on the proximity state between the object and the target surface, it is possible to improve the recognition accuracy of the operation in which the operation type is determined after the state transition from the touch state to the non-touch state. .

The figure which shows an example of the external appearance and structure of a table top system using information processing apparatus The flowchart which shows an example of the flow of touch recognition processing The flowchart which shows an example of the flow of recognition processing of the operation decided after release Flowchart showing an example of the temporary tap recognition process and the temporary flick recognition process Flowchart showing an example of tap or flick selection processing in the first embodiment The figure which shows the specific example of a user's arm and finger | toe state when a flick or a tap is input The figure which shows the magnitude | size of the locus | trajectory of a fingertip in case a tap and a flick are input, and a 1st vector Diagram showing the angle between the trajectory and the discrete fingertip trajectory observed when a tap and flick are input The flowchart which shows an example of the tap or flick selection process in a modification The figure which showed the judgment result and the final recognition result in each step of the touch recognition processing with respect to the input The figure which shows an example of the application in 1st Embodiment and a modification.

  Embodiments of the present invention will be described below in detail with reference to the drawings. In addition, embodiment described below shows an example at the time of implementing this invention concretely, and is not restricted to this.

<First Embodiment>
Of the operations in which the operation type is determined after the state transition from the touch state to the non-touch state, a tap is generally input to designate one point on a predetermined surface that is a target surface for touch input. It is often an operation type. In many cases, a condition that must be satisfied in order to recognize a tap (hereinafter referred to as a tap condition) is released without moving in a direction along the target surface after the object is in a touched state with respect to the target surface. That is. Further, the tap condition may be detected from a touch to release in a very short time. In this case, in order to satisfy the condition regarding time, it is assumed that the speed of movement of the fingertip for tapping is relatively fast. On the other hand, among the operations in which the operation type is determined after the state transition from the touch state to the non-touch state, the flick is generally an operation that is often input to specify a certain direction along the target surface. It is a type. Further, for flicking, a flicking speed based on the moving speed of the fingertip immediately before release may be specified. In many cases, a condition to be satisfied in order to recognize a flick (hereinafter referred to as a flick condition) is that the object exceeds a predetermined speed immediately before the object and the target surface transition from the touch state to the non-touch state. The object moves in a direction along the target surface. In many cases, the flick condition does not include a condition related to the movement of the fingertip while the object and the target surface are in the touch state. In other words, the flick may be input without moving in the direction along the target surface like a tap, or may be input after the fingertip has moved to some extent along the target surface in the touch state. .

  FIG. 11 shows an example of an application that uses flicks and taps. FIG. 11A shows a state in which thumbnails are slid by flicking. In this application, a display surface on which a plurality of thumbnails are displayed becomes a touch target surface, and the thumbnail can be selected by specifying within a range where one thumbnail is displayed with a tap. Also, by moving the fingertip along the target surface while maintaining the thumbnail touch state, the thumbnails to be displayed at a predetermined position are advanced one by one by flicking one thumbnail row. Can do. In FIG. 11A, the arm outlined by the solid line indicates the latest position of the user's arm, and the arm outlined by the dotted line indicates the user in the past (one frame before) before moving. Represents the position of the arm.

  In the case of FIG. 11A, thumbnails 1, 2, and 3 are displayed on the target surface in the initial state. Then, the thumbnail slides halfway through the move, and then the flick is input, so that the thumbnail display position has advanced one by one from the initial state. After flicking, two thumbnails 2, 3, and 4 are displayed. Has been. In FIG. 11A, when the flick is performed, the arm itself hardly moves, and only the index finger is moved quickly. In this way, as a tendency when a flick is input when the target (thumbnail) to be moved by the touch operation is small or the distance to which the target is to be moved is short, the finger is used rather than the arm as described above. There are many movements to move.

  There are various situations in which the user taps and flicks. For example, in the case of tapping, there is a method of lowering on the touch target surface almost vertically from the point you want to specify and raising it vertically as it is, but on the other hand, you can also tap while moving your hand continuously Conceivable. In the latter situation, the movement of the fingertip immediately after release has a component in a direction parallel to the touch target surface. If the transition from the touch state to the non-touch state is detected later than the timing recognized by the user, the system moves the fingertip along the touch target surface before transitioning to the non-touch state. You will recognize that you moved in the direction. When the moving speed satisfies the flick condition, the flick input is recognized even though the user intends to tap. As shown in FIG. 11A, when a flick is input, the movement of the arm itself in the same direction may be dull or may not move compared to the movement distance of moving in the direction along the fingertip touch target surface. is there. On the other hand, when a tap is input, it is required that there is no movement in the direction along the target surface between the touch and release under the tap condition. Therefore, in order to prevent the user from consciously sliding the finger, even if there is a movement of the arm itself before and after the tap, there is a tendency that the fingertip does not move much with respect to the arm. Therefore, in this embodiment, tap and flick are examples of operations for determining the operation type after the state transition from the touch state to the non-touch state, and in order to improve the recognition accuracy, in addition to the movement of the fingertip before and after the release. Then, the information detected regarding the movement of the arm is used. Specifically, in order to obtain the movement distance of the arm itself, a reference point is set in a part other than the finger of the user's hand, and when determining whether a flick or a tap is input, before the release is detected The relationship between the movement of the fingertip and the movement of the reference point is used in the same period. However, taps and flicks are examples. This embodiment is applicable to the case of recognizing and distinguishing an operation whose operation type is determined after transitioning to a non-touch state.

  FIG. 1A shows an example of the appearance of a tabletop interface system in which the information processing apparatus 100 described in this embodiment is installed. The information processing apparatus 100 can set an arbitrary plane such as a table or a wall surface as a target surface by irradiating the projection light from the projection light irradiation unit 105 of the projector. In the case of the table top interface system shown in FIG. 1, the information processing apparatus 100 is installed on a table serving as the touch target surface 101 and a display image is projected onto the table surface. Here, the circular image 2 is a UI component projected on the table surface by the projector. Hereinafter, various images such as UI parts and photo thumbnails projected on the table surface by the projector are collectively referred to as display items.

  The light receiving unit 106 indicates a viewpoint of a distance image obtained by a distance image sensor using an infrared pattern projection method. In the present embodiment, the light receiving unit 106 is installed at a position for imaging at an angle of view looking down from above on the target surface. Accordingly, each pixel of the distance image obtained by the distance image sensor 115 reflects the distance from the light receiving unit 106 to the subject. The distance image acquisition method will be described based on an infrared pattern projection method that is less affected by ambient light or display on the table surface as an example. However, a parallax method or an infrared light reflection time method (Time- of-Flight method) can also be used. Here, in the operation area 104, the range that can be projected by the projector and the field-of-view range of the distance image sensor 115 are the same, and the range is hereinafter referred to as the operation area 104.

  In this embodiment, the user uses the arm 103 to input a touch operation or a space gesture operation to the table top system. The touch target surface 101 to which the touch operation is input is on the table surface on which the operation area 104 is defined. The distance image sensor 115 is installed so that a space including the table surface is included in the angle of view, and the user operates with the arm 103 inserted into the space. However, in the present embodiment, not only when the touch target surface 101 is a table surface (a surface directed vertically), for example, a surface of an object that is not a wall surface or a plane, or a virtual surface. It is also applicable to cases. In the touch operation and the space gesture operation, for example, a display item projected on the touch target surface 101 can be moved as an operation target. In this embodiment, as shown in FIG. 1, the three-dimensional position is set by setting the x-axis and y-axis on a two-dimensional plane parallel to the operation region 104 and the z-axis in the height direction orthogonal to the operation region 104. Treat information as coordinate values. Therefore, in the present embodiment, the direction along the touch target surface or the direction parallel to the touch target surface refers to a direction defined on the xy plane. However, when the target surface is not a plane or depending on the positional relationship between the user and the target surface, the coordinate axes do not necessarily have to be parallel or orthogonal to the target surface. In this case as well, the z axis is set in a direction for detecting the proximity relationship between the recognition target and the target surface (the degree of the distance between the two), and the x axis and the y axis are set in the direction intersecting with the z axis. Is done.

  In the following description, a case where the user's own arm 103 is used as an operation body used by the user to input a touch operation will be described as an example. However, in this embodiment, a part (protrusion part) used for touch input in the operation body, such as when a touch operation is input with a stylus held by a hand instead of a finger or when a robot arm is used instead of a human body. However, it can be applied to the case where it is movable independently of the operating body itself.

  FIG. 1B is a block diagram illustrating an example of a hardware configuration of the information processing apparatus 100 according to the present embodiment. In the figure, a CPU 110 comprehensively controls each device connected via a bus 113. In addition to the operating system (OS), each processing program, device driver, and the like according to the present invention as shown in the flowcharts described later are stored in the ROM 112, temporarily stored in the RAM 111, and appropriately executed by the CPU 110. The RAM 111 is used as a temporary storage area such as a main memory or work area of the CPU 110 that can be accessed at high speed. Here, the OS, each processing program, and the like may be stored in the external storage device 116, and in this case, information necessary when the power is turned on is appropriately read into the RAM 111. The display I / F 114 converts a display item (display image) generated inside the information processing apparatus 100 into a signal that can be processed by the projector 118. The input / output I / F 114 acquires distance information from the distance image sensor 115, converts it into information that can be processed by the information processing apparatus 100, and converts data between the storage device 116 and the information processing apparatus 100.

  In the present embodiment, it is assumed that the digital data projected by the information processing apparatus 100 is stored in the storage device 116. As the storage device 116, a storage device 116 connected via various input / output I / Fs 114 such as a disk device, a flash memory, a network, and a USB is used. In the present embodiment, the distance image sensor 115 is an imaging unit used to acquire information on the operation area 104. The image acquired by the distance image sensor 115 is temporarily stored in the RAM 111 as an input image, and is appropriately processed and discarded by the CPU 110. However, necessary data may be stored in the storage device 116.

  FIG. 1C is a block diagram illustrating an example of a functional configuration of the information processing apparatus 100 according to the present embodiment. The information processing apparatus 100 includes an image acquisition unit 120, an operation tool acquisition unit 121, an indication point detection unit 122, a reference determination unit 123, an operation recognition unit 124, and an output control unit 125. Each of these functional units is realized by the CPU 110 developing a program stored in the ROM 112 in the RAM 111 and executing processing according to each flowchart described later. Further, for example, when hardware is configured as an alternative to software processing using the CPU 110, arithmetic units and circuits corresponding to the processing of each functional unit described here may be configured. The image acquisition unit 120 inputs an image obtained by the distance image sensor 115 capturing an image of the space on the touch target surface 101 where the user of the information processing apparatus 100 inserts his / her arm and operates. Get as. The input image in the present embodiment is a distance image in which the distance from the distance image sensor 115 to the subject is reflected in the pixel value of each pixel. The operation tool acquisition unit 121 extracts an arm region in which the user's arm as the operation tool is captured as a subject from the input image acquired by the image acquisition unit 120. In the present embodiment, the extracted arm region is tracked as an operating body. The indication point detection unit 122 detects a position corresponding to the fingertip in the user's hand from the arm region extracted by the operating tool acquisition unit 121 as an indication point. When a stylus or the like is used instead of the finger, the tip is detected as an indication point as corresponding to the fingertip. In the present embodiment, it is assumed that the coordinates of one point among a number of pixels of the input image are detected as the position of the designated point. The reference determination unit 123 determines a reference point to be treated as a reference for the position of the arm as the operating body at a position different from the fingertip detected by the indication point detection unit 122 in the arm region. For example, the coordinates of one point corresponding to the center of the back of the hand or the center of the wrist are obtained. The operation recognition unit 124 receives a predetermined touch input to the information processing apparatus 100 based on whether the position of the indication point detected by the indication point detection unit 122, the direction and speed related to the movement of the indication point, satisfy the predetermined condition in the dictionary. Recognize operations. At that time, when the fingertip transitions from the touched state to the non-touched state with respect to the touch target surface, the release is performed using the movement of the indication point in the predetermined period and the movement of the reference point in the same period. It is determined whether or not a predetermined touch operation that determines the presence or absence of input is input later. The output control unit 125 controls various outputs corresponding to the operations recognized by the operation recognition unit 124. For example, an image to be projected onto the touch target surface 101 is generated and output to the projector 118. Each functional unit uses the RAM 111, the ROM 112, or the storage device 116 as a holding unit that holds data and information being processed.

  In addition, it is possible to configure a function unit according to the purpose of use and application of the information processing apparatus 100, such as a gesture recognition unit that recognizes a spatial gesture operation such as a hand gesture.

  Next, the touch recognition process performed by the information processing apparatus 100 will be described in detail using the flowchart of the touch recognition process shown in FIG. In the present embodiment, the process of the flowchart of FIG. 2 is started in response to the distance image captured by the distance image sensor 115 being input to the information processing apparatus 100. In the present embodiment, the process of the flowchart of FIG. 2 is repeated each time a distance image is input from the distance image sensor 115. Therefore, the cycle in which the process is repeated matches the frame rate of the captured image of the distance image sensor 115.

  In step S <b> 100, the image acquisition unit 120 acquires the distance image captured by the distance image sensor 115 as an input image at regular intervals, and stores it in the RAM 111 as needed. The pixel value of the distance image captured by the distance image sensor 115 represents the distance to the distance image sensor 115.

  In step S <b> 101, the operation tool acquisition unit 121 extracts a region (arm region) where the arm of the person who is the operation tool is captured from the distance image acquired by the image acquisition unit 120. Then, position information indicating the contour of the extracted arm region is acquired, and the information is stored in the RAM 111. Here, the person's arm refers to all parts from the person's shoulder to the fingertip, and an area where a portion corresponding to the person's arm is captured in the captured image is an arm area. Moreover, in this embodiment, a hand refers to all of the arms from the wrist. The hand includes parts such as five fingers, a palm, and an instep. As an actual process of the operation tool acquisition unit 121, a threshold value process is performed on the coordinate value in the z direction indicated by each pixel of the acquired distance image, so that the coordinate value is higher than the table and the image A region in contact with the end is extracted as an arm region. However, the extraction method of the arm region is not limited to this, and for example, a portion corresponding to the skin color region in the RGB image of the operation region 104 separately captured may be extracted. In addition, the operating tool obtaining unit 121 according to the present embodiment obtains the coordinates of the contour line based on applying a differential filter to the input image from which the arm region is extracted. Further, the operating tool acquisition unit 121 acquires an average position (hereinafter referred to as an intrusion position) of pixels in which the arm region and the image edge are in contact with each other, and stores this in the RAM 111.

  In step S <b> 102, the reference determination unit 123 determines a reference point at a position that is not a fingertip in the arm region acquired by the operating tool acquisition unit 121. The reference point is determined in order to obtain coordinates representing the position of the arm itself. In the present embodiment, a point corresponding to the center of the back of the hand in the arm region is specified, and the position is determined as a reference point. As a method of specifying the center of the back of the hand, for example, for each pixel in the arm region, the product of the distance from the pixel acquired as the intrusion position and the shortest distance to the contour line of the arm region is obtained, and the pixel having the maximum The position of is calculated. However, this method is an example, and another method may be used. The reference determination unit 123 acquires the three-dimensional position of the reference point and holds the information in the RAM 111. The reference point pixel in the distance image represents the distance to the surface portion of the center of the back of the hand as viewed from the distance image sensor 115. In the table top interface system of FIG. 1A, since the input image is captured from above, the reference point is expressed as “the center of the back of the hand”. However, depending on the setting method of the imaging system, the same position is used. In some cases, it is more natural to express the “center of palm”. For example, in the case of a configuration in which an input image obtained by capturing a touch target surface having transparency from below is obtained, the distance image is captured on the surface on the palm side of the arm. In this way, the name of the hand part may be appropriately selected.

  In the present embodiment, the calculation process in which the calibration parameter is reflected on the position and the pixel value of the pixel determined as the reference point in the distance image is shown in FIG. 1A (x, y). , Z) to coordinates. Then, the conversion result is acquired as three-dimensional position information of a reference point representing the position of the entire arm. In the present embodiment, the example of obtaining the center of the back of the hand as the reference point of the arm region has been described. Absent. For example, the center of the wrist may be obtained.

  In step S <b> 103, the indication point detection unit 122 identifies a pixel corresponding to the fingertip as an indication point indicating the position indicated by the operation tool from the arm region acquired by the operation tool acquisition unit 121. For example, the point where the Euclidean distance from the intrusion position is maximum in the arm region is detected as a fingertip. However, this detection method is an example, and another method may be used. Further, the pointing point detection unit 122 acquires the three-dimensional position information of the fingertip from the position on the image and the pixel value of the pixel corresponding to the fingertip in the same manner as in step S102, and stores the coordinate information in the RAM 111. To do. Here, the z-coordinate of the fingertip represents the distance between the target surface and the fingertip. In other words, the z coordinate of the pixel corresponding to the fingertip represents the height of the fingertip from the target surface 101. Hereinafter, the z coordinate of the indication point acquired in step S103 is referred to as fingertip height. In this embodiment, the Euclidean distance (mm) in the three-dimensional space is used as a measure of the fingertip height, but another distance measure may be used.

  In steps S104 to S106, the operation recognition unit 124 determines whether the fingertip is in a touch state or a non-touch state. Note that the initial state of the information processing apparatus 100 is a non-touch state. First, in step S104, the operation recognition unit 124 determines whether the acquired fingertip height is equal to or less than a predetermined touch threshold value. The touch threshold is a value of a predetermined height for detecting a touch state that can be considered that the user's fingertip is touching the touch target surface. In the present embodiment, an appropriate value is set in advance in consideration of the detection error of the distance image sensor 115 and the error that occurs in the calculation for calculating the z coordinate based on the pixel value. For example, it is assumed that the setting is made before the operation for the application is started, such as when the information processing apparatus 100 is designed or installed. If the fingertip height is less than or equal to the predetermined touch threshold (YES in step S104), the process proceeds to step S107. If the fingertip height is not less than or equal to the predetermined touch threshold (NO in step S104), the process proceeds to step S105.

  In step S105, the operation recognition unit 124 determines whether the obtained fingertip height is equal to or less than a predetermined release threshold. The release threshold is a value of a predetermined height for detecting a non-touch state that can be considered that the user's fingertip has finished touch input. In the present embodiment, similarly to the touch threshold, an appropriate value is set in advance in consideration of the detection error of the distance image sensor 115 and the error that occurs in the calculation for calculating the z coordinate based on the pixel value. Has been. If the fingertip height is less than or equal to the predetermined release threshold (YES in step S105), the process proceeds to step S106. If the fingertip height is not less than the predetermined release threshold (NO in step S105), the process proceeds to step S113. In the threshold processing of step S104 and step S105, which determination result is used when the fingertip height matches each threshold is selected as appropriate according to the system configuration and environment, as in the method of determining the threshold. That's fine. In this embodiment, at least the case where the fingertip height is smaller than the touch threshold is defined as the touch state, and the case where the fingertip height is greater than the release threshold is defined as the non-touch state, and the fingertip height exceeds the release threshold. The transition from the touch state to the non-touch state is detected.

  In step S106, the operation recognition unit 124 determines whether the fingertip has already transitioned to the touch state as a result of the processing up to the previous frame. In the present embodiment, information indicating the last transition state is held in the RAM 111 in step S107 or step S113 described later. The operation recognition unit 124 performs the determination in step S <b> 106 by referring to the RAM 111. If already touched (YES in step S106), the process proceeds to step S108. If not in the touch state (NO in step S106), the process of the flowchart of FIG. 2 for the input image of the current frame is terminated. In the present embodiment, the case of not being in a touch state corresponds to the case of being in a non-touch state. If the transition to the touch state is not detected, such as immediately after the information processing apparatus 100 is activated, and the result of the state transition is not held in the RAM 111, the determination result in step S106 is always NO. Accordingly, the case where the determination of NO is made in step S106 is after the transition to the non-touch state or the initial state where the touch input has not yet been performed. It can be said that it is not necessary to recognize the touch operation at the time.

The processing from step S107 to step S112 is processing performed when the fingertip is in a touched state with respect to the touch target surface. In step S <b> 107, the operation recognition unit 124 changes the fingertip state to the touch state, and stores information indicating the change in the RAM 111. In step S108, the operation recognition unit 124 stores the current time in the RAM 111 as the touch time. In this embodiment, the touch start time is acquired in each frame until the next transition to the non-touch state is detected, and stored in the RAM 111. In step S109, the operation recognizing unit 124 stores the coordinate information indicating the position of the reference point determined in the current frame in the RAM 111 as the position of the latest reference point. Then, the coordinate information indicating the position of the fingertip (designated point) detected from the current frame is held in the RAM 111 as the latest designated point. The touch position is acquired in each frame until the next transition to the non-touch state is detected, and stored in the RAM 111. In steps S110 to S112, the operation recognition unit 124 recognizes the move. The move is used to input an instruction to move the display item as the fingertip is moved along the touch target surface while being in the touched state. A move is sometimes called a drag. In step S110, the operation recognizing unit 124 acquires the movement distance of the designated point in the touch state. The movement distance acquired here is a movement distance in a direction parallel to the touch target surface, that is, on the xy plane. In the present embodiment, the operation recognizing unit 124 refers to the position information of the pointing point held in the RAM 111, and determines between the two points from the x coordinate and the y coordinate of the position of the pointing point in the previous frame and the current frame. Get the distance. The acquired distance is the movement distance of the designated point between frames.

  In step S111, the operation recognition unit 124 determines whether the acquired movement distance is equal to or greater than a predetermined move threshold. If the acquired movement distance is equal to or greater than the predetermined move threshold (YES in step S111), the process proceeds to step S112. If the acquired movement distance is not greater than or equal to the predetermined move threshold (NO in step S111), the process of the flowchart of FIG. 2 for the current frame is terminated. The move threshold is set in advance to a distance that allows the user to consider that the fingertip has been intentionally moved in consideration of the detection error of the position of the indication point. The move threshold may be the same as the distance threshold included in the tap condition or may be a distance greater than the tap threshold, but at least if the move is recognized even once, the tap condition is set not to be satisfied. Need to be done. This is because “move” and “tap” are exclusive operation types. In the present embodiment, the distance acquired in step S110 is the distance on the xy plane, which is an operation of placing the finger on the touch target surface while the move itself is in the touch state, and moving in the z direction. This is because the quantity has no meaning as an operation. In an environment where the touch threshold can be set sufficiently small, the difference that occurs in the z-coordinate between frames when a move is performed is very small, and therefore the determination result is affected even if the z-coordinate is taken into account when calculating the movement distance. There are few. However, normally, the touch threshold is set to have a size sufficient to absorb the height detection error. Therefore, in this embodiment, the direction in which the movement distance is calculated is limited so that the influence of movement in the z-axis direction is not reflected in the determination of the presence or absence of the move.

  In step S112, the operation recognizing unit 124 recognizes that the move has been input, and notifies each functional unit related to the response to the input operation, such as the output control unit 125. For example, a move event is issued. The output control unit 125 acquires information to be output in response to the move event from the ROM 112 or the storage device 116 according to the application regulations, and causes the output device such as the projector 118 to output the information. When the process of step S112 is completed, the process of the flowchart of FIG. 2 for the current frame is terminated.

  On the other hand, step S113 to step S117 are processes executed when it is determined that the fingertip is in a non-touch state. In step S <b> 113, the operation recognition unit 124 changes the state of the fingertip to the non-touch state, and holds information indicating the change in the RAM 111. In this embodiment, the release operation is recognized when a transition from the touch state to the non-touch state is detected. In step S114, the operation recognition unit 124 stores the current time in the RAM 111 as the release time. As for the release time, only the latest information is always held in the RAM 111. In step S115, the operation recognizing unit 124 holds coordinate information representing the position of the reference point detected from the current frame in the RAM 111 as the position of the reference point at the time of release. Then, coordinate information representing the position of the fingertip (indicated point) detected from the current frame is held in the RAM 111 as position information of the reference point at the time of release. Position information of reference points and pointing points at the time of release Only the latest information is always held.

  In step S116, the operation recognizing unit 124 performs an operation recognizing operation to be confirmed after release. In the present embodiment, the presence or absence of a tap and flick is determined in the recognition process for the operation to be confirmed after release. When the process of step S116 is completed, it is determined that either a tap or a flick is recognized or neither. In the present embodiment, the case where neither a tap nor a flick is recognized is a case where a release is performed without flicking after a move is input during the touch state. Details of the processing in step S116 will be described later. In step S117, the operation recognizing unit 124 clears the touch start time, the position information of the indication point and the reference point detected in the touch state, and the information indicating the presence / absence of the move, which are stored in the RAM 111. The above is the processing of the flowchart of FIG.

  Next, the operation recognition process to be confirmed after release executed in step S116 of the present embodiment will be described using the flowchart of FIG. In the present embodiment, there are two types of touch operations, tap and flick, which are determined after release, that is, after transition to the non-touch state. As described above, the tap is a predetermined touch target surface between the start of the touch and the release of the fingertip during a predetermined time (usually a short time such as several frames) and between the touch and release. This operation is recognized on the condition that there is no movement in the direction along the line. A flick is an operation recognized on the condition that a high-speed movement is made in a direction along the target surface immediately before release. As described above, the recognition conditions include the conditions under which both the tap and the flick are released. Furthermore, these two are in an exclusive relationship and are not input simultaneously. In other words, when it is determined that a tap has been input, it can be considered that there is no possibility that a flick has been input. On the other hand, when it has been determined that a flick has been input, it is possible that a tap has been input. It can be considered that there is no sex.

  In step S <b> 200, the operation recognition unit 124 acquires a first vector representing the movement of the designated point in a specific period until the transition to the non-touch state is detected, and stores the first vector in the RAM 111. In the present embodiment, as a method for acquiring the first vector, the coordinate information indicating the position when the z coordinate is the smallest among the position information of the fingertip first detected during the touch state is used as the lowest point position of the indication point. Get as. Then, the first vector in the three-dimensional space is acquired by taking the difference between the three-dimensional coordinates of the indication point at the time of release held in step S115 and the three-dimensional coordinates of the lowest point position of the indication point. Can do. However, this method is an example, and another method may be used. For example, the first vector may be obtained as a two-dimensional vector defined by xy coordinates.

  In step S <b> 201, the operation recognition unit 124 acquires a second vector representing the movement of the reference point in a specific period until the transition to the non-touch state is detected, and stores the second vector in the RAM 111. The first vector and the second vector are vectors representing the movement of the indication point and the reference point in the same period. In this embodiment, as the second vector acquisition method, first, coordinate information representing the position of the reference point acquired in the same frame where the lowest point position of the indication point is detected is used as the lowest point of the reference point. Get as position. Then, the second vector in the three-dimensional space is acquired by taking the difference between the three-dimensional coordinate of the indication point at the time of release held in step S115 and the three-dimensional coordinate of the lowest point position of the reference point. Can do. However, this is an example, and another method may be used. For example, the second vector may be obtained as a two-dimensional vector defined by xy coordinates.

  In step S202, the operation recognition unit 124 performs a temporary tap recognition process. Here, the content of the temporary tap recognition process will be described with reference to the flowchart of FIG. First, in step S300, the operation recognition unit 124 acquires a movement distance in a direction along the target surface being released. In the present embodiment, first, the operation recognition unit 124 includes the coordinate information of the lowest point position of the indication point held in the RAM 111 in the process of step S200 and the touch time corresponding to the frame when the lowest point position is detected. Is also acquired from the RAM 111. The operation recognizing unit 124 then touches the touch target surface of the instruction point during this time based on the acquired information on the lowest point position of the instruction point and the position information of the instruction point at the time of release held in the RAM 111 in step S115. Get the distance traveled along the direction. Specifically, the movement distance is calculated using only the xy component of each three-dimensional subsecond information. Note that if the movement distance obtained here is an environment in which it can be considered that the magnitude of the first vector matches the magnitude of the first vector, the operation recognition unit 124 may acquire the magnitude information of the first vector from the RAM 111 in step S301. Further, the first vector may be projected on the xy plane to obtain the magnitude thereof.

  In step S301, the operation recognition unit 124 determines whether the acquired movement distance is smaller than the tap distance threshold. The tap distance threshold is a distance provided in order to determine that the movement in the direction along the touch target surface is not performed from the touch condition to the target surface until the non-touch state among the tap conditions. It is a threshold value. Therefore, as the tap distance threshold, an appropriate distance value is set so that the user does not intentionally move the finger in the direction along the target surface after absorbing the detection error of the position information. Need to be done. However, in the present embodiment, not only when the user performs an ideal input of lowering the fingertip in a direction perpendicular to the touch target surface and raising it directly up, but also when the tap is input while moving the arm itself In addition, in order to recognize the tap, a margin is given to the tap distance threshold. Conventionally, when the tap distance threshold is set large with a margin, confusion with flick input is likely to occur, but in this embodiment, tap or flick selection processing (to be described later) causes such a case. Is also processed so as not to reduce the recognition accuracy. If it is determined that the acquired movement distance is smaller than the tap distance threshold (YES in step S301), the process proceeds to step S302. If it is not determined that the acquired movement distance is smaller than the tap distance threshold (NO in step S301), the process proceeds to step S304.

  In step S302, the operation recognizing unit 124 determines whether the time in the touch state is shorter than the tap time threshold. In the present embodiment, after obtaining the difference between the touch time when the lowest point position of the indication point acquired in step S300 is detected and the release time held in step S114 as the time in the touch state. It is determined whether it is shorter than the tap time threshold. However, the time in the touch state can be calculated based on the time at which the transition from the non-touch state to the touch state is detected. Here, the tap time threshold is a threshold related to the time from touch to release among the tap conditions. For example, a time such as 1 second is set. However, when the time condition is not included in the tap condition, step S302 may be omitted. However, when a short time is set as the tap time threshold, confusion between taps and flicks tends to occur more easily. In this embodiment, it is possible to perform processing so as not to lower the recognition accuracy even in such a case to be described later. If it is determined that the time in the touch state is shorter than the tap time threshold (YES in step S302), the process proceeds to step S303. If it is not determined that the time in the touch state is shorter than the tap time threshold (NO in step S302), the process proceeds to step S304.

  In step S303, the operation recognition unit 124 sets the tap flag to “ON”, holds the tap flag in the RAM 111, and ends the temporary tap recognition process. It is assumed that the initial value of the tap flag is “OFF”. In this embodiment, when the tap flag is “OFF”, it is a case where the tap can be uniquely regarded as not being input. In step S304, the operation recognition unit 124 sets the tap flag to “OFF” and holds it in the RAM 111, and ends the temporary tap recognition process.

  Returning to the processing of the flowchart of FIG. 3, in step S <b> 203, the operation recognition unit 124 performs flick provisional recognition processing. Here, the contents of the flick provisional recognition process will be described with reference to the flowchart of FIG. In step S400, the operation recognition unit 124 acquires the release speed of the designated point. The release speed acquired here is the moving speed of the indication point in the direction along the touch target surface during a predetermined time until the transition from the touch state to the non-touch state is detected. In the present embodiment, the difference between the xy coordinates of the pointing point detected in the frame before the transition to the non-touch state is detected and the xy coordinates of the pointing point at the release held in step S115, and the frame rate And get the release speed of the indicator. However, another method may be used. For example, using the difference between the xy coordinates of the indication point detected in the frame before the transition to the non-touch state is detected and the xy coordinates of the indication point at the time of release held in step S115, the indication point The movement distance per frame may be acquired as the release speed.

  In step S <b> 401, the operation recognition unit 124 determines whether the flick speed of the acquired indication point is equal to or higher than a flick threshold that represents a predetermined speed that is a flick condition. If it is determined that the acquired flick speed is equal to or higher than the flick threshold (YES in step S401), the process proceeds to step S402. If it is not determined that the acquired flick speed is equal to or higher than the flick threshold (NO in step S401), the process proceeds to step S403. The flick condition may be provided for each of the release speed in the x direction and the release speed in the y direction.

  In step S402, the operation recognition unit 124 sets the flick flag to “ON”, holds the flick flag in the RAM 111, and ends the flick provisional recognition process. It is assumed that the initial value of the flick flag is “OFF”. In the present embodiment, when the flick flag is “OFF”, it is a case where the flick flag can be uniquely regarded as not being input. In step S403, the operation recognition unit 124 sets the flick flag to “OFF”, holds the flick flag in the RAM 111, and ends the flick provisional recognition process.

  Returning to the flowchart of FIG. 3, in step S204, the operation recognition unit 124 determines whether the tap flag is “ON” and the flick flag is “OFF”. If the determination result is YES, the process proceeds to step S205. If the determination result is NO, the process proceeds to step S206. In step S205, the operation recognition unit 124 determines that a tap has been input. Therefore, the fact that the tap has been recognized is notified to each function unit related to the response to the input operation, such as the output control unit 125. For example, a tap event is issued. The output control unit 125 acquires information to be output in response to the move event from the ROM 112 or the storage device 116 according to the application regulations, and causes the output device such as the projector 118 to output the information. When the process of step S205 is completed, the process of the flowchart of FIG. 3 is terminated, and the process proceeds to step S117 described above.

  In step S206, the operation recognition unit 124 determines whether the flick flag is “ON” and the tap flag is “OFF”. If the determination result is YES, the process proceeds to step S207, and if the determination result is NO, the process proceeds to step S208. In step S207, the operation recognition unit 124 determines that a flick has been input. Accordingly, the fact that the flick has been recognized is notified to each function unit related to the response to the input operation, such as the output control unit 125. For example, a flick event is issued. The output control unit 125 acquires information to be output in response to the move event from the ROM 112 or the storage device 116 according to the application regulations, and causes the output device such as the projector 118 to output the information. When the process of step S205 is completed, the process of the flowchart of FIG. 3 is terminated, and the process proceeds to step S117 described above.

  In step S208, the operation recognition unit 124 determines whether both the tap flag and the flick flag are “ON”. If the determination result is YES, the process proceeds to step S209. If the determination result is NO, the process of the flowchart in FIG. 3 is terminated, and the process proceeds to step S117 described above. In addition, when it determines with NO by step S208, it is a case where both a tap and a flick are not input. In the present embodiment, this corresponds to the case where the release is performed without flicking after the move is input during the touch state.

  In step S209, the operation recognition unit 124 performs tap or flick selection processing, and ends the processing of the flowchart of FIG. The tap or flick selection process is a process of selecting a more likely operation in a state in which both taps and flicks are temporarily recognized as having been input.

  Next, the contents of the tap or flick selection process will be described with reference to the flowchart of FIG. First, in step S500, the operation recognition unit 124 regards the first vector acquired in step S200 and the second vector acquired in step S201 to be regarded as having similar components in the direction parallel to the touch target surface (xy component). Determine. In the present embodiment, the similarity between the xy components of the first vector and the second vector is acquired, and the determination is performed by a comparison process with a first threshold related to the vector similarity. In this embodiment, a vector inner product is used as the vector similarity. Therefore, the closer the similarity is to 0, the closer the vector is to the vertical. Therefore, the value of the inner product is set as the first threshold. The operation recognizing unit 124 determines that two vectors are regarded as similar when the similarity is equal to or greater than a first threshold related to the similarity of the vectors. If the two vectors are considered to be similar (YES in step S500), the process proceeds to step S501. If the two vectors are not considered similar (NO in step S500), the process proceeds to step S502.

  In step S501, the operation recognition unit 124 recognizes a flick as in step S207, and ends the tap or flick selection process.

  In step S502, the operation recognizing unit 124 recognizes the tap as in step S205, and ends the tap or flick recognition process. The above is a series of touch recognition processes executed in the present embodiment. In the present embodiment, an example has been described in which a vector is used as information representing the movement of the fingertip detected as the indication point and the movement of the reference point. However, the information indicating movement is not limited to the vector format. For example, each of the movement distance and the movement direction corresponding to the magnitude and direction of the vector may be used.

  Next, a specific example of the state of the user's arm and finger when a flick or tap is input will be described with reference to FIGS. FIGS. 6A to 6D illustrate a state where the user's arm during the touch operation is viewed from the position of the distance image sensor 115. A broken arm outline 202 indicates an arm area at the lowest point of touch, and a solid arm outline 200 indicates an arm area at the time of release. A dashed circle 203 indicates a reference point at the lowest point of touch, and a solid circle 201 indicates a reference point at the time of release. The solid line arrow indicates the first vector, and the double line arrow indicates the second vector. However, when there is no movement at the lowest point of touch and at the time of release, the arm contour 202, the reference point 203, the first vector, and the second vector at the lowest point of touch are not shown. Further, in all of FIGS. 6A to 6D, the difference between the touch time at the lowest touch point and the release time is smaller than the tap time threshold.

  FIG. 6A shows a state where the user is inputting a tap. In this case, the arm itself often hardly moves in the xy plane direction. Since the magnitudes of the first vector and the second vector are almost zero, they are not shown. Since the xy coordinates of the fingertip hardly move from touch to release, the tap flag is set to “ON” in the tap recognition process, the flick flag is set to “OFF” in the flick recognition process, and the tap is finally recognized.

  FIG. 6B shows a state where the user flicks by moving the fingertip at high speed without moving the arm itself. However, it is assumed that the movement in the xy plane direction is not performed in a short time when the finger is in the touched state. Thus, although the first vector is illustrated, the magnitude of the second vector is approximately zero, and thus is not illustrated. In the tap recognition process, the tap flag is set to “ON”. Since the fingertip quickly moves in the xy direction at the moment of release and the fingertip xy moving speed at release is large, the flick flag is set to “ON” in the flick recognition process. Finally, in the tap or flick selection process, since the similarity between the first vector and the second vector is small, the flick is recognized.

  FIG. 6C illustrates a state in which a tap operation is input while the user is moving the arm in the xy plane direction without stopping the arm. In the tap recognition process, the tap flag is set to “ON”. In the flick recognition process, the flick flag is set to “ON” because the moving speed of the fingertip xy at the time of release is large. As shown in the figure, the first vector and the second vector are almost parallel, so the degree of similarity is high. Therefore, the tap is finally recognized in the tap or flick selection process. In a table top type device, the user often taps while pulling the arm from the operation plane in the direction of the body after touching, but in the present embodiment, such a tap can also be recognized robustly. However, in this embodiment, there is no restriction on the direction in which the arm is moved after the touch, and the tap can be recognized robustly regardless of the direction in which the arm is moved after the touch.

  FIG. 6D shows a state in which the user inputs a flick by moving only the fingertip at high speed while moving the entire arm. However, it is assumed that the movement in the xy plane direction is not performed in a short time when the finger is in the touched state. Thereby, the tap flag is set to “ON” in the tap recognition process. In the flick recognition process, the flick flag is set to “ON” because the fingertip xy moving speed at the time of release is large. As shown in the figure, since the fingertip moves at the time of flicking, the similarity between the first vector and the second vector is low, so that they are not regarded as similar. Therefore, in the final tap or flick selection process, the flick is recognized. In the present embodiment, the process for recognizing the touch operation according to the user operation input to the information processing apparatus 100 has been described above. As described above, in the present embodiment, when an arm is used as the operating body, a flick and a tap are performed according to the similarity between the movement of the fingertip used for touch input and the movement of the reference point representing the position of the arm itself. Improve recognition accuracy. For example, when the image acquisition interval of the distance image sensor is long and noise is included in the acquired image, it is difficult to distinguish operations with similar fingertip trajectories even using only fingertip information. In the example of FIG. 6, a flick operation that does not move the arm shown in FIG. 6B and a tap that moves the arm shown in FIG. 6C only by using the movement trajectory in the direction parallel to the touch target surface of the fingertip. It is difficult to distinguish operations. Therefore, in this embodiment, the recognition accuracy of the touch operation can be improved by considering the movement information of the reference point in addition to the fingertip. In the present embodiment, the touch operation is recognized on a rule basis, but a discriminator learned by machine learning may be used.

<Modification>
Here, as a modified example of the first embodiment, an example in which a determination process related to another index is added to the tap or flick selection process described above will be described. In the first embodiment, the determination is performed using only the component in the direction parallel to the touch target surface out of the first vector representing the movement of the designated point and the second vector representing the movement of the reference point. On the other hand, in the modification, three-dimensional (xyz) information is used for the movement of the operating tool.

  As shown in FIG. 11A referred to in the first embodiment, when the object to be moved by the touch operation is small or the distance to move the object is short, the flick is a finger rather than an arm. It is often input by moving a lot, that is, bending a finger joint. However, when flicking a relatively large display object, it may be more natural to flick by moving the entire arm without moving the finger joint. For example, as shown in FIG. 11B, a document that is largely displayed on the target surface may be paged by flicking. In FIG. 11B, as in FIG. 11A, the arm outlined by the solid line indicates the latest position of the user's arm, and the arm outlined by the dotted line traces the past ( This represents the position of the user's arm in the previous frame). In FIG. 11B, the first page is displayed in the initial state. Then, after the page is slid and moved halfway by the move, a flick is input to promptly display a state where the subsequent page is completed. In FIG. 11B, the flick is input by moving the entire arm largely without moving the finger joint. According to the modification, even when the flick is input by such an arm movement, the recognition accuracy of the tap and the flick can be improved by considering the movement of the indication point and the reference point in the z-axis direction. .

  In the following description, a flick accompanied by a movement that bends a finger joint is defined as “finger flick”, and a flick performed by moving an arm without a finger joint movement is defined as an “arm flick”. The schematic view of the arm flick as viewed from the distance image sensor is substantially the same as the tap for moving the arm in FIG. 6C, and the first vector and the second vector acquired by the same method as in the first embodiment. Among them, the component in the xy direction has a high similarity. However, the fine trajectory of the fingertip is different from the tap that moves the arm.

  The external appearance, hardware configuration, and functional configuration of the system in the modified example conform to those shown in FIG. 1 of the first embodiment. However, the operation recognizing unit 124 of the modified example executes a plurality of processes to be described later in the tap or flick selection process in the first embodiment.

  Similarly to the first embodiment, the information processing apparatus 100 according to the modification also executes the touch recognition process according to the flowchart of FIG. However, in step S209, the tap or flick recognition process shown in the flowchart of FIG. 9 is executed. The processing of the flowchart of FIG. 9 is different from the tap or flick selection processing (flowchart of FIG. 5) in the first embodiment in that the z component is used for movement of the fingertip until release. Hereinafter, the contents of the tap or flick recognition process in the modified example will be described with reference to the flowchart of FIG. 9, but the same step numbers are assigned to processes having the same contents as the flowchart of FIG. 5, and detailed description thereof is omitted. Then. In the case of the modification, when it is considered that the operation surface direction components of the first vector and the second vector are similar (YES in step S500), the process proceeds to step S600. If the operation surface direction components of the first vector and the second vector are not considered to be similar (NO in step S500), the process proceeds to step 501 and a flick is recognized, as in the first embodiment.

  In step S <b> 600, the operation recognition unit 124 compares the absolute value of the difference between the magnitude of the xy component and the magnitude of the z component of the first vector with a second threshold relating to the magnitude of the vector component. When it is determined that the absolute value of the difference between the magnitude of the xy component and the magnitude of the z component of the first vector is smaller than the second threshold (YES in step S600), the process proceeds to step S602. If it is not determined that the difference between the magnitudes of the xy component and the z component of the first vector is smaller than the second threshold value (NO in step S600), the process proceeds to step S601.

  The process of step S600 uses the magnitude relationship between the xy component (component parallel to the target surface) and the z component (component perpendicular to the target surface) of the first vector, so that the flick and tap can be distinguished only by the first vector. This is a process for determining whether it is possible. The concept of distinction between flicks and taps based on the first vector will be described later with reference to FIG. If it is possible to determine in step S600, the process proceeds to determination of tap and flick using the z component and xy component magnitudes of the fingertip movement vector in step S601. If it is determined that the flick and tap cannot be distinguished only by the first vector (NO in step S600), in step S602, the tap and flick are distinguished using the shape of the locus of the pointing point in the touch state. . The concept for using the shape of the locus of the pointing point in the touch state will be described later with reference to FIG.

  In step S601, the operation recognition unit 124 compares the absolute value of the z component and the absolute value of the xy component of the first vector. Details will be described later. If the xy component is larger, the flick recognition is confirmed, and the process proceeds to step S501. If the z component is larger, tap recognition is confirmed and the process proceeds to step S502.

  In step S <b> 602, the operation recognition unit 124 determines whether the shape of the post-touch movement trajectory of the fingertip is a shape that is convex on the side closer to the touch target surface (lower side in the case of the table top system of the present embodiment). If the post-touch movement trajectory is a shape that is convex toward the touch target surface, flick recognition is confirmed, and the process proceeds to step S501. If the movement locus after the touch is convex upward, the tap is recognized and the process proceeds to step S502.

  The details of the process of comparing the magnitude of the absolute value of the z component and the absolute value of the xy component of the first vector performed in step S600 and step S601 will be described using FIG. FIG. 7 is a diagram showing various touch operations from a direction horizontal to the operation plane. A dotted line indicates a touch threshold, and a broken line indicates a release threshold. The thin solid arrow indicates the trajectory of the fingertip until release. The trajectory of the fingertip is not actually observed continuously as shown in FIG. 7, but is acquired discretely according to the distance image acquisition interval. The black arrow indicates the z component of the first vector, and the white arrow indicates the xy component. 7A is a tap where the arm does not move, FIG. 7B is a finger flick where the arm does not move, FIG. 7C is a tap where the arm moves after touching, and FIG. 7D is a flick where the arm moves after touching. FIG. 7E shows an arm flick. In FIG. 7, when the user intentionally inputs a tap (A, C), the magnitudes of the xy component and the z component in the vector representing the movement from the lowest point in the touch state to the release. In many cases, the z component is larger than the xy component. This includes a tendency that the pointing point does not substantially move in the xy direction in the tap condition, so that the tendency of the user to raise the fingertip at a fast timing is reflected. On the other hand, when the user inputs a flicking intention (B, D, E), the xy component and the z component of the vector representing the movement from the lowest point in the touch state to the release. In many cases, there is a difference in the size of the xy component, and the xy component is larger than the z component. This is because the flick condition includes that the pointing point moves at high speed in the xy direction immediately before the release. Therefore, in the modification, as described above, the magnitudes of the xy component and the z component of the first vector are compared, and if there is a difference that can be determined that either one is greater, the z component is greater. If the tap and xy components are larger, it is determined as a flick. By processing in this way, in the case of FIG. 7, it is determined in FIG. 7A that there is a sufficient difference between the z component and the xy component of the first vector and that the z component is larger than the xy component. And the tap is recognized. In FIGS. 7D and 7E, there is a sufficient difference between the z component and the xy component, and it is determined that the xy component is larger than the z component, and the flick is recognized. In FIGS. 7B and 7C, since there is not a sufficient difference between the z component and the xy component of the first vector, it is considered that the tap and the flick cannot be distinguished from only the information of the first vector. become.

  Next, the unevenness determination of the post-touch locus performed in step S602 will be described in detail with reference to FIG. FIGS. 8A to 8E are diagrams showing the touch operation from a direction horizontal to the operation plane, as in FIGS. 7A to 7E. A dotted line indicates a touch threshold, and a broken line indicates a release threshold. FIG. 8 also shows “OFF” as in FIG. 7, FIG. 8A is a tap where the arm does not move, FIG. 8B is a finger flick where the arm does not move, and FIG. 8C is after touching. FIG. 8D shows a finger flick in which the arm moves after touching, and FIG. 8E shows a state of the arm flick.

  Open circles represent the observation points of the fingertips obtained discretely. The lowest point, the lowest point, the midpoint of the release position, and the release position are shown from the side closer to the target surface. The fan shape centered at the midpoint shows the angle formed by the three points.

  In FIG. 8, when the user inputs a flick intentionally (B, D, E), when the user reaches the release from the lowest point in the touch state, the fingertip first goes in a direction parallel to the touch target surface. It has risen after moving. As a result, the shape of the movement trajectory from the lowest point in the touch state to the release swells downward, which is the direction approaching the touch target surface. In the modified example, this state is called a convex on the side where the moving shape approaches the touch target surface. On the other hand, in the case where the user intends to input a tap (A, C), when the user reaches the release from the lowest point in the touch state, the user moves upward without moving in the direction parallel to the touch target surface. Priority is given to movement. As a result, the shape of the movement locus swells upward, which is the direction away from the touch target surface, contrary to the case of flicking. This phenomenon is likely to occur even when the xy component and the z component of the first vector have substantially the same size. In step S602, the case where the angle formed by the above three points is 180 ° or more is determined as a case where a convex is drawn downward, a case where the angle is smaller than 180 ° is a case where a convex is not drawn downward (convex upward). Then define. Accordingly, in the specific example shown in FIG. 8, it is determined that the projection is upward in FIGS. 8A and 8C, and the projection is determined to be downward in FIGS. D) and (E).

  The table of FIG. 10 summarizes how the touch operations of FIGS. 7 and 8 (A) to (E) are recognized and determined at each step of the flowcharts of FIGS. 3 and 9. . When the recognition / judgment can be made, ◯, when the judgment cannot be made, ×, when the judgment is not made,-is shown.

  In the tap (A) where the arm does not move, the tap flag is set to “ON” in step S202, and the flick flag is set to “OFF” in step S203. Since only the tap flag is “ON”, the tap is recognized without performing the tap or flick selection process.

  In the finger flick (B) where the arm does not move, the tap flag and the flick flag are set to “ON” in step S202 and step S203, and the tap or flick selection process in step S209 is performed. In the determination based on the movement vector of the fingertip and the reference point in Step S600, since the similarity between the first vector and the second vector is low as shown in FIG.

  In the tap (C) in which the arm moves after the touch, the tap flag and the flick flag are set to “ON” in steps S202 and S203, and the tap or flick selection process in step S209 is performed. First, in the determination based on the movement vector of the fingertip and the reference point in step S600, since the similarity between the first vector and the second vector is high as shown in FIG. 6C, the recognition is not confirmed and the process proceeds to step S600. . In step S600, as shown in FIG. 7C, since the z component and the xy component are substantially equal, the process proceeds to step S602. In step S602, as shown in FIG. 8C, since the angle formed by the three points is smaller than 180 degrees and is convex upward, tap recognition is confirmed.

  In the finger flick (D) in which the arm moves after the touch, the tap flag and the flick flag are set to “ON” in steps S202 and S203, and the tap or flick selection process in step S207 is performed. In the determination based on the movement vector of the fingertip and the reference point in Step S600, since the similarity between the first vector and the second vector is low as shown in FIG.

  In the arm flick (E), the tap flag and the flick flag are set to “ON” in step S202 and step S203, and the tap or flick selection process in step S209 is performed. First, in the determination based on the movement vector of the fingertip and the reference point in step S600, since the similarity between the first vector and the second vector is high as shown in FIG. 6C, the recognition is not confirmed and the process proceeds to step S600. . In step S600 and step S601, as shown in FIG. 7E, since the xy component is larger than the z component, the recognition of the flick is determined.

  The above is the process of recognizing the touch operation in accordance with the user operation input to the information processing apparatus 100 in the modification.

  According to the modification, even when the movement of the pointing point and the movement of the reference point are similar in the first embodiment, as in the case where the movement of the finger joint does not accompany the flick, the recognition accuracy of the flick and the tap is similar. Can be improved. In the modification, the touch operation is recognized on a rule basis, but a classifier learned by machine learning may be used.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 120 Image acquisition part 121 Operation body acquisition part 122 Instruction point detection part 123 Operation recognition part 124 Holding part

Claims (20)

Image acquisition means for acquiring an input image obtained by imaging a space in which a person inserts an arm and performs an operation;
Detecting means for detecting an indication point corresponding to a fingertip of the person's hand from an area in which the person's arm is captured as a subject in the input image;
Reference determining means for determining a position of a reference point serving as a reference for the position of the person's arm at a position different from the fingertip in an area of the input image in which the person's hand is captured as a subject;
When the touch state with respect to the predetermined surface included in the space is changed to the non-touch state, the fingertip is detected by the detection unit from a plurality of the input images acquired by the image acquisition unit during the same period. An operation recognition unit that determines whether a predetermined touch operation is input using the movement of the position corresponding to the fingertip and the movement of the position of the reference point determined by the reference determination unit;
An information processing apparatus comprising: The predetermined operation is a flick,
The operation recognizing means has a moving speed of the fingertip before the transition higher than a predetermined speed, and
When the movement of the position corresponding to the fingertip detected by the detection means in the same period and the movement of the position of the reference point determined by the reference determination means are regarded as not similar, it is determined that a flick has been input The information processing apparatus according to claim 1, wherein: The predetermined operation is a tap,
The operation recognition means, the length of time that the fingertip is in a touched state with respect to the predetermined surface before the transition and the movement distance of the fingertip while in the touched state satisfy a predetermined tap condition, And,
When the movement of the position corresponding to the fingertip detected by the detection means in the same period and the movement of the position of the reference point determined by the reference determination means are regarded as not similar, it is determined that a tap is input The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. The operation recognition means satisfies a predetermined tap condition based on a length of time that the fingertip is in a touched state with respect to the predetermined surface before the transition and a moving distance of the fingertip while in the touched state. If
Using the movement of the position corresponding to the fingertip detected by the detection means and the movement of the position of the reference point determined by the reference determination means from the plurality of input images acquired in the same period The information processing apparatus according to claim 1, wherein the information processing apparatus determines whether a tap is input or a flick is input as the predetermined operation. The predetermined tap condition is at least a condition that a moving distance of the fingertip is smaller than a predetermined distance threshold while the fingertip is in a touch state with respect to the predetermined surface before the transition. The information processing apparatus according to claim 3 or 4, characterized in that: The predetermined tap condition is that a length of time that the fingertip is in a touched state with respect to the predetermined surface before the transition is shorter than a predetermined time threshold, and
5. The condition according to claim 3, wherein the moving distance of the fingertip while the fingertip is in a touch state with respect to the predetermined surface before the transition is a condition that the moving distance of the fingertip is smaller than a predetermined distance threshold. The information processing apparatus described. The operation recognition means includes
When the movement of the position corresponding to the fingertip detected by the detection means in the same period and the movement of the position of the reference point determined by the reference determination means are regarded as not similar, it is determined that a tap has been input. ,
If it is considered that the movement of the position corresponding to the fingertip detected by the detection means in the same period and the movement of the position of the reference point determined by the reference determination means are not similar, it is determined that a flick has been input. The information processing apparatus according to claim 4. The operation recognizing means includes a first vector representing movement of the position corresponding to the fingertip detected by the detecting means in the same period, and a second vector representing movement of the position of the reference point determined by the reference determining means. The information processing apparatus according to claim 1, wherein the information processing apparatus determines whether the predetermined touch operation has been input using a vector similarity.   The information processing apparatus according to claim 8, wherein the first vector and the second vector include at least a component in a direction parallel to the predetermined plane.   The information processing apparatus according to claim 8, wherein the first vector and the second vector include a component in a direction parallel to the predetermined plane and a component in a direction perpendicular to the predetermined plane. . The operation recognizing means further taps as the predetermined operation based on a magnitude relationship between a component of the first vector in a direction parallel to the predetermined plane and a component in a direction perpendicular to the predetermined plane. The information processing apparatus according to claim 8, wherein it is determined whether or not a flick is input. The operation recognizing means determines that a flick is input when a component of the first vector in a direction parallel to the predetermined plane is larger than a component in a direction perpendicular to the predetermined plane. The information processing apparatus according to claim 11. The operation recognizing unit may further input a tap as the predetermined operation based on a shape of a movement locus of the fingertip from when the fingertip is closest to the predetermined surface until transition to the non-touch state. The information processing apparatus according to claim 8, wherein it is determined whether or not a flick has been input. The operation recognizing means, when the shape of the movement locus of the fingertip from when the fingertip is closest to the predetermined surface until transition to the non-touch state draws a convex in the direction approaching the predetermined surface, The information processing apparatus according to claim 13, wherein the information processing apparatus determines that a flick has been input. The reference determining means determines the center of the person's instep as the reference point by using information on an outline of a region of the input image where the person's arm appears as a subject. The information processing apparatus according to any one of 14. The operation recognition unit detects that the distance between the predetermined surface and the position of the fingertip detected by the detection unit exceeds a predetermined distance as a transition from the touch state to the non-touch state. The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. The information processing apparatus according to any one of claims 1 to 16, wherein the input image is a distance image captured by a distance image sensor installed at an angle of view overlooking the predetermined surface from above. . An image acquisition step of acquiring an input image obtained by imaging a space in which a person inserts an arm and performs an operation;
A detection step of detecting an indication point corresponding to a fingertip of the person's hand from a region in which the person's arm is captured as a subject in the acquired input image by a detection unit;
A reference determination step of determining a position of a reference point representing the position of the person's hand at a position different from the fingertip in a region where the person's hand is captured as a subject in the acquired input image by the reference determination means. When,
When the fingertip makes a transition to a non-touch state on a predetermined surface included in the space, the fingertip detected from the plurality of input images acquired in the same period by the operation recognition unit An operation recognition step of determining whether a predetermined touch operation is input using the movement of the corresponding position and the movement of the position of the reference point;
A method for controlling an information processing apparatus, comprising: Computer
Image acquisition means for acquiring an input image obtained by imaging a space in which a person inserts an arm and performs an operation;
Detecting means for detecting an indication point corresponding to a fingertip of the person's hand from an area in which the person's arm is captured as a subject in the input image;
Reference determining means for determining a position of a reference point representing the position of the person's hand at a position different from the fingertip in a region of the input image in which the person's hand appears as a subject;
When the touch state with respect to the predetermined surface included in the space is changed to the non-touch state, the fingertip is detected by the detection unit from a plurality of the input images acquired by the image acquisition unit during the same period. Operation recognition means for determining whether or not a predetermined touch operation has been input using the movement of the position corresponding to the fingertip and the movement of the position of the reference point determined by the reference determination means. A program that functions as a processing device.   A computer-readable storage medium storing the program according to claim 19.

Images