JP2019016029A - Information processing device, and control method and program thereof - Google Patents

Abstract

To provide a mechanism capable fo appropriately recognizing a tapping operation of a user.SOLUTION: An information processing device recognizes operation of an operation body to a displayed button 701 displayed on an operation plane. The information processing device determines a tap operation in which the operation body approaches to a position closer to the operation plane than a first distance in a height direction and then moves away to a position more distant than a second distance on the basis of a plurality of operation positions P1 to P6 of the operation body, and recognizes the tap operation to the displayed button 701 by the operation body, when a two dimensional operation trajectory on the operation plane determined by the operation positions P1 to P6 of the tapping operation is included in the area of the displayed button 701.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an information processing apparatus for recognizing an operation of an operating body on a display object displayed on an operation surface, a control method thereof, and a program for causing a computer to function as the information processing apparatus.

  In recent years, device operation using gesture recognition technology for operating a user interface (UI) in accordance with the movement and position of an operation body such as a human hand or a pen detected using various cameras and sensors is becoming widespread. For example, in a table top interface in which an image or UI is projected on a table surface, which is an operation surface, and the image or UI is touched and operated by an operating body such as a hand or a pen, an operation of a fingertip or a pen tip is performed without using a touch panel. Techniques for detecting the state where the end of the body is touched on the table are beginning to be used.

  Patent Document 1 describes a technique for operating a UI screen displayed on an operation surface by a tap operation that is a series of operations in which a user brings a fingertip into contact with or close to the operation surface and then moves the fingertip away from the operation surface. ing. In the technique of this patent document 1, the position at which the fingertip movement is regarded as stationary in the tap operation (the position at which the fingertip movement is slowest) is set as the operation position of the tap operation. In Patent Document 1, it is assumed that the movement of the fingertip is the slowest when the fingertip is closest to the operation surface as a reason for determining the operation position of the tap operation. That is, in Patent Document 1, it is assumed that the UI screen displayed on the operation surface can be operated by the position where the fingertip is closest to the operation surface, and comfortable operability can be provided to the user.

JP2015-43154A

  As described above, in Patent Document 1, it is assumed that the UI screen displayed on the operation surface is operated (tap operation) based on the position where the fingertip of the hand as the operation body is closest to the operation surface. there were. However, with the technique of Patent Document 1, when measuring the distance between the fingertip of the hand, which is the operating body, and the operation surface, the position where the fingertip is closest to the operation surface cannot be accurately measured due to the influence of measurement noise. There was a case. In this case, there may be a problem that it is difficult to properly recognize the user's tap operation.

  The present invention has been made in view of such problems, and an object of the present invention is to provide a mechanism that can properly recognize a user's tap operation.

An information processing apparatus according to the present invention is an information processing apparatus that recognizes an operation of an operating body on a display object displayed on an operation surface, and is acquired by an acquisition unit that acquires an operation position of the operating body, and the acquisition unit. Determination means for determining a series of operations that are further away from the second distance after the operation body is closer than the first distance in the height direction to the operation surface based on the plurality of operation positions; Recognizing means for recognizing an operation of the operating body on the display object when a two-dimensional operation trajectory on the operation surface determined by the plurality of operation positions related to the series of operations is included in the area of the display object; Have.
The present invention also includes a method for controlling the information processing apparatus described above and a program for causing a computer to function as each unit in the information processing apparatus described above.

  According to the present invention, it is possible to properly recognize a user's tap operation.

It is a figure which shows an example of a schematic structure of a tabletop interface system containing the information processing apparatus which concerns on embodiment of this invention, and a definition of position information. It is a figure which shows an example of the internal structure of the information processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows an example of the process sequence in the control method of the information processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows an example of the detailed process sequence in step S307 of FIG. It is a flowchart which shows an example of the detailed process sequence in FIG.3 S308. It is a flowchart which shows an example of the detailed process sequence in step S309 of FIG. FIG. 4 is a diagram illustrating an example of a display button in a UI screen projected on an operation surface (xy plane) illustrated in FIG. 1 and an example of a tap operation on the display button according to the embodiment of the present invention. FIG. 2 is a diagram illustrating an example of each operation position in a z-direction space above the operation surface illustrated in FIG. 1 according to the embodiment of the present invention.

  Hereinafter, embodiments (embodiments) for carrying out the present invention will be described with reference to the drawings. In addition, embodiment described below illustrates one form for implementing this invention, and this invention is not limited to this embodiment.

  FIG. 1 is a diagram illustrating an example of a schematic configuration of a tabletop interface system 100 including an information processing apparatus 110 according to an embodiment of the present invention and definition of position information.

First, FIG. 1A will be described.
FIG. 1A is a diagram illustrating an example of a schematic configuration of the table top interface system 100. As shown in FIG. 1A, the table top interface system 100 includes an information processing apparatus 110, a distance image sensor 120, a visible light camera 130, a projector 140, and an operation surface 150.

  The operation surface 150 corresponds to a table surface in the table top interface system 100. The user can input a touch operation by touching the operation surface 150 with the fingertip of the hand H. However, in the present embodiment, no touch sensor is mounted on the operation surface 150. For this reason, the information processing apparatus 110 does not determine whether an operation body such as a finger or a pen operated by the user has actually contacted the operation surface 150, but a state close enough to be regarded as being in contact (hereinafter referred to as a touch operation input state). , Referred to as “touch state”), the operation by the operating body is recognized.

  In the table top interface system 100 shown in FIG. 1A, the distance image sensor 120 is installed above the outer peripheral portion of the operation surface 150 so that the operation surface 150 is looked down obliquely from above. Here, the distance image reflects information corresponding to the distance from the reference position (for example, the center of the lens) of the imaging unit that captures the distance image to the subject surface captured by the pixel in the value of each pixel. It is an image that was made. In the present embodiment, the pixel value of the distance image captured by the distance image sensor 120 reflects the distance from the distance image sensor 120 to the operation surface 150 or the object surface existing above it. The distance image captured by the distance image sensor 120 is input to the information processing apparatus 110 as a distance image.

  In the present embodiment, as the distance image sensor 120, a distance image sensor that acquires distance information based on the reflection time (or reflection pattern) of infrared light is used. However, for example, it is possible to obtain a distance image by installing a stereo camera system or an infrared light emitting element and an infrared light receiving element. Further, in the space including the operation surface 150, as long as the three-dimensional position information including the height direction (z direction) of the operation body is obtained, the present invention is not limited to the mode of capturing a distance image, for example, an electrostatic sensor, A form in which three-dimensional position information is obtained by a pressure sensor or a temperature sensor is also applicable to the present invention. Note that the operation surface 150 may be configured by a touch panel display.

  In this embodiment, the information processing apparatus 110 acquires a three-dimensional position of the user's hand H, the operation surface 150, and the like by analyzing the distance image acquired from the distance image sensor 120, and recognizes the input operation. Perform the process. Therefore, the user moves the operating body such as the hand H within the range that can be imaged by the distance image sensor 120 in the space on the operation surface 150 (the space between the operation surface 150 and the distance image sensor 120). It is possible to input a space gesture operation. Similarly, the user can perform an operation input based on the proximity state between the operation body 150 such as the hand H and the operation surface 150.

  In the table top interface system 100 shown in FIG. 1A, the visible light camera 130 is installed so that the operation surface 150 is looked down obliquely from above. The information processing apparatus 110 can function as a document camera that controls the visible light camera 130 to capture an image of an object placed on the operation surface 150 and obtain a read image thereof. Therefore, the information processing apparatus 110 detects an object existing in the space on the operation surface 150 based on the visible light image obtained by the visible light camera 130 or the distance image obtained by the distance image sensor 120, and further identifies the object. Do. Here, the detected object includes, for example, a user's hand, a document such as a paper medium or a book, and other three-dimensional objects. However, in the case of the table top interface system 100 shown in FIG. 1A, the angle of view of the distance image sensor 120 and the visible light camera 130 does not include the user's head existing around the table. Therefore, in the obtained distance image, the end of the image intersects some part of the user's arm (portion beyond the shoulder).

  Further, in the table top interface system 100 shown in FIG. 1A, a projector 140 that projects the UI screen 152 obliquely from above the operation surface 150 is installed. In the table top interface system 100 illustrated in FIG. 1A, the user performs an operation input by touch or space gesture on a display object (for example, a display button) included in the projected UI screen 152.

  As described above, in this embodiment, the distance image acquired by the distance image sensor 120 is used for detecting the hand H and recognizing the operation. By using this distance image, there is an advantage that even if the color of the user's hand H changes due to the projection light of the projector 140, the distance image is hardly affected. The display device of the table top interface system 100 may be configured by using the operation surface 150 as a liquid crystal display instead of the projector 140. In this case, even if a form in which a human hand H is detected from an image by detecting a skin color region from a visible light image obtained by the visible light camera 130, the hand H is detected without being affected by the projection light. Is possible.

  Note that the distance image sensor 120 and the visible light camera 130 themselves do not necessarily have to be installed above as long as an image obtained by viewing the operation surface 150 from above is obtained. For example, the reflected light is reflected using a mirror. You may comprise so that it may image. Similarly, in the example shown in FIG. 1A, the projector 140 projects the UI screen 152 onto the operation surface 150 so as to look down from the oblique upper side of the operation surface 150. However, the projector 140 is projected in a different direction. The UI screen 152 may be projected by reflecting the projection light on the operation surface 150 using a mirror or the like. The same applies to the case where the operation surface 150 is installed on a surface along a vertical direction such as a wall surface or a whiteboard.

  Further, in the present embodiment, the coordinate axes of the x axis, the y axis, and the z axis shown in FIG. 1A are defined in the three-dimensional space on the operation surface 150, and position information based on the origin 151 is handled. To do. Specifically, in the example shown in FIG. 1A, a two-dimensional surface parallel to the operation surface 150 is an xy plane, and a direction (normal direction) perpendicular to the operation surface 150 and extending upward is the positive direction of the z axis. It is said. At this time, the z-axis direction corresponds to the height direction in the real world coordinate system. However, in the present embodiment, when a non-horizontal surface such as a whiteboard or a wall surface is used as the operation surface 150, the operation surface 150 is uneven, or a virtual surface generated using MR. Is also applicable.

Subsequently, FIG. 1B will be described.
FIG. 1B shows an example of a distance image 160 captured by the distance image sensor 120 shown in FIG. 1A and is a diagram for explaining the definition of position information.
The distance image 160 in the present embodiment reflects the distance to the subject surface corresponding to the time taken for reflection for each pixel by measuring the phase delay of the received reflected light. It is. However, here, the distance information reflected in the pixel value is omitted, and only the edge of the subject is specified.

  As described above, the distance image sensor 120 captures the distance image 160 so that the operation surface 150 is looked down obliquely from above. Therefore, the distance information represented by the pixel value of the distance image 160 indicates the operation surface 150 from the distance image sensor 120. This is the distance in the depth direction when looking down. In other words, each pixel value of the distance image 160 includes information for obtaining position information (z coordinate) in the height direction from the operation surface 150. However, in the present embodiment, the distance image sensor 120 is installed so as to have an angle with respect to the z-axis. Therefore, in the present embodiment, the pixel value of the distance image 160 is not used as it is as the z coordinate, but is subjected to coordinate conversion using parameters according to the sensor and the installation environment, as will be described later.

  In the distance image 160 shown in FIG. 1B, portions corresponding to the operation surface 150 and the origin 151 shown in FIG. In this distance image 160, as shown in FIG. 1B, a two-dimensional coordinate system based on the u axis and the v axis is set. Further, in the distance image 160 shown in FIG. 1B, a hand region 161 indicated by diagonal lines is an image of the user's hand H shown in the distance image 160. The image end 163 corresponds to the angle of view of the distance image sensor 120. In the example of FIG. 1B, it is assumed that the resolution of the distance image 160 is 640 [dot] × 480 [dot]. Further, in the distance image 160 shown in FIG. 1B, the position coordinates in the distance image of the operation position 162 that is the fingertip of the user's hand H is (u, v), and the distance image sensor 120 to the operation position 162. Let d be the pixel value corresponding to the distance.

  The information processing apparatus 110 according to the present embodiment performs coordinate conversion on the position information defined in the distance image 160 as described above based on the lens characteristics of the distance image sensor 120 and the relative positional relationship with the operation surface 150. . As a result, the coordinates of each pixel can be mapped to a coordinate system in the real world defined in the table, and the three-dimensional position (x, y, z) in the real space can be acquired for the operation position 162. For example, when the distance image sensor 120 is installed, the transformation matrix used for the coordinate transformation is acquired by performing adjustment work in advance. Note that the distance d at the operation position 162 used for calculating the three-dimensional coordinates is not limited to a single pixel value, but is averaged for a plurality of pixels near the operation position 162 in the hand region 161. .

  In addition, the information processing apparatus 110 according to the present embodiment detects a region in which a subject existing at a position higher than the operation surface 150 that is the surface of the table as a hand region 161 by performing threshold processing on the z coordinate. The information processing apparatus 110 according to the present embodiment detects one operation position 162 for each detected hand region 161. In the present embodiment, the operation position 162 is a coordinate of a position estimated to be pointed by the user using a fingertip. In the present embodiment, as a premise, it is defined that when the user designates a point for operation input, a “pointing pose” in which only one finger of the hand H is extended is taken. This is because the pose of extending one finger is considered to be a natural posture for many people to point to one point. Since the end portion of the hand H corresponds to the operation position 162 in the pointing pose, in the present embodiment, the end portion of the hand region 161 is specified as the operation position 162. Specifically, the information processing apparatus 110 detects the hand region 161 from the distance image 160, and the coordinates indicating the pixel existing at the position farthest from the image end 163 in the detected hand region 161 correspond to the operation position 162. This is detected as one point. Note that the method of specifying the operation position 162 is not limited to this, and for example, the five fingers of the hand H may be detected from the hand region 161 and the end portion of the predetermined finger may be specified as the operation position 162.

  In the present embodiment, the center position of the portion where the hand region 161 intersects the image end 163 in each frame of the distance image 160 repeatedly captured according to the frame rate is defined as the entry position 164 of the hand H. This intrusion position 164 is also acquired as three-dimensional position information in the real space by converting the position information defined in the distance image 160 in the same manner as other positions. In addition, when a plurality of hand regions 161 are detected from the distance image 160, the information processing apparatus 110 according to the present embodiment detects the operation position 162 and the intrusion position 164 for each hand region 161. Then, the information processing apparatus 110 according to the present embodiment recognizes an operation input for each of the plurality of hand regions 161.

  Hereinafter, in the present specification, a case where the user's hand 106 and its fingertip are applied as an example of the operation body used by the user for operation input and the end thereof will be described. However, in the present embodiment, the present invention can be applied not only to the fingers as the operating body, but also when using an instrument such as a stylus or a robot arm. Note that the end of the operating body refers to a part used to point to the operation position 162 for operation input. However, if the part belongs to a part of the operating body and allows operation input, it has a protruding shape. It is not limited to the end.

Next, the internal configuration of the information processing apparatus 110 will be described.
FIG. 2 is a diagram illustrating an example of an internal configuration of the information processing apparatus 110 according to the embodiment of the present invention. In FIG. 2, the same components as those shown in FIG.

First, FIG. 2A will be described.
FIG. 2A is a diagram illustrating an example of an internal hardware configuration of the information processing apparatus 110. As illustrated in FIG. 2A, the information processing apparatus 110 has hardware configurations of a CPU 211, a ROM 212, a RAM 213, a storage device 214, and a system bus 215 that are central processing units.

  The CPU 211 uses the RAM 213 as a work memory, executes an OS and a program stored in the ROM 212 and the storage device 214, and performs various processing operations and logical determinations. Further, the CPU 211 controls each component connected to the system bus 215. In addition, the CPU 211 holds the execution result of each process in the RAM 213.

  The storage device 214 is an external storage device or the like connected by a hard disk drive or various interfaces, and stores a program and various data related to the operation recognition process of the present embodiment.

  In addition, the distance image sensor 120 captures and captures a distance image of the space on the operation surface 150 including the table on which the item (display object) is displayed and the user's hand H that operates the item according to the control of the CPU 211. The distance image is output to the system bus 215. In this embodiment, the distance image acquisition method will be described based on a reflection time method (Time-of-Flight method) that is less affected by ambient light and the display on the operation surface 150. It is also possible to use an infrared pattern method or the like. Further, the projector 140 projects and displays an item (display object) to be operated on the operation surface 150 according to the control of the CPU 211. The visible light camera 130, the distance image sensor 120, and the projector 140 are external devices connected to the information processing apparatus 110 via an input / output interface, respectively. The system 100 is configured. However, these devices may be integrated with the information processing apparatus 110. The distance image sensor 120 may be an electrostatic sensor, a pressure sensor, a temperature sensor, or the like, and acquires three-dimensional position information of the operation position 162 and contact information such as a contact pressure or a contact area of the operation position 162.

Subsequently, FIG. 2B will be described.
FIG. 2B is a diagram illustrating an example of a software configuration (functional configuration) inside the information processing apparatus 110. As illustrated in FIG. 2B, the information processing apparatus 110 includes a distance image acquisition unit 221, an operation position acquisition unit 222, an operation determination unit 223, an operation locus holding unit 224, a recognition unit 225, and a display control unit 226. Each software configuration (functional configuration) is included. Each of the components 221 to 226 shown in FIG. 2B is realized by the CPU 211 shown in FIG. 2A developing, for example, a program stored in the ROM 212 on the RAM 213 and executing the program. When configuring hardware as an alternative to the components 221 to 226 shown in FIG. 2B realized by the CPU 211 executing the program, the components 221 shown in FIG. 2B. A calculation unit or a circuit corresponding to ˜226 may be configured.

  The distance image acquisition unit 221 acquires the distance image 160 captured by the distance image sensor 120 from the distance image sensor 120 at regular intervals according to the frame rate, and holds this. Note that the object acquired by the distance image acquisition unit 221 from the distance image sensor 120 is actually a signal corresponding to the distance image data, but will be described as simply “acquire a distance image” in this specification.

  The operation position acquisition unit 222 performs threshold determination based on distance information for each pixel of the distance image 160 acquired by the distance image acquisition unit 221 and detects a hand region (including an arm) 161 in the distance image. Here, the hand region 161 is a pixel group in which the hand H that the user uses as the operating tool in the input distance image 160 is shown as a subject. And the operation position acquisition part 222 provides hand ID to the detected hand area | region 161, and hold | maintains this. Here, the operation position acquisition unit 222 assigns the same hand ID when the hand regions detected in the frames of different distance images are regarded as the same hand region 161. A condition that is regarded as the same hand region 161 is that it is detected in a frame of a distance image that is close in time and that the intrusion position 164 of the hand region 161 is short. On the other hand, if the operation position acquisition unit 222 is not regarded as the same hand region 161, the operation position acquisition unit 222 assigns different numbers in ascending order to the order in which the hand region 161 is detected (hand ID = 1, 2,...). Further, the operation position acquisition unit 222 detects and acquires one point corresponding to the end of the hand region 161 as the operation position 162 based on the detected contour information of the hand region 161. In the present embodiment, the operation position acquisition unit 222 detects and acquires, as the operation position 162, coordinates indicating a pixel that is farthest from the image end 163 in the contour of the hand region 161. At this time, the operation position acquisition unit 222 sets the center of the portion where the hand region 161 and the image edge 163 intersect as the intrusion position 164 of the hand region, and operates the coordinates indicating the pixel existing farthest from the intrusion position 164. The position 162 is detected and acquired. Then, the operation position acquisition unit 222 converts the position information of the detected operation position 162 in the distance image 160 into xyz coordinates in the real world, and assigns and holds the same hand ID as the hand region 161. In the present embodiment, three-dimensional position information (coordinate values) and contact information (contact pressure, contact area, etc.) of the fingertip position close to the operation surface 150 obtained by an electrostatic sensor, a pressure sensor, or the like. May be acquired and held as position information of the operation position 162.

  The operation determination unit 223 determines that the operation position 162 is in a touch state in accordance with the degree of proximity of the operation position 162 acquired by the operation position acquisition unit 222 to the operation surface 150 (the degree of proximity in the z direction that is the height direction). It is determined whether or not there is, and the determination result is held. For example, when the distance (z coordinate corresponding to the height) between the operation position 162 and the operation surface 150 is smaller than a predetermined threshold (first distance), it is assumed that the touch state is a touch input. judge. Then, the operation determination unit 223 determines a touch operation such as touch, release, move, flick, pinch, and rotate by tracking the position of the operation surface 150 indicated by the operation position 162 in the touch state. The position on the operation surface 150 instructed by the operation position 162 is two-dimensional position information (xy coordinates) parallel to the operation surface 150 based on the operation position 162.

  The operation locus holding unit 224 holds the position information and the acquisition time of the operation position 162 from the start to the end of the touch state determined by the operation determination unit 223 as the operation locus of the touch operation. Specifically, in the present embodiment, the operation trajectory holding unit 224 indicates that the fingertip of the hand H, which is the operating body, is closer to the operation surface 150 than the first distance in the height direction as the operation trajectory of the touch operation. After that, the operation locus of the tap operation, which is a series of operations away from the second distance, is held. In addition, the operation locus of the tap operation held in the operation locus holding unit 224 is a two-dimensional operation locus on the operation surface 150 determined by the plurality of operation positions 162 related to the tap operation.

  The recognizing unit 225 performs the operation on the display object according to the positional relationship between the operation locus of the touch operation held by the operation locus holding unit 224 and the display object such as the display button in the UI screen 152 projected on the operation surface 150. It recognizes the operation of the fingertip of the hand H, which is the operating body. Specifically, in the present embodiment, the recognizing unit 225 determines that the operation locus of the tap operation held by the operation locus holding unit 224 is included in the area of a display object such as a display button projected on the operation surface 150. Then, the operation of the fingertip of the hand H, which is the operation body for the display object, is recognized.

  The display control unit 226 generates drawing data reflecting the recognition result (touch event or the like) of the touch operation notified from the recognition unit 225, and outputs the data to the projector 140, thereby displaying the UI screen on the operation surface 150. The display is controlled. At this time, the touch operation by the user is defined by the operation position 162, the movement of the operation position 162, and the like, and any instruction can be input to the information processing apparatus 110.

  FIG. 3 is a flowchart illustrating an example of a processing procedure in the control method of the information processing apparatus 110 according to the embodiment of the present invention. Hereinafter, the flowchart is realized by the CPU 211 executing a control program.

  When the power of the information processing apparatus 110 is turned on, first, in step S301 of FIG. 3, the display control unit 226 displays the UI screen 152 including display objects such as display buttons based on data acquired from the storage device 214, for example. It is generated and output to the projector 140. The projector 140 projects the acquired UI screen 152 on the operation surface 150 based on the control of the display control unit 226.

  Subsequently, in step S <b> 302, the distance image acquisition unit 221 acquires a distance image from the distance image sensor 120. The distance image sensor 120 captures a distance image 160 in a space between the operation surface 150 and the distance image sensor 120 at a predetermined time interval while the information processing apparatus 110 is turned on.

  Subsequently, in step S303, the operation position acquisition unit 222 performs detection processing of the user's hand region (including arms) 161 on the distance image acquired in step S302, and determines whether or not the hand region 161 has been detected. Judging. In this step S303, when the operation position acquisition unit 222 determines that the hand region 161 has not been detected (S303 / No), the process returns to step S302, and the processes after step S302 are performed again.

On the other hand, when the operation position acquisition unit 222 determines in step S303 that the hand region 161 has been detected (S303 / Yes), the process proceeds to step S304.
In step S304, the operation position acquisition unit 222 assigns a hand ID to the hand region 161 detected in step S303 and holds it. Here, for example, for different hand regions 161, different numbers are assigned to the detected order in ascending order as hand IDs (hand ID = 1, 2,...).

  Subsequently, in step S305, the operation position acquisition unit 222 acquires the operation position (fingertip position) 162 in the distance image of the hand region 161 detected in step S303. Then, the operation position acquisition unit 222 assigns the same hand ID as the hand region 161 including the operation position 162 to the operation position 162 acquired here, and holds this. Furthermore, the operation position acquisition unit 222 holds the time when the distance image 160 from which the operation position 162 is acquired is acquired (or captured) as the time at which the operation position 162 is acquired.

  Subsequently, in step S306, the operation position acquisition unit 222 acquires the position information in the real world coordinates of the operation position 162 acquired in step S305 as the operation position 162 in the real world.

  Subsequently, in step S307, the operation determination unit 223 determines the touch state based on the operation position 162 acquired in step S306 based on the operation position 162 acquired in step S305. Then, the operation track holding unit 224 acquires the position information and the acquisition time of the operation position 162 acquired in step S306 from the start to the end of the touch operation as the operation track of the touch operation. More specifically, in the present embodiment, the operation trajectory holding unit 224 determines the touch trajectory as the operation trajectory after the fingertip of the hand H, which is the operating body, approaches the operation surface 150 from the first distance. The operation locus of the tap operation that is a series of operations away from the distance of 2 is held. The detailed processing procedure of step S307 will be described later with reference to FIG.

  Subsequently, in step S308, the operation determination unit 223 determines the touch operation based on the operation position 162 acquired in step S306 based on the operation position 162 acquired in step S305. The detailed processing procedure of step S308 will be described later with reference to FIG.

  Subsequently, in step S309, the recognizing unit 225 displays the operation locus held in step S307 and the display object in the UI screen 152 projected in step S301 based on the determination result of the touch operation in step S308. The operation of the display button is determined according to the positional relationship with the button. The detailed processing procedure of step S309 will be described later with reference to FIG.

  Subsequently, in step S310, the display control unit 226 performs control to update the display of the UI screen 152 currently displayed according to the determination result of the display button operation in step S309. Specifically, when the display control unit 226 determines (recognizes) that the display button has been operated in step S309, the display control unit 226 performs control to change the display of the display button from the display state before the recognition. For example, when the display control unit 226 determines (recognizes) that the display button has been operated in step S309, the display control unit 226 can update the display so that the line of the frame of the display button becomes thick and blinks (for example, FIG. 7 (a) 701, FIG. 7 (b) 702, FIG. 7 (c) 704). The display button display may be updated by changing the color or brightness, turning off the display, or the like.

  Subsequently, in step S311, the information processing apparatus 110 determines whether or not the power is turned off. In this step S311, when the information processing apparatus 110 determines that the power is not turned off (S311 / No), the process returns to step S302, and the processes after step S302 are performed again.

  On the other hand, when the information processing apparatus 110 determines in step S311 that the power has been turned off (S311 / Yes), the process of the flowchart illustrated in FIG. 3 ends.

  FIG. 4 is a flowchart showing an example of a detailed processing procedure in step S307 of FIG.

  First, in step S401 in FIG. 4, the operation determination unit 223 determines that the operation position 162 acquired in step S306 is based on the operation position 162 acquired in step S305 and the touch threshold value corresponding to the first distance (FIG. 8). 801) It is determined whether or not.

If the operation determination unit 223 determines in step S401 that the operation position 162 is equal to or smaller than the touch threshold (S401 / Yes), the process proceeds to step S402.
In step S402, the operation determination unit 223 determines whether or not the previous operation position 162 is in a touch state. If the operation position 162 to be determined in step S401 is the first operation position 162, it is determined in step S402 that it is not in the touch state (that is, in the release state).

In step S402, when the operation determination unit 223 determines that the previous operation position 162 is in a touch state (S402 / Yes), the process proceeds to step S403.
In step S403, the operation determination unit 223 (or operation locus holding unit 224) holds that the operation position 162 to be determined in step S401 is in a touch state.

  Subsequently, in step S404, the operation track holding unit 224 uses the information related to the operation position 162 to be determined in step S401 as information related to the operation position in the operation track of the touch operation (more specifically, the tap operation). Keep holding. When the process of step S404 is completed, the process of the flowchart shown in FIG. 4 is terminated, and the process proceeds to step S308 of FIG.

If the operation determination unit 223 determines in step S402 that the previous operation position 162 is not in the touched state (that is, in the released state) (S402 / No), the process proceeds to step S405.
In step S405, the operation determination unit 223 (or operation locus holding unit 224) holds that the operation position 162 to be determined in step S401 is in a touch state. Further, the operation determination unit 223 (or the operation locus holding unit 224) determines that the previous operation position is in the released state when the operation position 162 to be determined in step S401 is the first operation position 162. Hold. Thereby, a process switches from a release state to a touch state.

  Subsequently, in step S406, the operation locus holding unit 224 relates the information related to the operation position 162 to be determined in step S401 to the start operation position in the operation locus of the touch operation (more specifically, the tap operation). Hold as information. When the process of step S406 is completed, the process of the flowchart shown in FIG. 4 is ended, and the process proceeds to step S308 of FIG.

If the operation determination unit 223 determines in step S401 that the operation position 162 is not less than or equal to the touch threshold (S401 / No), the process proceeds to step S407.
In step S407, the operation determination unit 223 determines whether or not the previous operation position 162 is in a touch state. In this step S407, when the operation determination unit 223 determines that the previous operation position 162 is not in the touched state (that is, in the released state) (S407 / No), the processing of the flowchart shown in FIG. The process ends, and the process proceeds to step S308 in FIG.

On the other hand, when the operation determination unit 223 determines in step S407 that the previous operation position 162 is in a touch state (S407 / Yes), the process proceeds to step S408.
In step S408, the operation determination unit 223 determines that the operation position 162 acquired in step S306 based on the operation position 162 acquired in step S305 is a release threshold (802 shown in FIG. 8) corresponding to the second distance. It is determined whether this is the case.

In step S408, if the operation determination unit 223 determines that the operation position 162 is equal to or greater than the release threshold (S408 / Yes), the process proceeds to step S409.
In step S409, the operation determination unit 223 (or operation locus holding unit 224) holds that the operation position 162 to be determined in step S401 is in the released state. Thereby, a process switches from a touch state to a release state.

  Subsequently, in step S410, the operation locus holding unit 224 uses the information related to the operation position 162 to be determined in step S401 as the end (last) operation in the operation locus of the touch operation (more specifically, the tap operation). Stored as information related to the position. Thereafter, the operation trajectory holding unit 224 ends the holding of information related to the operation position in the operation trajectory. When the process of step S406 is completed, the process of the flowchart shown in FIG. 4 is ended, and the process proceeds to step S308 of FIG.

If the operation determination unit 223 determines in step S408 that the operation position 162 is not equal to or greater than the release threshold (No in S408), the process proceeds to step S411.
In step S411, the operation determination unit 223 (or operation locus holding unit 224) holds that the operation position 162 to be determined in step S401 is in a touch state.

  Subsequently, in step S412, the operation locus holding unit 224 uses the information related to the operation position 162 to be determined in step S401 as information related to the operation position in the operation locus of the touch operation (more specifically, the tap operation). Keep holding. Then, when the process of step S412 is finished, the process of the flowchart shown in FIG. 4 is finished, and the process proceeds to step S308 of FIG.

  FIG. 5 is a flowchart showing an example of a detailed processing procedure in step S308 of FIG.

  First, in step S501 in FIG. 5, the operation determination unit 223 refers to the touch state held in step S307, and determines whether the current operation position 162 is in the touch state and the previous operation position 162 is in the release state. Judge whether or not.

In step S501, if the current operation position 162 is in the touch state and the previous operation position 162 is in the release state (S501 / Yes), the operation determination unit 223 proceeds to step S502.
In step S502, the operation determination unit 223 notifies the recognition unit 225 of a touch event. When the process of step S502 is completed, the process of the flowchart shown in FIG. 5 is terminated, and the process proceeds to step S309 of FIG.

On the other hand, in step S501, if at least the current operation position 162 is not in the touched state or the previous operation position 162 is not in the released state (S501 / No), the operation determining unit 223 proceeds to step S503.
In step S503, the operation determination unit 223 refers to the touch state held in step S307 and determines whether or not the current operation position 162 is in the release state and the previous operation position 162 is in the touch state. to decide. In this step S503, when the operation determination unit 223 is at least the current operation position 162 is not in the release state or the previous operation position 162 is not in the touch state (S503 / No), the processing of the flowchart shown in FIG. And the process proceeds to step S309 in FIG.

On the other hand, in step S503, if the current operation position 162 is in the released state and the previous operation position 162 is in the touch state, the process proceeds to step S504.
In step S504, the operation determination unit 223 notifies the recognition unit 225 of a release event.

  Subsequently, in step S505, the operation determination unit 223 corresponds to the second distance after the fingertip of the hand H, which is the operation body, approaches the operation surface 150 from a touch threshold corresponding to the first distance. The tap operation, which is a series of operations away from the release threshold, is determined. In addition, although the example which uses the distance with respect to the operation surface 150 of the fingertip of the hand H, which is the operation body, as the determination criterion is shown here as the determination criterion for the tap operation, in the present embodiment, it is not limited to this. There may be provided a further criterion. For example, the criterion for determining the tap operation described above is that the distance on the operation surface 150 between the operation position 162 that first approaches the touch threshold and the operation position 162 that first approaches the release threshold is equal to or less than a predetermined threshold. The aspects included in the above are also applicable to this embodiment. Further, the above-described criterion for determining the tap operation is a condition in which the time from the time at the operation position 162 that first approaches the touch threshold to the time at the operation position 162 that is first far from the release threshold is equal to or less than a predetermined threshold. The aspects included in the above are also applicable to this embodiment. Thereafter, the operation determination unit 223 notifies the recognition unit 225 of the tap event related to the tap operation and the tap operation position. When the process of step S505 ends, the process of the flowchart shown in FIG. 5 ends, and the process proceeds to step S309 of FIG.

  FIG. 6 is a flowchart showing an example of a detailed processing procedure in step S309 of FIG.

  First, in step S601 in FIG. 6, the recognition unit 225 determines whether or not the tap operation determination in step S505 has been made in the touch operation determination processing in step S308.

If the recognition unit 225 determines in step S601 that the tap operation has been determined in the touch operation determination process in step S308 (S601 / Yes), the process proceeds to step S602.
When the processing proceeds to step S602, the recognition unit 225 superimposes the operation locus of the tap operation and the display button area in the UI screen 152 projected in step S301 (the operation locus of the tap operation is superimposed on the display button area). Judge whether or not. Here, the display button area is an area associated with the display button, and in this embodiment, is a display button area displayed on the UI screen 152 projected on the operation surface 150. As the display button area here, an area larger or smaller than the area of the display button actually displayed on the UI screen 152 projected on the operation surface 150 may be set.

If the recognition unit 225 determines in step S602 that the operation locus of the tap operation and the display button area in the UI screen 152 projected in step S301 are superimposed (S602 / Yes), step S603 is performed. Proceed to
In step S603, the recognizing unit 225 recognizes (detects) the operation of the display button determined to be superimposed in step S602.

  Then, when the process of step S603 is completed, when the recognition unit 225 determines in step S601 that the tap operation has not been determined (S601 / No), or in step S602, the recognition unit 225 determines the operation history of the tap operation. If it is determined that the display button area is not superimposed (S602 / No), the process of the flowchart shown in FIG. 6 is terminated, and the process proceeds to step S310 in FIG.

  Next, processing of the information processing apparatus 110 when an operation input is performed with the fingertip of the user's hand H will be described with reference to FIGS. 7 and 8.

  FIG. 7 shows an embodiment of the present invention and is a diagram showing an example of a display button in the UI screen 152 projected on the operation surface 150 (xy plane) shown in FIG. 1 and a tap operation on the display button. Specifically, FIG. 7A is a diagram illustrating a first example of a display button and a tap operation on the display button. FIG. 7B is a diagram illustrating a display button and a second example of a tap operation on the display button. FIG. 7C is a diagram illustrating a display button and a third example of a tap operation on the display button.

  FIG. 8 is a diagram illustrating an example of each operation position in a space in the z direction above the operation surface 150 illustrated in FIG. 1 according to the embodiment of the present invention. Specifically, FIG. 8A is a diagram illustrating an example of each operation position obtained based on the distance image 160 acquired by the distance image sensor 120. FIG. 8B is a diagram illustrating an example of each operation position indicating an operation locus where the user has actually moved the fingertip. Moreover, each operation position P0-P6 shown to Fig.8 (a) respond | corresponds to each operation position P0-P6 shown in FIG.8 (b), respectively. Further, the operation positions P1 to P6 shown in FIGS. 8A and 8B correspond to the operation positions P1 to P6 shown in FIGS. 7A to 7C, respectively. . 8A and 8B show a touch threshold value 801 used as a determination criterion in step S401 in FIG. 4 and a release threshold value 802 used as a determination criterion in step S408 in FIG.

  In the example shown in FIG. 8, the position where the user's fingertip is closest to the operation surface 150 is the operation position P3 as shown in FIG. 8B, but the distance image is shown in FIG. 8A. When acquired from the distance image 160 of the sensor 120, the operation position P4 is obtained. That is, FIG. 8 shows a case where the actual fingertip position and the fingertip position acquired from the distance image 160 are different. This is because, when the distance image 160 is acquired by the distance image sensor 120, when the fingertip position is close to the operation surface 150, the distance measurement noises of both the fingertip and the operation surface 150 influence each other, and the distance measurement noise is generated. This is thought to be due to the increase. Therefore, for example, as assumed in Patent Document 1, when the user's operation determination is performed based on the position where the fingertip of the user's hand H that is the operation body is closest to the operation surface 150, the user's operation is correctly recognized. It may be difficult to do so. On the other hand, in the embodiment of the present invention, the user's operation determination is performed based on the operation locus (the operation positions P1 to P6 in the example illustrated in FIG. 8) at the fingertip of the user's hand H that is the operation body. Can be properly recognized.

<First example of tap operation>
First, the processing of the information processing apparatus 110 in the first example of the tap operation on the display button and the display button will be described with reference to FIGS. 7A and 8A.

  In FIG. 7A, a display button 701 is shown as a display button in the UI screen 152 projected onto the operation surface 150 (xy plane). The display button 701 is displayed by the display control unit 226 in step S301 in FIG.

  Further, in FIG. 7A, the operation positions P1 to P6 are based on a touch threshold value 801 in which the fingertip of the hand H that is the operation body is the first distance from the operation surface 150 as shown in FIG. The operation locus of the tap operation, which is a series of operations that move away from the release threshold value 802, which is the second distance, is shown. The operation position P1 is a fingertip position at the start of the tap operation, and the operation position P6 is a fingertip position at the end of the tap operation. In FIG. 7, an operation position corresponding to the fingertip position at the end of the tap operation is indicated by a white circle. FIG. 7A shows, in chronological order, that the operation was performed in the order from the operation position P1 to the operation position P6 as the tap operation by the user. Below, the operation positions P1-P6 shown to Fig.7 (a) are demonstrated.

First, the operation position P1 shown in FIG. 7A, which is the operation position at the start of the tap operation, will be described.
In step S <b> 302 of FIG. 3, the distance image acquisition unit 221 acquires the distance image 160 from the distance image sensor 120. 3, when the operation position acquisition unit 222 detects the hand region 161 from the distance image 160 (S303 / Yes), the operation position acquisition unit 222 moves the hand region 161 to the hand region 161 in step S304. 1 is assigned as the ID. Subsequently, in step S305 of FIG. 3, the operation position acquisition unit 222 acquires the operation position P1 as the operation position 162 in the distance image 160, and holds the time t1 when the operation position P1 was acquired together with the operation position P1. To do. Subsequently, in step S306 of FIG. 3, the operation position acquisition unit 222 acquires position information (x1, y1, z1) at the operation position P1 in the real world.

  Next, in step S307 in FIG. 3, the operation determination unit 223 determines the touch state of the operation position P1 and holds the operation locus. Specifically, in FIG. 4 showing the detailed processing procedure in step S307 in FIG. 3, first, in step S401, the operation determination unit 223 determines that the operation position P1 is less than or equal to the touch threshold value 801 as shown in FIG. It is determined that there is (S401 / Yes). Subsequently, in step S402 in FIG. 4, the operation determination unit 223 determines that the operation position P0 immediately before the operation position P1 is not in the touch state as illustrated in FIG. 8A (S402 / No). Subsequently, in step S405 of FIG. 4, the operation determination unit 223 (or the operation track holding unit 224) holds that the operation position P1 is in the touch state. Furthermore, the operation determination unit 223 (or the operation track holding unit 224) holds that the operation position P0 immediately before the operation position P1 is in the released state. Subsequently, in step S406 of FIG. 4, the operation track holding unit 224 uses the information (x1, y1, z1, t1) related to the operation position P1 as information related to the start (first) operation position in the operation track of the tap operation. Hold as.

  Next, in step S308 of FIG. 3, the operation determination unit 223 determines a touch operation based on the operation position P1. Specifically, in FIG. 5 showing the detailed processing procedure in step S308 of FIG. 3, first, in step S501, the operation determination unit 223 determines that the current operation position P1 is in the touch state and the previous operation position P0. Is in a released state (S501 / Yes). Subsequently, in step S <b> 502 of FIG. 5, the operation determination unit 223 notifies the recognition unit 225 of a touch event.

  Next, in step S309 in FIG. 3, the recognition unit 225 determines display button operation. Specifically, in FIG. 6 showing the detailed processing procedure in step S309 of FIG. 3, in step S601, the recognition unit 225 determines that the determination of the tap operation (S505) has not been made in the processing of the operation position P1 (S601). / No). Next, in step S310 of FIG. 3, the display control unit 226 does not update the display of the UI screen 152 currently displayed because the display button operation is not recognized in step S309. Subsequently, in step S311 in FIG. 3, the information processing apparatus 110 determines that the power is not off (S311 / No), and then returns to step S302 in FIG.

Next, the operation position P2 shown in FIG.
Regarding the processing at the operation position P2, the processing from step S302 to step S304 in FIG. Subsequently, in step S305 in FIG. 3, the operation position acquisition unit 222 acquires the operation position P2 as the operation position 162 in the distance image 160, and holds the time t2 at which the operation position P2 was acquired together with the operation position P2. . Subsequently, in step S306 of FIG. 3, the operation position acquisition unit 222 acquires position information (x2, y2, z2) at the operation position P2 in the real world.

  Next, in step S307 in FIG. 3, the operation determination unit 223 determines the touch state of the operation position P2 and holds the operation locus. Specifically, in FIG. 4 showing the detailed processing procedure in step S307 in FIG. 3, first, in step S401, the operation determination unit 223 determines that the operation position P2 is equal to or less than the touch threshold value 801 as shown in FIG. It is determined that there is (S401 / Yes). Subsequently, in step S402 in FIG. 4, the operation determination unit 223 determines that the operation position P1 immediately before the operation position P2 is in the touch state as illustrated in FIG. 8A (S402 / Yes). . Subsequently, in step S403 in FIG. 4, the operation determination unit 223 (or the operation locus holding unit 224) holds that the operation position P2 is in the touch state. Furthermore, the operation determination unit 223 (or the operation track holding unit 224) holds that the operation position P1 immediately before the operation position P2 is in the touch state. Subsequently, in step S404 of FIG. 4, the operation locus holding unit 224 continuously holds the information (x2, y2, z2, t2) relating to the operation position P1 as information relating to the operation position in the operation locus of the tap operation. To do.

  Next, in step S308 of FIG. 3, the operation determination unit 223 determines a touch operation based on the operation position P2. Specifically, in FIG. 5 showing the detailed processing procedure in step S308 of FIG. 3, first, in step S501, the operation determination unit 223 determines that the current operation position P2 is in the touch state and the previous operation position P1. Is not in the released state (S501 / No). Subsequently, in step S503 of FIG. 5, the operation determination unit 223 determines that the current operation position P2 is not in the released state (S502 / No).

  Next, in step S309 in FIG. 3, the recognition unit 225 determines display button operation. Specifically, in FIG. 6 showing the detailed processing procedure in step S309 in FIG. 3, in step S601, the recognition unit 225 determines that the determination of the tap operation (S505) is not made in the processing of the operation position P2 (S601). / No). Next, in step S310 of FIG. 3, the display control unit 226 does not update the display of the UI screen 152 currently displayed because the display button operation is not recognized in step S309. Subsequently, in step S311 in FIG. 3, the information processing apparatus 110 determines that the power is not off (S311 / No), and then returns to step S302 in FIG.

  Since the processing at the operation position P3 and the operation position P4 shown in FIG. 7A is the same as the processing at the operation position P2 described above, the description thereof is omitted.

Next, the operation position P5 shown in FIG.
Regarding the processing at the operation position P5, the processing from step S302 to step S304 in FIG. Subsequently, in step S305 of FIG. 3, the operation position acquisition unit 222 acquires the operation position P5 as the operation position 162 in the distance image 160, and holds the time t5 at which the operation position P5 was acquired together with the operation position P5. . Subsequently, in step S306 in FIG. 3, the operation position acquisition unit 222 acquires position information (x5, y5, z5) at the operation position P5 in the real world.

  Next, in step S307 of FIG. 3, the operation determination unit 223 determines the touch state of the operation position P5 and holds the operation locus. Specifically, in FIG. 4 showing the detailed processing procedure in step S307 in FIG. 3, first, in step S401, the operation determination unit 223 determines that the operation position P5 is not less than or equal to the touch threshold value 801 as shown in FIG. (S401 / No). Subsequently, in step S407 in FIG. 4, the operation determination unit 223 determines that the operation position P4 immediately before the operation position P5 is in the touch state as illustrated in FIG. 8A (S407 / Yes). . Subsequently, in step S408 in FIG. 4, the operation determination unit 223 determines that the operation position P5 is not equal to or greater than the release threshold as illustrated in FIG. 8A (S408 / No). Subsequently, in step S411 in FIG. 4, the operation determination unit 223 (or the operation track holding unit 224) holds that the operation position P5 is in the touch state. Furthermore, the operation determination unit 223 (or the operation track holding unit 224) holds that the operation position P4 immediately before the operation position P5 is in the touch state. Subsequently, in step S <b> 412 of FIG. 4, the operation trajectory holding unit 224 continuously holds information (x5, y5, z5, t5) related to the operation position P5 as information related to the operation position in the operation trajectory of the tap operation. To do.

  In the processing at the operation position P5, the subsequent processing from step S308 to step S311 in FIG. 3 is the same as the processing at the operation position P2 described above, and thus description thereof is omitted.

Next, the operation position P6 shown in FIG. 7A, which is the operation position at the end of the tap operation, will be described.
Regarding the processing at the operation position P6, the processing from step S302 to step S304 in FIG. Subsequently, in step S306 of FIG. 3, the operation position acquisition unit 222 acquires the operation position P6 as the operation position 162 in the distance image 160, and holds the time t6 when the operation position P6 was acquired together with the operation position P6. . Subsequently, in step S306 of FIG. 3, the operation position acquisition unit 222 acquires position information (x6, y6, z6) at the operation position P6 in the real world.

  Next, in step S307 in FIG. 3, the operation determination unit 223 determines the touch state of the operation position P6 and holds the operation locus. Specifically, in FIG. 4 showing the detailed processing procedure in step S307 in FIG. 3, first, in step S401, the operation determination unit 223 determines that the operation position P6 is not less than or equal to the touch threshold value 801 as shown in FIG. (S401 / No). Subsequently, in step S407 in FIG. 4, the operation determination unit 223 determines that the operation position P5 immediately before the operation position P6 is in a touch state as illustrated in FIG. 8A (S407 / Yes). . Subsequently, in step S408 of FIG. 4, the operation determination unit 223 determines that the operation position P6 is equal to or greater than the release threshold as illustrated in FIG. 8A (S408 / Yes). Subsequently, in step S409 in FIG. 4, the operation determination unit 223 (or the operation track holding unit 224) holds that the operation position P6 is in the released state. Furthermore, the operation determination unit 223 (or the operation track holding unit 224) holds that the operation position P5 immediately before the operation position P6 is in a touch state. Subsequently, in step S410 of FIG. 4, the operation locus holding unit 224 uses the information (x6, y6, z6, t6) relating to the operation position P6 as the information relating to the end (last) operation position in the operation locus of the tap operation. Hold as. Thereafter, the operation trajectory holding unit 224 ends the holding of information related to the operation position in the operation trajectory.

  Next, in step S308 in FIG. 3, the operation determination unit 223 determines a touch operation based on the operation position P6. Specifically, in FIG. 5 showing the detailed processing procedure in step S308 of FIG. 3, first, in step S501, the operation determination unit 223 determines that the current operation position P6 is not in the touch state and the previous operation position. It is determined that P5 is not in the released state (S501 / No). Subsequently, in step S503 of FIG. 5, the operation determination unit 223 determines that the current operation position P6 is in the released state and the previous operation position P5 is in the touch state (S503 / Yes). Subsequently, in step S <b> 504 in FIG. 5, the operation determination unit 223 notifies the recognition unit 225 of a release event. Subsequently, in step S505 of FIG. 5, the operation determination unit 223 performs a series of operations of moving away from the release threshold value 802 after the fingertip of the hand H, which is the operation body, approaches the operation surface 150 from the touch threshold value 801. A tap operation is determined. Here, as a determination criterion for the tap operation, the distance on the operation surface 150 (xy plane) between the operation position P1 that is first closer than the touch threshold value 801 and the operation position P6 that is first farther than the release threshold value 802 is used. In addition, a condition that is equal to or less than a predetermined threshold is included. Here, it is assumed that the distance on the operation surface 150 (xy plane) between the operation position P1 (x1, y1) and the operation position P6 (x6, y6) is equal to or less than a predetermined threshold L1.

  Next, in step S309 in FIG. 3, the recognition unit 225 determines display button operation. Specifically, in FIG. 6 showing the detailed processing procedure in step S309 of FIG. 3, in step S601, the recognition unit 225 determines that the determination of the tap operation (S505) has been made in the processing of the operation position P6 (S601 / Yes). ) Subsequently, in step S602 in FIG. 6, the recognition unit 225 determines that the operation locus (operation positions P1 to P6) of the tap operation and the area of the display button 701 illustrated in FIG. S602 / Yes). Specifically, in the example illustrated in FIG. 7A, operation positions P <b> 2 and P <b> 3 that are part of the operation locus of the tap operation are superimposed on the area of the display button 701. Subsequently, in step S603 in FIG. 6, the recognition unit 225 recognizes (detects) the operation of the display button 701 that is determined to be superimposed on the operation locus (operation positions P1 to P6) of the tap operation.

  Next, in step S310 in FIG. 3, since the display control unit 226 determines (recognizes) that the display button 701 has been operated in step S309, the display control unit 226 performs control to change the display of the display button 701 from the display state before the recognition. Do. For example, the display control unit 226 controls to update the display so that the frame line of the display button 701 is thickened and blinks as shown in FIG. Note that the display button 701 may be updated by changing the color or brightness, turning off the display, or the like. Thereafter, the processing after step S311 in FIG. 3 is performed.

<Second example of tap operation>
Next, the processing of the information processing apparatus 110 in the case of the second example of the tap operation on the display button and the display button will be described with reference to FIGS. 7B and 8A.

  The difference between the second example of the tap operation and the first example of the tap operation described above is that the two display buttons A (702) and B (703) are close to each other as shown in FIG. 7B. They are displayed side by side, and the operation locus of the tap operation is superimposed on the two display buttons. Further, the operation positions P1 to P6 shown in FIG. 7B correspond to the operation positions P1 to P6 shown in FIG. 8A, respectively.

  First, the processing from the operation position P1 to the operation position P5 shown in FIG. 7B is the same as that in the first example of the tap operation described above with reference to FIG. .

Next, the operation position P6 shown in FIG. 7B, which is the operation position at the end of the tap operation, will be described.
Regarding the processing at the operation position P6, the processing from step S302 to step S308 in FIG. 3 is the same as that in the case of the first example of the tap operation described above, and thus the description thereof is omitted. Subsequently, in step S309 in FIG. 3, the recognition unit 225 determines display button operation. Specifically, in FIG. 6 showing the detailed processing procedure in step S309 of FIG. 3, in step S601, the recognition unit 225 determines that the determination of the tap operation (S505) has been made in the processing of the operation position P6 (S601 / Yes). ) Subsequently, in step S602 in FIG. 6, the recognizing unit 225 determines that the operation locus of the tap operation (operation positions P1 to P6) and the area of the display button shown in FIG. 7B overlap (S602). / Yes). Specifically, in the example illustrated in FIG. 7B, the operation positions P2 and P3, which are part of the operation locus of the tap operation, are superimposed on the area of the display button A (702), and a part of the operation locus of the tap operation. The operation positions P5 and P6 are superimposed on the area of the display button B (703).

  Subsequently, in step S603 in FIG. 6, the recognition unit 225 recognizes (detects) the operation of the display button that is determined to be superimposed on the operation locus (operation positions P1 to P6) of the tap operation. Specifically, when the operation locus of the tap operation is superimposed on a plurality of display buttons as shown in FIG. 7B, the recognition unit 225 displays the display button A closest to the operation position P1 at which the operation locus starts. This operation is recognized as (702). The reason is described below. Generally, the user tends to pay more attention (concentrate operation) at the start of the operation (or in the first half). Therefore, by recognizing the operation of the display button closest to the operation position at the start of the tap operation, it is possible to select the display button more in line with the user's operation intention and improve the operability. Because it is.

  Next, in step S310 in FIG. 3, the display control unit 226 determines (recognizes) that the display button A (702) has been operated in step S309, and therefore displays the display of the display button A (702) before the recognition. Control to change from state. For example, the display control unit 226 controls to update the display so that the frame line of the display button A (702) becomes thick and blinks as shown in FIG. 7B, for example. Note that the display button A (702) may be updated by changing the color or brightness, turning off the display, or the like. Thereafter, the processing after step S311 in FIG. 3 is performed.

<Third example of tap operation>
Next, the processing of the information processing apparatus 110 in the case of the third example of the tap operation on the display button and the display button will be described with reference to FIGS. 7C and 8A.

  The third example of the tap operation is different from the second example of the tap operation described above in that a double tap operation (two consecutive tap operations) is performed instead of a mere tap operation, as shown in FIG. This is the case. Specifically, FIG. 7C shows a first tap operation indicated by an operation locus from the operation position P1 to the operation position P6 and a second tap operation indicated by an operation locus from the operation position P7 to the operation position P12. The case where the double tap operation comprised by these is performed is shown. Here, the operation positions P1 to P6 shown in FIG. 7C correspond to the operation positions P1 to P6 shown in FIG. 8A, respectively. Further, the operation positions P7 to P12 shown in FIG. 7C correspond to the operation positions P1 to P6 shown in FIG. 8A, respectively.

  First, the processing from the operation position P1 to the operation position P5 of the first tap operation shown in FIG. 7C is the same as that in the second example of the tap operation described above (that is, using FIG. 7A). Since this is the same as in the case of the first example of the tap operation described above, the description thereof is omitted.

Next, the operation position P6 shown in FIG. 7C, which is the operation position at the end of the first tap operation, will be described.
Regarding the processing at the operation position P6, the processing from step S302 to step S307 in FIG. 3 is the same as in the second example of the tap operation described above (that is, the first tap operation described above with reference to FIG. 7A). Since this is the same as the case of the example), the description thereof is omitted.

  Next, in step S308 in FIG. 3, the operation determination unit 223 determines a touch operation based on the operation position P6. Specifically, in FIG. 5 showing the detailed processing procedure in step S308 in FIG. 3, the processing in step S501, step S503, and step S504 is the same as that in the second example of the tap operation described above (that is, FIG. the same as in the case of the first example of the tap operation described above using a), and the description thereof will be omitted. Subsequently, in step S505 of FIG. 5, the operation determination unit 223 performs a series of operations of moving away from the release threshold value 802 after the fingertip of the hand H, which is the operation body, approaches the operation surface 150 from the touch threshold value 801. A tap operation is determined.

Furthermore, in the third example of the tap operation, the operation determination unit 223 determines the double tap operation. That is, the operation determination unit 223 determines whether a plurality of tap operations (a plurality of series of operations) are double tap operations that are integrated operations, depending on whether or not a predetermined condition is satisfied. In the present embodiment, the following four conditions are applied as determination conditions for the double tap operation by the operation determination unit 223.
[First condition] Distance on the operation surface 150 (xy plane) between the operation position at the end of the first tap operation and the operation position at the start of the second tap operation performed after the first tap operation Is below a predetermined threshold.
[Second Condition] The distance on the operation surface 150 (xy plane) between the start operation position in the second tap operation and the end operation position in the second tap operation is not more than a predetermined threshold.
[Third Condition] The elapsed time from the time at the end operation position in the first tap operation to the time at the start operation position in the second tap operation performed after the first tap operation is predetermined. It must be below the threshold value.
[Fourth Condition] The elapsed time from the time at the start operation position in the second tap operation to the time at the end operation position in the second tap operation is not more than a predetermined threshold.
In the present embodiment, the operation determination unit 223 determines a double tap operation in consideration of the first condition to the fourth condition.

  Here, in the process of the operation position P6, since there is no tap operation performed before the tap operation including the operation position P6, it is not determined that the double tap operation has been performed. In the process at the operation position P6, the process after step S309 in FIG.

Also, for the processing of the operation positions P7 to P12 shown in FIG. 7C, the same processing as the processing of the operation positions P1 to P6 in the third example of the tap operation described above is performed. Here, in the process of the operation position P12, in step S505 of FIG. 5, the operation determination unit 223 determines the double tap operation. At this time, in the example shown in FIG. 7C, the distance on the operation surface 150 (xy plane) between the operation position P6 at the end of the first tap operation and the operation position P7 at the start of the second tap operation is It is less than a predetermined threshold value and satisfies the first condition described above. In the example shown in FIG. 7C, the distance on the operation surface 150 (xy plane) between the start operation position P7 in the second tap operation and the end operation position P12 in the second tap operation is predetermined. And the second condition described above is satisfied. Further, in the example shown in FIG. 7C, the elapsed time from the time t6 at the end operation position P6 in the first tap operation to the time t7 at the start operation position P7 in the second tap operation is a predetermined time. It is less than the threshold value and satisfies the third condition described above. Further, the elapsed time from the time t7 at the start operation position P7 in the second tap operation to the time t12 at the end operation position P12 in the second tap operation is equal to or less than a predetermined threshold, and the above-described fourth The condition shall be met.
Accordingly, in this case, the operation determination unit 223 determines a double tap operation (S505) in the process of the operation position P12.

With respect to the process at the operation position P12, the processes after step S309 in FIG. 3 will be described.
In FIG. 6 showing the detailed processing procedure in step S309 of FIG. 3, first, in step S601, the recognition unit 225 determines that the determination of the double tap operation (S505) has been made in the processing of the operation position P12 (S601 / Yes). To do. Subsequently, in step S602 of FIG. 6, the recognition unit 225 includes an operation locus (operation positions P1 to P6 and operation positions P7 to P12) of the double tap operation, a display button A (704) illustrated in FIG. It is determined that the area of the display button B (705) is superimposed (S602 / Yes). Specifically, in the example shown in FIG. 7C, the operation positions P2 and P3, which are part of the operation locus of the double tap operation, are superimposed on the area of the display button A (704), and the double tap operation is performed. The operation positions P5, P6, and P8 to P10, which are a part of the operation locus, are superimposed on the area of the display button B (705).

  Regarding the processing at the operation position P12, the recognition unit 225 subsequently recognizes (detects) the operation of the display button determined to be superimposed on the operation locus of the double tap operation in step S603 of FIG. In the third example of the tap operation shown in FIG. 7C, a double tap operation including a first tap operation from the operation positions P1 to P6 and a second tap operation from the operation positions P7 to P12 is performed. The operation locus of each tap operation is superimposed on the display button. In this case, in the present embodiment, the recognizing unit 225 recognizes (detects) the operation of the display button with priority given to the superimposition by the operation locus of the first first tap operation in the double tap operation. Specifically, in the example illustrated in FIG. 7C, the operation locus (operation positions P1 to P6) of the first tap operation is initially superimposed on the display button A (704). The operation of the display button A (704) is recognized (detected). Specifically, when the operation locus of the double tap operation is superimposed on a plurality of display buttons as shown in FIG. 7C, the recognition unit 225 operates the first operation track of the first tap operation in the double tap operation. Priority is given to the result of the superimposition determination made by, and the operation for the display button is recognized. The reason is described below. Generally, the user tends to pay more attention (concentrate operation) at the start of the operation (or in the first half). Accordingly, by recognizing the operation of the display button by giving priority to the result of superimposition determination based on the operation trajectory of the first first tap operation in the double tap operation, the display button can be selected in accordance with the user's operation intention. This is because it is considered that the operability can be improved.

  Thereafter, similarly to the first example of the tap operation, the processing after step S310 in FIG. 3 is performed.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
This program and a computer-readable storage medium storing the program are included in the present invention.

  Note that the above-described embodiments of the present invention are merely examples of implementation in practicing the present invention, and the technical scope of the present invention should not be construed as being limited thereto. It is. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

DESCRIPTION OF SYMBOLS 100: Table top interface system, 110: Information processing apparatus, 120: Distance image sensor, 130: Visible light camera, 140: Projector, 150: Operation surface

Claims (15)

An information processing apparatus for recognizing an operation of an operating body on a display object displayed on an operation surface,
Obtaining means for obtaining an operation position of the operating body;
Based on a plurality of the operation positions acquired by the acquisition means, a series of operations in which the operating body moves away from the second distance after the operating body approaches the operating surface from the first distance in the height direction. Determination means for determining;
Recognizing means for recognizing an operation of the operating body on the display object when a two-dimensional operation trajectory on the operation surface determined by the plurality of operation positions related to the series of operations is included in the area of the display object; ,
An information processing apparatus comprising:   The recognizing unit recognizes an operation of the operating body on the display object when at least a part of the operation trajectory is superimposed on an area of the display object. Information processing device.   In the determination unit, a distance on the operation surface between the operation position first approaching the first distance and the operation position first distant from the second distance is equal to or less than a predetermined threshold. The information processing apparatus according to claim 1, wherein the series of operations is determined.   In the determination unit, a time from the time at the operation position first approaching the first distance to the time at the operation position first distant from the second distance is equal to or less than a predetermined threshold. The information processing apparatus according to claim 1, wherein the series of operations is determined.   The recognizing means recognizes the operation of the operating body with respect to the display object closest to the operation position at the start of the operation trajectory when the operation trajectory is superimposed in a plurality of display object areas. The information processing apparatus according to any one of claims 1 to 4, wherein the information processing apparatus is characterized in that:   The recognizing unit uses the operation locus on the operation surface determined by the plurality of operation positions related to the series of operations determined first when the determining unit determines a plurality of the series of operations. The information processing apparatus according to claim 1, wherein the recognition is performed. The determination means further determines a plurality of the series of operations satisfying a predetermined condition as an integrated operation,
The information processing apparatus according to claim 6, wherein the recognition unit performs the recognition on the series of operations determined as the integrated operation by the determination unit.   The predetermined condition includes the operation position at the end of the first series of operations among the plurality of series of operations, and the start of the second series of operations performed after the first series of operations. The information processing apparatus according to claim 7, further comprising a condition that a distance from the operation position on the operation surface is a predetermined threshold value or less.   The predetermined condition includes the operation position at the start of a second series of operations performed after the first series of operations of the plurality of series of operations and the end of the second series of operations. The information processing apparatus according to claim 7, further comprising a condition that a distance from the operation position on the operation surface is a predetermined threshold value or less.   The predetermined condition is a start in a second series of operations performed after the first series of operations from a time at the operation position at the end of the first series of operations among the plurality of series of operations. The information processing apparatus according to claim 7, further comprising a condition in which a time until the time at the operation position is equal to or less than a predetermined threshold.   The predetermined condition is the end of the second series of operations from the time at the operation position of the start of the second series of operations performed after the first series of operations of the plurality of series of operations. The information processing apparatus according to claim 7, further comprising a condition in which a time until the time at the operation position is equal to or less than a predetermined threshold.   12. The display control unit according to claim 1, further comprising a display control unit that performs control to change the display of the display object that has recognized the operation of the operating body by the recognition unit from a display state before the recognition. The information processing apparatus according to claim 1. A holding unit that holds the operation locus related to the series of operations determined by the determination unit;
The information processing apparatus according to claim 1, wherein the recognition unit performs the recognition using the operation locus held by the holding unit. An information processing apparatus control method for recognizing an operation of an operating tool on a display object displayed on an operation surface,
An acquisition step of acquiring an operation position of the operating body;
Based on the plurality of operation positions acquired in the acquisition step, a series of operations in which the operating body moves away from the second distance after approaching the operating surface from the first distance in the height direction. A determination step for determining;
A recognition step for recognizing an operation of the operating body on the display object when a two-dimensional operation locus on the operation surface determined by the plurality of operation positions related to the series of operations is included in the area of the display object; ,
A method for controlling an information processing apparatus, comprising:   The program for functioning a computer as each means in the information processing apparatus of any one of Claims 1 thru | or 13.