JP2017045332A - Information processing device, control method thereof, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem that, when a touch of the finger to the predetermined object surface is detected based on a degree of approach between both the finger and the object surface, an erroneous recognition of the touch operation caused due to the influence of the other fingers moved in association with the finger used for the operation.SOLUTION: Information processing device includes: in order to solve the problem, an image acquisition part 120 for acquiring an input image reflected with position information inside a space into which a person inserts a hand to perform an operation; a position detection part 121 for detecting each position information of a plurality of fingertips from the hand of the person from the acquired input image; a specifying part 123 for specifying the fingertip serving as an object of processing for determining whether the predetermined object surface is touched among the plurality of finger tips using the relationship among the plurality of detected finger tips; the detected position information about the specified fingertip; and recognition part 125 for recognizing the touch operation which is input on the prescribed object surface by the fingertip.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for recognizing a touch operation based on an image.

  In recent years, a new user interface (hereinafter referred to as UI) system that performs superimposed display by a projector has been developed. In the above-described system, a user interface component (hereinafter referred to as UI component) such as a button is projected onto a table, a wall, a document, or an object, and the user touches the UI component. The above-described system can be configured smaller than an apparatus having a large display, and thus is relatively easy to carry, and has the advantage that rich interaction combining physical and electronic media is possible. is there. In the above-described system, an operation such as a finger of a user is performed using an image captured by a sensor or a camera in order to provide a target surface on various objects and recognize a touch operation on the target surface without using a touch panel. Often it is detected that the body has touched the target surface. Specifically, a threshold is set for the distance from the target surface, and a state where the fingertip is present in a range closer to the target surface than the threshold is regarded as a touched state.

  Japanese Patent Application Laid-Open No. 2004-228561 discloses a method of acquiring a three-dimensional position between a fingertip and an object using a distance sensor and performing touch recognition based on the speed of the finger or the distance between the finger and the object. Further, in Patent Document 2, since the position closest to the target surface among the operating bodies is determined as the fingertip position, the position of the fingertip closest to the target surface is specified among the plurality of fingertips of the hand, and the position is determined as the fingertip position. It is disclosed that.

JP 2014-157515 A JP 2014-142695 A

  When the user touches the target surface with a part of the plurality of fingers, there is a phenomenon that other fingers not used for the touch approach the target surface along with the movement of the finger used for the touch. When using a device that detects that an object is in direct contact, such as a touch panel, finger movement that is not used for such touch does not affect touch detection accuracy. However, when a touch is detected using a captured image based on a comparison between the distance from the target surface of the operating tool and a threshold value, the range of the finger that is not intended to be used for the touch is closer to the target surface than the threshold value The touch may be detected by intruding into the screen. On the other hand, when a touch operation using a plurality of fingers called a multi-touch operation is input, the user may bring the plurality of fingers close to the target surface simultaneously with an intention.

  In the prior art, when multiple fingers are close to the touch target surface, there are both the possibility of including fingers that are not used for operation and the possibility of using multiple fingers for touch for multi-touch operations. It is not considered to distinguish between them.

  The present invention has been made in view of the above problems, and in the case of detecting the touch of a finger with respect to a predetermined target surface based on the degree of proximity between the two, it moves in conjunction with the finger used for the operation. An object of the present invention is to reduce misrecognition of touch operations caused by the influence of fingers.

  In order to solve the above-described problems, the present invention provides an image acquisition unit that acquires an input image reflecting position information in a space where a person inserts a hand and performs an operation, and the input image acquired by the image acquisition unit Position detecting means for detecting position information of each of a plurality of fingertips of the person's hand based on the position information reflected in the portion of the person's hand, and detected by the position detecting means A specifying unit that specifies a fingertip that is a target of processing for determining whether the predetermined target surface is touched among the plurality of fingertips using the relationship between the plurality of fingertips; and the fingertip specified by the specifying unit Recognizing means for recognizing a touch operation input on the predetermined target surface by the fingertip using the position information detected by the position detecting means and the position information of the touch target surface. Equipped with a.

  According to the present invention, when a finger touch on a predetermined target surface is detected based on the degree of proximity between the two, a touch operation caused by the influence of another finger moving in conjunction with the finger used for the operation False recognition can be reduced.

The figure showing an example of the external appearance of the table top system using information processor The figure which shows an example of a structure of the table top system using information processing apparatus Diagram showing the principle of touch recognition The figure showing an example of the state of the hand which inputs single touch, and a recognition result The figure showing an example of the state of the hand which inputs multi-touch, and a recognition result The figure showing an example of the state of the hand which inputs multi-touch, and a recognition result The figure which shows an example of the application which utilizes the recognition result of a multi-touch recognition process The flowchart which shows an example of the flow of touch operation recognition processing The flowchart which shows an example of the flow of a single touch operation recognition process The flowchart which shows an example of the flow of the determination process whether each fingertip is considered as the finger in connection with operation The flowchart which shows an example of the flow of multi-touch recognition processing The flowchart which shows an example of the flow of the determination process of the threshold value based on the relationship between fingers Diagram showing the principle of touch recognition for objects other than planes The figure which shows an example in the case of setting the conditions of touch determination for every finger | toe

  Embodiments of the present invention will be described below in detail with reference to the drawings. In addition, embodiment described below shows an example at the time of implementing this invention concretely, and is not restricted to this.

[First Embodiment]
<Appearance and configuration of system>
FIG. 1 shows an example of the appearance of a table top system in which an information processing apparatus 100 described in this embodiment is installed. The information processing apparatus 100 can set an arbitrary plane such as a table or a wall surface as a target surface by irradiating the projection light from the projection light irradiation unit 105 of the projector. In the case of the table top system shown in FIG. 1, the information processing apparatus 100 is installed on the table surface 101, and a display image is projected on the table surface. Here, the image 102 is one of UI parts projected on the table surface by the projector. In the following, all of the UI parts and various images such as photographs projected on the table surface by the projector are collectively referred to as display items.

  The light receiving unit 106 indicates the viewpoint of the distance image obtained by the distance image sensor 115 of the infrared pattern projection method (or the time-of-flight method). As an example, in the case of the system of FIG. 1, the infrared light emitted from the light emitting unit 107 receives reflected light reflected by the subject surface by the light receiving unit 106 and images it. In the present embodiment, the light receiving unit 106 is installed at a position for imaging at an angle of view looking down from above on the target surface. Accordingly, the distance from the light receiving unit 106 to the subject surface is reflected in each pixel of the distance image obtained by the distance image sensor 115. The distance image acquisition method will be described based on an infrared pattern projection method that is less affected by ambient light and table surface display as an example, but a parallax method or an infrared light reflection time method is used depending on the application. It is also possible to do. The operation area 104 represents a range in which a user operation can be recognized in the present embodiment. In addition, the operation area 104 matches the range in which the projector can project and the visual field range of the distance image sensor 115.

  Note that the light emitting unit 107 and the light receiving unit 106 do not necessarily have to be installed above as long as an image obtained by viewing the target surface 101 from above is obtained. For example, you may comprise so that reflected light may be imaged using a mirror. Similarly, in the example of FIG. 1, the projection light irradiation unit 105 also projects onto the target surface 101 so as to look down from obliquely above. However, the projection light projected in a direction different from that shown in FIG. You may guide to the object surface 101 using a mirror etc. Even when this embodiment is applied to a distance image sensor or projector used in a system in which the target surface 101 is installed along the vertical direction, an optical system including a mirror can be used in the same manner.

  In the present embodiment, as shown by the hand 103a, the hand 103b, etc., the user moves his / her hands from a plurality of directions in the operation space on the operation area 104 (the surface of the operation area 104 set on the target surface 101 and the distance image sensor 115. It can be inserted into the space between the light receiving portions 106. An operation of the information processing apparatus 100 when the user inputs a gesture operation with a display item as an operation target to the table top system by using a hand will be described. However, this embodiment is applicable not only when the display item is projected on the table surface 101 but also when the projection light is irradiated on a whiteboard or an arbitrary wall surface, or when the projection surface is not a flat surface. Is possible.

  In the present embodiment, as shown in FIG. 1, the x-axis and y-axis are set on a two-dimensional plane parallel to the operation area 104, and the z-axis is set in the height direction orthogonal to the operation area 104, thereby obtaining three-dimensional position information Treat as coordinate values. However, when the target surface is not a flat surface or depending on the positional relationship between the user and the target surface, the coordinate axis may not necessarily be parallel or orthogonal to the target surface. In this case as well, the z axis is set in a direction intersecting the target surface in order to detect the degree of proximity between the recognition target and the target surface (the degree of the distance between the two), and the x axis and the y axis are z. Set to the direction intersecting the axis. In the present embodiment, coordinate conversion based on the lens characteristics of the light receiving unit 106 and the relative positional relationship with the target surface 101 is performed on each pixel value of the input distance image. In each pixel of the distance image acquired in the present embodiment, a value corresponding to the magnitude of the distance from the light receiving unit 106 to the subject surface is held as a pixel value. Therefore, by coordinate conversion, the coordinates of each pixel in the distance image are mapped to the real-world coordinate system defined in the table, and the values of the x, y, and z coordinates can be obtained.

  In this embodiment, the degree of proximity between the fingertip and the touch target surface that is detected to determine whether or not the user's fingertip has touched the touch target surface is distance information obtained based on the pixel value of the distance image. Will be described as an example. However, information that can be used as the degree of proximity between the fingertip and the touch target surface is not limited thereto. For example, it may be information on the amount of change in capacitance caused by the operating body approaching the touch target surface, or information on the amount of change in temperature, pressure, or contact area.

  FIG. 2A is a block diagram illustrating an example of a hardware configuration of the information processing apparatus 100 according to the present embodiment. In the figure, a CPU 110 comprehensively controls each device connected via a bus 113. In addition to the operating system (OS), each processing program, device driver, and the like according to the present invention as shown in the flowcharts described later are stored in the ROM 112, temporarily stored in the RAM 111, and appropriately executed by the CPU 110. The RAM 111 is used as a temporary storage area such as a main memory or work area of the CPU 110 that can be accessed at high speed. Here, the OS, each processing program, and the like may be stored in the external storage device 116, and in this case, information necessary when the power is turned on is appropriately read into the RAM 111. The display I / F 117 converts a display item (display image) generated inside the information processing apparatus 100 into a signal that can be processed by the projector 118. The input / output I / F 114 acquires distance information from the distance image sensor 115, converts it into information that can be processed by the information processing apparatus 100, and converts data between the storage device 116 and the information processing apparatus 100.

  In the present embodiment, it is assumed that the digital data projected by the information processing apparatus 100 is stored in the storage device 116. As the storage device 116, a storage device 116 connected via various input / output I / Fs 114 such as a disk device, a flash memory, a network, and a USB is used. In the present embodiment, the distance image sensor 115 is an imaging unit used to acquire information on the operation area 104. The image acquired by the distance image sensor 115 is temporarily stored in the RAM 111 as an input image, and is appropriately processed and discarded by the CPU 110. However, necessary data may be stored in the storage device 116.

  In the example of FIG. 2A, the distance image sensor 115 and the projector 118 are devices integrated with the information processing apparatus 100, respectively. However, it may be an external device connected via an input / output interface, and the information processing system may be configured in cooperation with the information processing device 100. Further, a visible light camera that captures a visible light image of a document or a three-dimensional object placed in the operation area 104 may be added to constitute a system having a function as a document camera.

  FIG. 2B is a block diagram illustrating an example of a functional configuration of the information processing apparatus 100 according to the present embodiment. The information processing apparatus 100 includes at least an image acquisition unit 120, a region detection unit 121, a position detection unit 122, a specification unit 123, a condition determination unit 124, an identification unit 125, and a recognition unit 126. Each of these functional units is realized by the CPU 110 developing a program stored in the ROM 112 in the RAM 111 and executing processing according to each flowchart described later. Further, for example, when hardware is configured as an alternative to software processing using the CPU 110, arithmetic units and circuits corresponding to the processing of each functional unit described here may be configured. The holding unit 126 is a functional unit corresponding to either the ROM 112 or the storage device 116, and stores a look-up table used for linked fingertip correction, which will be described later, image data that is a basis of an image output to the projector 118, and the like. Hold.

  The image acquisition unit 120 acquires a distance image captured by the distance image sensor 115 as an input image at regular intervals, and stores the acquired distance image in the RAM 111 as needed. The object acquired by the image acquisition unit 120 and exchanged with each functional unit is actually a signal corresponding to the image data, but in this specification, simply “acquire a distance image” or “acquire an input image”. ".

  The region detection unit 121 performs threshold determination and noise reduction processing on each pixel of the input image acquired by the image acquisition unit 120, and extracts one or more hand regions from the distance image. The hand region is a region in which the hand used by the user as an operation tool is shown in the input distance image. In the case of the present embodiment, an area in which a subject that is indicated to be present at a position higher than the target surface 101 is captured by the pixel value of the distance image and an area in contact with the image edge is extracted as a hand area.

  The position detection unit 122 detects one or more fingers of the user's hand based on the contour information of the hand region extracted by the region detection unit 121, and represents a coordinate value representing a three-dimensional position of one point regarded as the tip of the finger Is acquired as position information of the fingertip. In the case of a distance image handled as an input image in the present embodiment, the pixel value of the pixel corresponding to the fingertip corresponds to the luminance value of infrared light reflected by the upper surface of the user's fingertip. Since this luminance value represents the reflected light intensity of infrared light emitted from the light emitting unit 107, the stronger the intensity, the closer the distance from the distance image sensor 115 to the upper surface of the user's finger, and the weaker the intensity, the distance image sensor. It can be considered that the distance from 115 to the finger is long. As described above, in the present embodiment, the pixel value of the fingertip position specified in the distance image is handled as distance information reflecting the distance between the distance image sensor 115 and the fingertip. Therefore, the information processing apparatus 100 is illustrated in FIG. 1 by reflecting predetermined calibration parameters on the position information (position information indicating the pixels in the image) and the pixel value of the fingertip position in the operation space. It is converted into (x, y, z) coordinates and acquired as a three-dimensional position of the fingertip. The coordinate value of the fingertip in the z direction is also referred to as “fingertip height”. In this embodiment, the Euclidean distance (mm) in the three-dimensional space is used as a measure of the fingertip height, but another distance measure may be used.

  When there are a plurality of fingertips detected by the position detection unit 122, the specifying unit 123 specifies a finger that is considered to be used for input of the touch operation based on the relative relationship between the position information of each of the plurality of fingertips. To do. In this embodiment, after identifying one fingertip that is considered to be most likely to be used for input of the touch operation, another fingertip is connected to the touch operation based on the relative relationship with the specified fingertip. A condition for determining whether the finger is used for input is determined. Then, by narrowing down the other fingers using the determined condition, the finger that is considered to be used for the input of the touch operation is specified from the whole. The identified fingertip is a target of processing for determining whether or not the touch target surface is touched, which is performed by the recognition unit 126 described later.

  The condition determination unit 124 determines a condition for determining whether the fingertip of each fingertip detected by the position detection unit 122 has touched the table 101 that is the target surface. In the present embodiment, the condition determining unit 124 further determines a condition for determining that the fingertip is intentionally released from the table 101, that is, the touch input is ended. In this embodiment, the touch is detected by comparing the distance between the touch target surface and the fingertip with a threshold value. Accordingly, the condition determining unit 124 determines the size of the threshold as the above condition.

  The identification unit 125 identifies the left and right sides of the arm based on the detected geometric positional relationship of the fingertips. For example, when five fingertips are detected and the coordinates of the leftmost fingertip are closer to the coordinates of the intrusion position than any other fingertip as viewed from the intrusion position, the shortest finger (thumb) is located on the left side Therefore, it is recognized that the arm region corresponds to the right hand. However, the left / right recognition method of the arm is not limited to this, and an SVM (Support Vector Machine) in which an arm-shaped pattern is learned may be used. Further, the identification unit 125 identifies the types of the plurality of detected fingertips based on the identification results of the left and right arms and the geometric positional relationship of the fingertips. For example, when five fingertips of the right hand are detected, it is identified as a little finger, a ring finger, a middle finger, an index finger, and a thumb clockwise from the intrusion position.

  The recognition unit 126 recognizes that the fingertip has touched the predetermined target surface using the position information of each fingertip of the user detected by the position detection unit 122 and the condition determined by the condition determination unit 124. In the present embodiment, touch operations that can be recognized by the recognition unit 126 include a single touch operation and a multi-touch operation. Specific contents of the process of recognizing the touch by the recognition unit 126 will be described later with reference to FIG.

  The output control unit 127 uses the information stored in the ROM 112 and the storage device 116 to generate an image to be projected on the table 101 by the projector 118 that is the display unit of the present embodiment. For example, based on the recognition result of the multi-touch operation, at least a part of the displayed image is subjected to deformation such as enlargement / reduction and rotation, and is output to the display unit. The output image is projected onto the table 101 that is the target surface by the projector 118. The projected image may include a plurality of display items, and may be moved, enlarged / reduced, and rotated for each item, or the entire image may be a target of movement, enlargement / reduction, and rotation.

  In addition, a function recognition unit that configures a function unit according to the purpose of use or application of the information processing apparatus 100, such as a gesture recognition unit that recognizes a spatial gesture operation such as a hand gesture, a display control unit that controls an image output to the projector 118, or the like. Can do. Each functional unit described above uses the RAM 111 as a holding unit that temporarily holds information used in the processing according to the present embodiment. However, the holding unit may be a part of the ROM 112, the storage device 116, or the like.

  Hereinafter, the first embodiment will be described using the user's hands 103a and 103b and their fingers as an example of an operation body used for input of an operation by the user and its tip. However, in the present embodiment, the operation body is not limited to a live finger, but can also be applied to a case where a part corresponding to a finger needs to be identified using a prosthetic hand or a robot arm.

<Touch recognition process>
Here, an outline of the touch recognition process executed by the recognition unit 126 will be described. The recognition unit 126 detects the following two states for each detected fingertip. One is a touch state. The touch state is a state where the fingertip height of the fingertip is below the touch threshold and is sufficiently close to the extent that the fingertip and the target surface are considered to be touching. A condition for determining whether or not they are close enough is determined for each finger by the condition determination unit 124. Another state is a non-touch state. The non-touch state is a state in which the user intentionally moves the fingertip away from the target surface, and is recognized when the fingertip height exceeds the release threshold. In the present embodiment, the release threshold is also determined for each finger by the condition determination unit 124.

  In the present embodiment, different values are used as the touch threshold and the release threshold, and in particular, the release threshold is larger than the touch threshold. This is based on the fact that the movement of the fingertip differs between when the fingertip is brought close to and away from the fingertip, and detection errors occur in the sensor, until the fingertip is sufficiently separated from the target surface during the touch state. This is a device to prevent the release from being detected. As a result, erroneous recognition that a release is erroneously detected during the continuation of the touch operation due to detection error or fingertip shake is reduced. However, the difference between the touch threshold value and the release threshold value may be adjusted according to the detection accuracy of the sensor and the installation environment of the device, and they may be the same. In many cases, the size of the touch threshold and the release threshold is set sufficiently large with respect to the thickness of the finger. This is to enable robust touch recognition even when there is noise in the distance information obtained from the distance image sensor. However, this point may also be adjusted according to the detection accuracy of the sensor and the installation environment of the apparatus.

  Hereinafter, “touch recognition” in the present embodiment refers to determining the state of the fingertip by determining whether the fingertip is smaller than the touch threshold or whether the fingertip is larger than the release threshold. On the other hand, “recognition of touch operation” is an instruction input to the information processing apparatus 100 when the position of the fingertip or the movement locus in the touch state corresponds to various commands stored in the dictionary. Interpretation. In the present embodiment, the term touch operation refers to a general operation with a fingertip in which a transition between a touch state and a non-touch state occurs due to a change in fingertip height. In particular, the term “single touch operation” refers to a general touch operation with one fingertip. In the present embodiment, a tap operation will be described as an example of a single touch operation. The tap operation generally refers to an operation of releasing within a predetermined time without touching a certain position on the target surface, and is mainly used when selecting an item on the target surface. . In contrast to the single touch operation, the term multi-touch operation refers to a general touch operation in which one instruction is input by a plurality of fingertips. In the present embodiment, as an example of the multi-touch operation, a pinch operation instructing enlargement / reduction of the display image by widening or narrowing the interval between the two touch positions and the two touch positions are relatively rotated. Thus, a rotation operation for instructing rotation of the display image is given.

  In the present embodiment, the recognition unit 126 notifies the application of a TOUCH event when the fingertip transitions from the non-touch state to the touch state. When the touch state is changed to the non-touch state, a RELEASE event is issued. Hereinafter, the event that the recognition unit 126 notifies the output control unit 127 and the application of an event that triggers subsequent processing is expressed as “issue an event”. The event information is notified together with information indicating which fingertip has made a state transition at which position. Upon receiving the notification, the application causes the output control unit 127 to update the projected image based on the event information, the fingertip information, and the UI component information.

  Here, with reference to FIGS. 3A to 3C, the fingertip height variation accompanying the touch, the above-described fingertip state transition, and event issuance will be specifically described. FIG. 3A shows a state before the fingertip 203 of the finger 202 is touched. Since the fingertip height 204 exceeds the release threshold 201, the fingertip 203 is recognized as being in a non-touch state. Since no state transition has occurred, no event is issued. FIG. 3B shows a state at the moment when the fingertip touches the target surface (surface of Z = 0). Since the fingertip height 205 is below the touch threshold value 200, the fingertip 203 is recognized as being in a touch state. Since the state of the fingertip 203 has transitioned from the non-touch state to the touch state, a TOUCH event is issued. FIG. 3C shows a state in which the fingertip is released from the target surface immediately after touching (during the threshold time defined for the tap operation). Since the fingertip height 206 exceeds the release threshold 201, the fingertip 203 is recognized as being in a non-touch state. A RELEASE event is issued when the state of the fingertip 203 transitions from the touch state to the non-touch state. At this time, the time until the transition from the touch state to the non-touch state is within the tap time threshold, and the deviation between the xy coordinates at the moment when the touch state is reached and the xy coordinates at the moment when the touch state is not within the tap deviation threshold. Suppose that When the condition for recognizing the tap operation is satisfied, a TAP event indicating that the tap operation has been input is issued.

  The touch recognition process in the recognition unit 126 is roughly divided into a single touch recognition process and a multi-touch recognition process. The single touch recognition process recognizes whether one fingertip is touched or not touched, and issues a TOUCH event, a RELEASE event, and a TAP event. In the multi-touch recognition process, pairing of a plurality of fingertips in a touch state is performed based on the result of the single-touch recognition process executed as the previous stage and the positional relationship between the fingertips. Then, a pinch operation and a rotate operation are recognized by a change in the position of the paired fingertips, and a PINCH event and a ROTATE event indicating that they are input are issued.

  In this embodiment, in the touch recognition as described above, the height lower than the touch threshold as a result of the finger that is not used for the operation being moved in conjunction with the finger that is intended to be used for the operation and that is close to the target surface. It is recognized as a problem that the fingertip has been touched and touch is erroneously detected. Therefore, in the present embodiment, when a plurality of fingertips are detected, the fingers that are considered to be highly likely to be used for the operation are narrowed down, and touch recognition is performed on the narrowed fingers. In other words, erroneous recognition of a touch is reduced by not setting a finger that is close to the target surface in conjunction with another finger despite being a finger that is not used for an operation as a target of the touch recognition process.

<Flow of touch operation recognition processing>
With reference to the flowchart of FIG. 8, the flow of the recognition process of the touch operation which is the main process of this embodiment will be described. In the present embodiment, the process of the flowchart of FIG. 8 is periodically repeated every time a distance image is input from the distance image sensor 115. Therefore, the cycle in which the process is repeated matches the frame rate of the captured image of the distance image sensor 115.

  In step S <b> 100, the image acquisition unit 120 acquires the distance image captured by the distance image sensor 115 as an input image at regular intervals, and stores it in the RAM 111 as needed. The pixel value of the distance image captured by the distance image sensor 115 reflects the distance to the distance image sensor 115. In step S <b> 101, the region detection unit 121 extracts a region (arm region) in which a person's arm is captured from the distance image acquired by the image acquisition unit 120, acquires position information indicating the contour, and stores the information in the RAM 111. Hold.

  Here, the person's “arm” refers to all the parts from the person's shoulder to the fingertip, and an area where a portion corresponding to the person's arm is captured in the captured image is an arm area. In the operating body or in this embodiment, the “hand” refers to all of the arms from the wrist to the tip. “Hand” includes five fingers, palm, and back. As an actual process of the area detection unit 121, the threshold value process is performed on the coordinate value in the z direction indicated by each pixel of the acquired distance image, and thus the pixel group includes a pixel group having a coordinate value higher than the table. A region that is in contact with the edge of the image is extracted as an arm region. In FIG. 1, the arm region is a portion of the hand 103 a and the hand 103 b where the portion that has entered the operation region 104 (the portion inside the boundary of the operation region 104 indicated by a broken line) is reflected in the captured image. Equivalent to. However, the extraction method of the arm region is not limited to this, and for example, a portion corresponding to the skin color region in the RGB image of the operation region 104 separately captured may be extracted. Further, the region detection unit 121 according to the present embodiment acquires the coordinates of the contour line based on applying a differential filter to the input image from which the arm region is extracted. Further, the region detection unit 121 calculates an average position of pixels where the arm region and the image edge are in contact (hereinafter referred to as an arm region intrusion position with respect to the operation region), and stores this in the RAM 111.

  In step S102, the position detection unit 122 detects one or more fingertip positions from the acquired arm region. In the present embodiment, the position detection unit 122 first identifies one pixel that is regarded as the tip of the finger based on the angle formed by the contour line of the hand region inside the region. However, the method for specifying the tip of the finger is not limited to the method based on the inner angle formed by the contour, and for example, the extreme value of the contour may be calculated. Further, the position detection unit 122 acquires a three-dimensional position including the height as the fingertip position information from the detected pixel position and pixel value information of the fingertip, and holds the information in the RAM 111. In step S103, the identification unit 125 identifies the left and right sides of the arm and the type of the fingertip based on the detected geometrical positional relationship of the fingertip. The identification result of the arm and fingertip is held in the RAM 111. Further, the type recognition unit 124 performs tracking of the arm and the fingertip. If it is difficult to recognize the type of arm or fingertip of the current frame, the recognition information up to the previous frame is used to supplement the recognition information of the arm and fingertip of the current frame. In step S <b> 104, the condition determination unit 124 calculates the speed of the fingertip's vertical movement from the variation in fingertip height between the previous frame and the current frame.

  In step S <b> 105, the condition determination unit 124 determines a touch recognition condition for each detected finger. The touch recognition condition is a condition for determining whether the fingertip has touched the touch target surface. In the case of the first embodiment, the touch threshold value and the release threshold value that are set in advance are read from the ROM 112 and the storage device 116 to be determined as conditions.

  In step S106, the recognition unit 126 performs a single touch recognition process. In the present embodiment, touch recognition is performed after narrowing down the fingers that the user is likely to use for the touch operation in the single touch recognition process. Details of the single touch operation recognition process will be described later. In step S107, the recognition unit 126 performs multi-touch operation processing. In the present embodiment, when there are a plurality of fingers recognized as being in the touched state in the process of step S106 per hand, they are paired, and in step S107, the multi-touch operation is recognized. However, if the paired fingertips do not perform an operation having a meaning as a multi-touch operation, the recognition result may not be output in step S107, and each may be recognized as a single touch operation. Details of the multi-touch operation recognition process will be described later. In step S108, the output control unit 127 outputs a response to the touch operation recognized in step S106 or step S107. For example, an image to be projected onto the table 101 is generated by the projector 118 and output. The above is the main processing flow in this embodiment.

<Determining whether the fingertip is regarded as a finger related to the operation>
Here, a method for determining whether the fingertip executed in the present embodiment is regarded as a finger related to the operation will be described. In the present embodiment, in the single touch recognition process executed in step S106, a process of narrowing down fingers that are highly likely to be used by the user for the touch operation is performed. First, with reference to FIGS. 4A to 4C, this narrowing performed in the present embodiment will be described.

  FIG. 4A shows a state in which the right hand is rested on the table surface 101 in a state where the right hand is maintained in a natural posture without intention to bend or extend a finger from the y-axis direction. FIG. Hereinafter, such a natural state of the user's hand is referred to as a neutral position. The five circles in the figure indicate the five fingertips from which the fingertip position information has been detected. The height of the fingertip compared to the touch threshold to recognize the touch is the z coordinate of the tangent parallel to the top surface of the circle and the x axis. In FIG. 4A, the middle finger 302 has the highest fingertip height, the ring finger 301 and the index finger 303 have the highest height, and the little finger 300 and the thumb 304 have the lowest fingertip height.

  Next, consider a case where the state of the user's right hand changes from the neutral position and the index finger 303 touches the touch target surface (Z = 0 surface). First, FIG. 4B shows a case where touch recognition is performed by applying the constant touch threshold 200 and the release threshold 201 described with reference to FIG. 3 to all fingers. When a certain touch threshold value 200 is applied to all fingers, the fingertip recognized as being in a touch state is represented by a black circle. In the case of FIG. 4B, although the user intends to touch with the index finger 303, the little finger 300 and the thumb 304 are also below the touch threshold. In this case, in addition to the index finger 303, the little finger 300 and the thumb 304 are also recognized as being in a touch state, and a TOUCH event is issued for each fingertip. Such misrecognition of an unintended fingertip touch may cause a UI part in an unintended location to react or cause a multi-touch operation to be recognized even though a single touch operation input is intended. It can be.

Here, FIG. 4C illustrates a case where it is determined whether the touch target surface is touched by narrowing down the fingertip to be compared with the touch threshold 200 based on the relative positional relationship of the fingertip according to the present embodiment. The postures of the hand and each fingertip shown in FIG. 4C are the same as those in FIG. In the present embodiment, the finger with the smallest height is identified from among the plurality of detected fingertips, and the relative height with the identified finger is used to determine the touch threshold value 200 among other fingertips. Narrow down the fingers for the touch determination process. In the present embodiment, since the finger with the smallest height is regarded as the finger that is most likely to be used for the touch operation, efficient narrowing down can be performed by using the height as a reference. In the case of FIG. 4C, the fingertip having the smallest height among the plurality of detected fingertips is identified as the index finger 303. For example, the height of the identified index finger 303 is set to H _min . Then, it is determined whether or not the height of the fingertip is lower than H _min + ε _H (ε _H is a positive constant) representing the narrowing threshold 305 among the plurality of detected fingertips. Narrow down the fingertips with small height. The narrowing-down threshold 305 is a relative height threshold based on the fingertip having the smallest height. In the present embodiment, it is considered that there is a high possibility that the fingertip narrowed down in this way is a finger that the user intends to use for input of a touch operation. On the other hand, a fingertip that is not included in the narrowing-down result is regarded as a fingertip that is not intended to be used for operation input. In the present embodiment, whether or not the touch input is actually performed is determined based on whether or not the height of the narrowed fingertip is below the touch threshold value 200.

In the example of FIG. 4 (C), the fingertip height of the index finger 303 is H _min, subject to touch recognition process by comparing the touch threshold 200 because below the _{H min} + ε H. Since the fingertip height H _min of the index finger 303 is below the touch threshold value 200, the index finger 303 is recognized as being in a touch state. Since the fingertip height of the fingers other than the index finger is higher than H _min + ε _H , the finger is excluded from the touch determination target by the touch threshold 200 and is not recognized as being in the touch state. In this manner, it is possible to prevent erroneous recognition of the little finger 300 and the thumb 304 that are brought into a touch state when erroneous recognition suppression is not performed as illustrated in FIG.

  It should be noted that narrowing down the fingertips to be subjected to touch recognition processing as described above is not equivalent to setting the touch threshold 200 to a small value in advance. This is because the touch threshold 200 needs to be set to be large to some extent so as to allow detection errors and blurring of position information caused by the accuracy and environment of the sensor. If the touch threshold value is decreased, there is a possibility that the information processing apparatus 100 may not recognize it although the user intends to touch it due to a detection error or a shake. Therefore, in this embodiment, the touch threshold 200 having a sufficient size is set, and the fingertips to be compared with the touch threshold 200 are narrowed down, thereby preventing recognition failure when the user is trying to touch.

Next, a method of narrowing down by comparing the movement speeds of the fingertips will be described with reference to FIGS. In FIG. 5A, the fingertip position of the previous frame is indicated by a broken-line circle, and the fingertip position of the current frame is indicated by a solid-line circle. Furthermore, an arrow connecting the fingertip height positions of the previous frame and the current frame indicates the magnitude of the speed in the Z-axis direction. The fingertip with the highest fingertip speed among all the fingertips is the index finger 303, and the fingertip speed is V _max . It is considered that the fingertip with a large relative fingertip speed exceeding V _max −ε _V (ε _V is a positive constant) representing the narrowing threshold is likely to be a fingertip used for touch operation, It becomes a target of touch recognition processing by the touch threshold 200. In FIG. 5A, the fingertip speed of the index finger 303 is V _max, which is higher than V _max −ε _V , so that touch determination based on the touch threshold 200 is performed. Since the fingertip height of the index finger 303 is below the touch threshold value 200, the index finger 303 is recognized as being in a touch state. Since the fingertip speed of the fingers other than the index finger is lower than V _max −ε _V , the finger is excluded from the touch determination target by the touch threshold 200 and is not recognized as being in the touch state.

  In the present embodiment, it is possible to obtain an accurate narrowing result by performing both the above-described narrowing based on the fingertip height and the narrowing based on the fingertip speed in stages. However, since these have their respective effects, one of them may be omitted depending on the system configuration and situation. Moreover, you may use both processing together in the form which takes OR of a result.

<Single touch operation recognition process>
Next, the single touch operation recognition process executed in the process of step S106 of the present embodiment will be described with reference to the flowchart of FIG. When the process of the flowchart of FIG. 9 is started, the processes of steps S201 to S214 described later are repeated for each of the fingertips whose position information is detected in step S102. At the stage where the processing for all the fingertips is completed, the single touch recognition processing ends, and the process returns to the flowchart of FIG.

  In step S <b> 201, the specifying unit 123 determines whether the fingertip that is a processing target at that time is regarded as a finger used for a touch operation. The process of step S201 corresponds to the narrowing down described with reference to FIG. Note that the determination in step S201 is based on information detected using a sensor that the user is likely to have the intention of using the fingertip to be processed for input of the touch operation. It is. Therefore, the determination result does not depend on the actual user's thoughts and intentions.

Here, the flowchart of FIG. 10 represents a flow of processing executed in step S201. In step S300, the specifying unit 123 compares the fingertip height and _{H min} + ε H. If the fingertip height is smaller, the process proceeds to step S301; otherwise, the process proceeds to step S303. In step S301, the identifying unit 123 compares the fingertip speed with V _max −ε _V. If the fingertip speed is larger, the process proceeds to step S302; otherwise, the process proceeds to step S303. In step S302, it is determined that the finger is used for the touch operation, and the determination result in step S201 is “YES”. As a result, the fingertip that is the processing target at this time is a processing target for determining whether or not the fingertip is in the touch state by comparing the height with the touch threshold. In step S303, it is determined that the finger is not used for the touch operation, and the determination result in step S201 is “NO”. If it is determined in step S302 that the finger is used for a touch operation (YES in step S201), the process proceeds to step S202. If it is determined in step S303 that the finger is not used for the touch operation (NO in step S201), the process proceeds to step S208.

  In step S202, the recognition unit 126 determines whether the z-coordinate (height) of the fingertip position information detected in step S102 is smaller than the touch threshold determined in step S105. If it is determined that the height of the fingertip is smaller than the touch threshold (YES in step S202), the process proceeds to step S203. If it is not determined that the height of the fingertip is smaller than the touch threshold (NO in step S202), the process proceeds to step S209.

  In step S <b> 203, the recognition unit 126 determines whether the fingertip to be processed is already in the touch state at this time. If it is in the touch state (YES in step S203), the process proceeds to step S207. If it is not a touch state, that is, if it is a non-touch state (NO in step S203), the process proceeds to step S204. In step S204, the recognition unit 126 determines that there has been a state transition to the touch state of the fingertip to be processed. In the present embodiment, the value of the touch flag held in the RAM 111 is set to “TRUE”, and the process proceeds to step S205.

  In step S205, the recognition unit 126 issues a TOUCH event. In step S206, the recognizing unit 126 holds information on the time when the transition to the touch state is confirmed in step S204 in the RAM 111 as the touch time. In step S207, the recognition unit 126 holds the position information of the fingertip position to be processed in the RAM 111 as the touch position at this time. The process is repeated until all the detected fingertips are processed.

  In step S208, the recognition unit 126 determines whether or not the fingertip to be processed is in a touch state at this time. If it is in the touch state (YES in step S208), the process proceeds to step S209. If it is not in the touch state, that is, if it is in the non-touch state (NO in step S208), the process of the flowchart of FIG. Alternatively, the process is repeated by changing the finger to be processed.

In step S209, the recognition unit 126 determines whether the fingertip height is greater than the release threshold. If it is determined that the fingertip height is greater than the release threshold (YES in step S209), the process proceeds to step S210. If it is not determined that the fingertip height is greater than the release threshold (NO in step S209), the process proceeds to step S207. In step S210, the recognition unit 126 determines the transition from the touch state of the target fingertip to the non-touch state, and sets the value of the touch flag held in the RAM 111 to “FALSE”. In step S211, the recognition unit 126 issues a RELEASE event. In step S212, the recognition unit 126 holds, in the RAM 111, information on the time when the transition to the non-touch state is confirmed in step S210 as the release time. In step S213, the recognition unit 126 determines whether the recognition condition for the tap operation is satisfied with respect to the fingertip that is the processing target, based on information obtained from when the touch event is issued until the release event is issued. Specifically, it is determined whether the condition of the following formula 1 is satisfied.
(Release time−touch time) <ε _T and (Distance between release position and touch position) <ε _D (ε _T and ε _D are positive constants) (Equation 1)
If it is determined that the condition for the tap operation is satisfied (YES in step S213), the process proceeds to step S214. If it is not determined that the tap operation condition is satisfied (NO in step S213), the process of the flowchart of FIG. 9 ends. Alternatively, the process is repeated by changing the finger to be processed. The single touch recognition process ends when the process is performed for all fingers.

<Multi-touch operation recognition process>
Next, multi-touch operation recognition processing executed in step S107 of this embodiment will be described with reference to the flowchart of FIG. In the present embodiment, the single touch recognition process is performed before the multi-touch operation recognition process, and the state of each fingertip is obtained.

  In step S400, the recognition unit 126 pairs a plurality of fingertips in a touch state. In the present embodiment, the two fingertips are sequentially paired along the touch time from the fingertip with the earliest touch time. However, this is an example, and other pairing methods may be used. When a plurality of combinations are paired in step S400, the processes in steps S401 to S406 below are repeated a plurality of times for each of the combinations.

  In step S401, the recognition unit 126 acquires a distance between two paired fingertips. In the present embodiment, a two-dimensional Euclidean distance on the xy plane is used, but another distance measure may be used. In step S402, the recognition unit 126 determines whether there is a difference between the acquired distance between the fingertips and the distance acquired in the previous frame. Here, the determination is made based on whether or not the difference in distance between frames is greater than a predetermined value. If it is determined that there is a difference in distance (YES in step S402), the process proceeds to step S403. If it is not determined that there is a difference in distance (NO in step S402), the process proceeds to step S404.

In step S403, the recognition unit 126 issues a PINCH event. The PINCH event is an event that is notified when an enlargement / reduction operation is recognized due to a change in the distance between two fingertips. The PINCH event is notified to the output control unit 127 together with information such as the enlargement / reduction ratio and the difference distance calculated from the change in the distance between the two fingertips. For example, the enlargement / reduction ratio (%) can be calculated by Equation 2.
(Current distance between fingertips) / (Distance between fingertips when paired) × 100 (Equation 2)
Further, for example, the difference distance can be calculated by (current distance between fingertips−distance between fingertips when paired). For example, the output control unit 127 can receive a notification of a PINCH event and can enlarge or reduce the size of the display item.

  In step S404, the recognizing unit 126 acquires the angle of the line segment connecting the paired fingertips. An example of an angle acquisition method will be described later. In step S405, the recognition unit 126 determines whether or not the calculated angle has changed compared to the previous frame. If it is determined that the angle has changed (YES in step S405), the process proceeds to step S406. If it is not determined that the angle has changed (NO in step S405), the process ends. Or it returns to step S401 and repeats.

In step S406, the recognition unit 126 issues a ROTATE event. The ROTATE event is an event that is notified when a rotation operation is recognized because the absolute angle of two paired fingertips has changed. As an absolute angle calculation method, for example, an angle formed by a line segment formed by connecting two fingertips and an x axis can be used, a clockwise angle can be positive, and a counterclockwise angle can be negative. The ROTATE event is notified to the output control unit 127 together with information such as a rotation ratio calculated from a change in absolute angle between two fingertips. For example, the rotation rate can be calculated by Equation 3.
(Absolute angles of the current two fingertips-Absolute angles of the two fingertips at the time of pairing) (Equation 3)
For example, the output control unit 127 can rotate the display item using information on the ROTATE event.

<Specific examples of multi-touch operation>
When the multi-touch operation is realized by touch recognition using the distance image sensor 115, it is necessary to recognize that a plurality of fingertips used for the operation are in a touch state. Also in recognition of multi-touch operation, if the fingertip of a finger that is not used for operation input is erroneously recognized as being in a touch state, it may cause a malfunction.

  First, when comparing the heights of all the fingertips with a certain touch threshold value 200 without performing the narrowing down of the present embodiment, problems that may occur when a multi-touch operation is input are shown in FIG. This will be described with reference to FIG. FIG. 7A shows a state in which the index finger 303 and the thumb 304 of the hand 103a are touched to perform a multi-touch operation on the UI component 102. FIG. The index finger and thumb are often used to perform enlargement / reduction operations and rotation operations, but for the user, only the index finger and thumb are stretched, and the posture of bending or changing the direction so that other fingers are not erroneously detected Taking can be burdensome. Therefore, as shown in FIG. 7A, the user is expected to have an environment in which a multi-touch operation can be performed with the hands and fingers naturally open.

  FIG. 6A shows the fingertip height in FIG. 7A mapped on the zx plane. The same elements as those shown in FIGS. 4 and 5 are given the same numbers, and the description thereof is omitted. In this case, it is the index finger 303 and the thumb 304 that the user intends to use for the operation input. However, since the other fingers are also spread, the user moves in conjunction with the movement of the index finger 303 and the thumb 304. In the case of FIG. 6A, the fingertip height of the little finger 300 is lower than the touch threshold 200 because the little finger 300 separated from the index finger 303 and the thumb 304 has been lowered in conjunction. Therefore, for example, the little finger 300 is erroneously paired with the thumb 304 or the index finger 303, and a multi-touch operation different from the user's intention may be recognized.

In the present embodiment, as described above, the fingertips to be subjected to the touch recognition process are narrowed down before performing the touch recognition process by comparing the touch threshold and the fingertip height. FIG. 6B shows the result of applying the narrowing of the present embodiment in the same situation as in FIG. In the hand state shown in FIG. 6B, the fingertip height of the index finger 303 is the smallest among the plurality of fingertips. The fingertip height is H _min . Since the index finger 303 and the thumb 304 used for operation input are below the height H _min + ε _H that is the narrowing threshold 305, they are targets for touch recognition. And since it is less than the touch threshold value 200, it is recognized as a touch state. On the other hand, although the fingertip height of the little finger 300 that is not a finger used for operation input is below the touch threshold value 200, it is higher than the height H _min + ε _H that is the narrowing threshold value 305. Is not compared. Accordingly, the touch state is not recognized.

Next, an example of fingertip narrowing based on the fingertip movement speed will be described with reference to FIG. In FIG. 6C, as in FIG. 5A, the fingertip position of the previous frame is indicated by a dashed circle, the fingertip position of the current frame is indicated by a solid circle, and the fingertip height position of the previous frame and the current frame are connected. The arrow indicates the magnitude of the speed in the Z-axis direction. The index finger 303 is the fingertip having the highest fingertip speed among all the fingertips. Let the fingertip speed be V _max . The fingertip speed is higher than the _{V max} -ε V is the index finger 303 and the thumb 304. Therefore, according to the present embodiment, the index finger 303 and the thumb 304 are targets for touch recognition processing, and since they are smaller than the touch threshold 300, it is recognized as being in a touch state. On the other hand, although the fingertip height of the little finger 300 is lower than the touch threshold value 200, the fingertip speed is lower than V _max −ε _V , so that it is not a target for touch recognition. Therefore, the little finger 300 is not recognized as being in a touch state.

  FIG. 7B shows a state in which the reduction operation is recognized and the UI component 102 is reduced due to the fingertip pairing being correctly performed according to the present embodiment. A PINCH event is issued by reducing the distance between the index finger 303 and the thumb 304, and the application refers to the enlargement / reduction ratio of the PINCH event and changes the size of the UI component 102. FIG. 7C shows a state in which a rotation operation is performed on the UI component 102. By rotating the index finger 303 and the thumb 304 clockwise, a ROTATE event is issued, and the application refers to the rotation rate of the ROTATE event and rotates the UI component 102 clockwise. As described above, in this embodiment, even when a multi-touch operation using only some fingers is performed, the multi-touch operation is recognized using appropriate finger information. It can be operated with a comfortable posture.

Note that the present embodiment has been described on the assumption that the user performs a multi-touch operation with one hand alone. Therefore, the comparison of height and speed was performed between a plurality of fingertips belonging to the same hand. However, the multi-touch operation may be performed with both hands. Further, there may be a case where an erroneous touch of the left hand is detected during the operation with the right hand. In order to cope with such a case, a combination of both hands of the same user may be specified in the extracted plurality of hand regions, and the height and speed may be compared among the fingertips belonging to both hands of one user. By doing so, it is possible to reduce misrecognition of the operation by estimating whether the user intends to use the operation for not only the finger but also the hand. Also, by comparing and narrowing down the heights and speeds of the fingertips of multiple users, it is possible to prevent an erroneous touch due to the fingertips of other users from being recognized when a user is actively operating. be able to. In the above-described example, two fingertips used for the multi-touch operation are paired, but this embodiment is also effective when three or more fingertips cooperate to perform the multi-touch operation.
In this way, when only the intended forefinger 303 and thumb 304 are touched, the two fingers are paired. Then, a multi-touch operation can be performed by moving the index finger 303 and the thumb 304 in a touched state.

<Touch operation to other than plane>
Up to this point, the touch operation for a plane has been described as an example. However, in the present embodiment, even if an object having an arbitrary shape is the target, the touch operation with the erroneous recognition suppressed can be performed by the same method. Is possible. As an example, FIG. 13A illustrates an example in which a touch operation is performed on a sphere 500 placed on a plane. A fixed touch threshold 401 and a release threshold 402 are set so as to cover the sphere 400. Then, the fingertip 403 is narrowed down with reference to the position information of the fingertip at which the distance 404 to the surface of the sphere 400 is minimum. Further, when acquiring the moving speed of the fingertip, the speed 405 in the direction in which the fingertip 403 moves toward the surface of the sphere 400 may be compared.

  Moreover, even if a virtual object is a target, it is possible to perform a touch operation in which erroneous recognition is suppressed by a similar method. An example of a touch operation on a sphere 406 that is a 3D object projected into the air by a 3D projector will be described with reference to FIG. Also in this case, the touch threshold value 407 and the release threshold value 408 are set so as to cover the sphere 406. Then, narrowing is performed based on the position information of the fingertip at which the milky beauty fingertip height 410 at which the distance 410 to the surface of the ball 406 of the fingertip 409 is the minimum distance is the minimum. Further, when acquiring the moving speed of the fingertip, the speed 411 in the direction in which the fingertip 409 moves toward the surface of the sphere 406 may be compared.

[Modification]
In the first embodiment described above, the condition for determining whether or not the narrowed fingertip is in the touch state is a predetermined touch threshold and release threshold set in advance. On the other hand, as a modification, an example in which a more appropriate threshold is determined for each user or for each finger will be described. Note that the touch threshold value determination process described in the following modification is performed in the process of step S105 in the flowchart of FIG. In the modification, the processes other than step S105 are the same as those in the first embodiment, and thus detailed description thereof is omitted.

<Modification 1: Threshold correction processing based on the relationship between fingers>
The first embodiment and its modification are intended to reduce erroneous recognition of touch operations caused by the influence of other fingers that move in conjunction with the fingers used for the operations. Here, depending on the type of finger used for the operation, it is possible to assume in advance a finger that tends to move in conjunction with the finger. For example, when the touch operation is performed with the little finger for the convenience of the user or the application, the ring finger adjacent to the little finger is likely to move in conjunction with the movement of the little finger. At that time, it is statistically understood that the height and speed are close to those of the little finger used for the input operation. Here, in Modification 1, the threshold value is determined based on such a relationship between fingers.

  FIG. 5B shows a state where the little finger 300 is touched. At this time, the ring finger 301 has a fingertip height and a fingertip speed that are substantially the same as those of the little finger 300, and therefore it may be difficult to exclude the ring finger 301 by narrowing down the first embodiment. Therefore, in the first modification, for the ring finger 301 and the middle finger 302 that have a strong tendency to move in conjunction with the little finger 300, the touch threshold value used when the little finger 300 is regarded as the finger most likely to be used for the touch operation, and The release threshold is corrected first. In the case of the first embodiment, the finger that is most likely to be used for the touch operation is the finger that has the shortest distance from the touch target surface and the fastest movement speed.

  In FIG. 5C, the condition for recognizing the touch of the ring finger 301 and the middle finger 302 becomes strict according to the fact that the little finger 300 is the closest finger from the touch target surface and the moving speed is fast. Represents the appearance. The touch threshold 306 of the ring finger 301 that moves in conjunction with the little finger 300 is determined to be smaller than the normal touch threshold 200. Similarly, the touch threshold value of the middle finger 302 is corrected from the threshold value 200 to the threshold value 307. Finally, by comparing the fingertip height and the fingertip speed, the touch determination is performed on the little finger 300 and the ring finger 301, but the touch is performed because the fingertip height of the ring finger 301 is not below the corrected touch threshold 306. It is not recognized as a state. Only the little finger 300 that is the intended touch is recognized as being in the touch state.

  FIG. 12A is a flowchart showing the flow of the threshold value determination process based on the relationship between fingers described above. This process is executed in step S105. In step S500, the touch recognition unit 126 identifies the fingertip having the minimum fingertip height, and determines the threshold value of the fingertip that moves in conjunction with the fingertip. In step S301, the touch recognition unit 126 identifies the fingertip with the highest fingertip speed, and determines the threshold value of the fingertip that moves in conjunction with the fingertip. In step S500 and step S501, for example, a corrected threshold value is calculated by adding or subtracting a predetermined correction amount to a preset touch threshold value, and determined as the threshold value of the target finger. However, the fingertip already determined in step S300 is not determined. As an example, the lookup table shown in FIG. 12B is used to acquire the fingertips that move in conjunction with each other and the correction amount in steps S300 and S301.

  In the first modification, the lookup table shown in FIG. 12B is used in order to acquire the fingertips that move in conjunction and the correction amount, but other methods are used to determine the fingertips that move in conjunction and the correction amount. It is also possible. For example, it can be determined probabilistically based on statistical information. In the first modification example, the finger type information is used to identify the fingertips that are linked and move easily, and the touch threshold is determined. However, even if there is no such information, the fingertips that move in conjunction with each other are determined. It is possible to specify. For example, it is possible to determine a fingertip adjacent to a fingertip having a low fingertip height or a high fingertip speed as a fingertip that moves in conjunction with the fingertip, and determine a threshold value for the fingertip.

<Modification 2: Threshold correction for each finger according to the user's neutral position>
In the first embodiment, a constant touch threshold and a release threshold are used for any user. However, the height of each finger in the neutral position (natural hand state) varies depending on the user. Therefore, a case will be described in which the threshold value is corrected based on the height of each finger when the user's hand is in the neutral position.

  FIG. 14B shows a state in which the threshold value is varied depending on the fingertip height at the neutral position. In the second modification, the condition determination unit 124 corrects the threshold value using the height information of each fingertip obtained in a state where it is confirmed that the hand is in the neutral position. For example, the processing of the flowchart of FIG. 8 is started after the user's hand is guided to the neutral position using a guidance display that projects the silhouette of the hand and allows the user to rest the hand on the silhouette. In this case, it is possible to recognize that the neutral position has been reached when five fingertips are detected on the silhouette and a certain time has elapsed.

  Then, the condition determining unit 124 compares the height of each fingertip of the hand in the neutral position with a preset height threshold value α. The height threshold α used here is different from the touch threshold and the release threshold, and it is considered that a finger at a higher position is less likely to be involved in the touch operation, and correction of the touch threshold is not necessary. It is the height of the standard for judging. In FIG. 14B, since the ring finger 301, the middle finger 302, and the index finger 303 have fingertip heights higher than a certain height α, the touch threshold value 200 and the release threshold value 201 are not changed as defaults. However, since the little finger 300 has a low fingertip height at the neutral position, the touch threshold value is reset to 510 and the release threshold value is set to 511 according to the height. Similarly, the thumb 304 also resets the touch threshold to 512 and the release threshold to 513. In the second modification, a natural touch operation with few misrecognitions can be performed by setting a threshold value according to the user.

  However, the threshold value may be changed according to the height of the finger at the neutral position for all the fingers without distinguishing by the predetermined height α. FIG. 14A shows an example in which a threshold is set according to the type of finger. The touch threshold of the little finger 300 is 500, and the release threshold is 501. Similarly, the touch threshold of the ring finger 301 is 502, and the release threshold is 503. The touch threshold of the middle finger 302 is 504, and the release threshold is 505. The touch threshold of the index finger 303 is 506, and the release threshold is 507. The touch threshold of the thumb 304 is 508, and the release threshold is 509. By matching the thresholds in detail in this way, it is possible to intuitively realize the same recognition rate and operational feeling at any fingertip.

<Modification 3: Threshold correction for each finger according to application>
As another modified example 3, a predetermined finger touch threshold value may be adjusted according to the type specified in step S103 without depending on the height at the neutral position. For example, depending on the application, it may be possible to realize a natural feeling of operation by easily recognizing a predetermined type of fingertip. A case where the index finger and the thumb are easily recognized in an application that frequently performs multi-touch operations using the index finger and the thumb and the other fingers are difficult to recognize will be described. FIG. 14C illustrates a state in which threshold values are set for the little finger 300, the ring finger 301, and the middle finger 302 so that the condition for detecting touch becomes severe. First, the touch threshold value 514 and the release threshold value 515 for the little finger 300, the ring finger 301, and the middle finger 302 are set low. Further, the touch threshold value 516 and the release threshold value 517 of the index finger 303 and thumb 304 are set higher to make it easier to recognize the touch.

  In addition, when performing a multi-touch operation, if fingers other than the operated one are bent, all five fingertips cannot be detected and a fingertip type recognition result may not be obtained. In such a case, even if a finger is not detected, an optimum threshold value can be set for each finger using the finger identification result in step S103. For example, in the case of the right hand, the first fingertip and the second fingertip from the left (right in FIG. 14D) when viewed from the intrusion position are considered to be the thumb and the index finger, respectively. It can be done easily. FIG. 14D shows a state in which the four fingertips 518, 519, 520, and 521 of the right hand are detected, but the type is unknown. Here, since it is considered that the fingertips 520 and 521 are the index finger and the thumb, a higher touch threshold 516 and a release threshold 517 are set, and a lower touch threshold 514 and a release threshold 515 are set for the fingertips 518 and 519. ing. Similarly, in the case of the left hand, the first fingertip and the second fingertip from the right when viewed from the intrusion position are considered to be the thumb and the index finger, respectively, so that the fingertips can be easily recognized.

  When the touch threshold is determined according to the first to third modifications described above, the overall accuracy of the touch operation recognition process can be further improved by determining the release threshold in accordance with the determined touch threshold. Also in the modified example, as described in the first embodiment, by using the determined touch threshold value and the release threshold value to narrow down the fingertip for determining whether or not it is in the touch state, the fingertip used for the operation is linked. Misrecognition of a touch operation caused by the movement of the user can be reduced.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 120 Image acquisition part 121 Area | region detection part 122 Position detection part 123 Specification part 124 Condition determination part 125 Identification part 126 Recognition part 127 Output control part

Claims (12)

An image acquisition means for acquiring an input image reflecting position information in a space where a person inserts a hand and performs an operation;
Position detection for detecting position information of each of a plurality of fingertips of the person's hand based on the position information reflected in a part of the input image acquired by the image acquisition means in which the person's hand is reflected Means,
Using the relationship between the plurality of fingertips detected by the position detection unit, a specifying unit that specifies a fingertip that is a target of a process of determining whether the predetermined target surface is touched among the plurality of fingertips;
Using the position information detected by the position detecting unit and the position information of the touch target surface for the fingertip specified by the specifying unit, a touch operation input on the predetermined target surface by the finger tip is recognized. Recognition means;
An information processing apparatus comprising: The specifying means specifies a fingertip that is closest to the predetermined target surface based on position information of the plurality of fingertips detected by the position detecting means, and based on a distance from the specified fingertip The information processing apparatus according to claim 1, further specifying a fingertip that is a target of the determination process. The specifying means specifies a fingertip that indicates that the speed of approaching the predetermined target surface is fastest according to position information of the plurality of fingertips detected by the position detecting means from the repeatedly acquired input image, 2. The fingertip as a target of the determination process is further specified based on a relative relationship between the speed of the specified fingertip and a speed at which another fingertip approaches the predetermined target surface. Or the information processing apparatus of 2.   The recognizing unit determines whether the predetermined target surface is touched by the fingertip based on a degree of proximity between each of the fingertips subjected to the determination process and the predetermined target surface. The information processing apparatus according to any one of claims 1 to 3, wherein the information processing apparatus is characterized in that: As a condition for determining the degree of proximity of the recognition means, further comprising a determination means for determining a threshold value,
When the recognizing means indicates that the position information of each of the plurality of fingertips detected by the position detecting means is closer to the predetermined target surface than the threshold value determined by the determining means, The information processing apparatus according to claim 4, wherein it is determined that the predetermined target surface is touched by a fingertip.   6. The recognition unit according to claim 1, wherein when there are a plurality of fingertips touching the predetermined target surface, the recognition unit recognizes a multi-touch operation input by the plurality of fingertips. The information processing apparatus described. The input image is a distance image captured at an angle of view looking down on the space from above, and each pixel value of the distance image includes position information in a direction corresponding to the height from the predetermined target surface. The information processing apparatus according to claim 1, wherein the information processing apparatus is reflected. The determination unit determines the condition based on a relative relationship of positional information of each fingertip with respect to a plurality of fingertips included in one person's hand extracted from the input image. The information processing apparatus according to any one of claims 1 to 7. An identification means for identifying a plurality of types of fingertips detected by the position detection means;
The information processing apparatus according to claim 1, wherein the recognition unit determines the threshold value for a specific type of finger identified by the identification unit. An image acquisition step of acquiring an input image reflecting position information in a space where a person inserts a hand and operates by an image acquisition means;
A position for detecting position information of each of a plurality of fingertips in the person's hand based on the position information reflected in a portion of the acquired input image in which the person's hand is captured by the position detecting means. A detection process;
A specifying step of specifying a fingertip that is a target of processing for determining whether the predetermined target surface is touched among the plurality of fingertips by using the relationship between the detected plurality of fingertips by an specifying unit;
Recognizing the touch operation input on the predetermined target surface by the fingertip using the position information detected in the position detection step and the position information of the touch target surface by the recognition means. A recognition process to
An information processing apparatus control method comprising:   A program for causing a computer to function as the information processing apparatus according to claim 1.   A computer-readable storage medium storing the program according to claim 11.

Images