JP2017084307A - Information processing device, control method therefor, program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device which, even when at least one of a plurality of indication positions constituting an operation being inputted cannot be detected due to the fact that a portion of an operation medium has entered the dead angle of imaging means, recognizes that a touch state is continued.SOLUTION: Provided is an information processing device 100 for recognizing input of an operation to an operation face 101 by the end part of an operation medium, the device comprising: a state acquisition unit which, when the end part of the operation medium exists within the range of spaces on the operation face in which the distance from the operation face 101 is smaller than a prescribed reference, acquires, from an image capturing the space on the operation face, information showing the state of the operation medium with respect to a distance image sensor 102 that has captured the image; and a recognition unit which, when it is assumed in association with the acquired information that the end part of the operation medium has entered the dead angle of the distance image sensor 102, recognizes that the end part of the operation medium is inputting an operation to the operation face using a reference different from the case when it is not assumed that a portion has entered the dead angle.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for recognizing an operator's operation based on a captured image.

  In recent years, in an environment such as AR (Augmented Reality) and MR (Mixed Reality), there is a scene where a touch operation is performed on an operation surface (virtual surface) that does not physically exist. In addition, there is a scene in which an operator interface (hereinafter referred to as UI) is projected onto an arbitrary operation surface such as a wall or a desk by projection, and a touch operation is performed on the projected UI. In such a scene, both are determined based on the distance between the operation surface and a part of a predetermined operation body (for example, the fingertip of the operator) using imaging means such as a stereo camera or a distance image sensor. It is often determined whether or not they are in contact. Specifically, a predetermined threshold is provided for the distance between the operation surface and the fingertip of the operator, and the case where the distance is less than the threshold is regarded as a state where both are in contact.

  In recognition of an operation based on the movement of a part of the body using an image, depending on the positional relationship between the imaging means and the operator, a part of the operator's body to be detected enters the blind spot of the imaging means, and the movement is detected. The problem is that it cannot be recognized correctly.

  With respect to the above problem, the technique disclosed in Patent Document 1 uses a three-dimensional model of the operator's hand to continuously track its position and posture, so that the hand position can be roughly determined even when entering the blind spot. It is possible to continuously recognize a spatial gesture operation based on the motion. Moreover, in patent document 2, the movement of the hand which was interrupted by the obstacle and temporarily disappeared from the image sensor is complemented by a movement locus registered in advance.

JP 2013-205983 A JP 2014-21944 A

  In Patent Document 1 and Patent Document 2, in an operation recognition method for interpreting an instruction associated with a shape of a trajectory drawn by an operating body, when a part of the trajectory is missing, the missing part is complemented to recognize the operation. It is possible to continue. However, regardless of which complementing method is used, it is difficult to accurately determine whether or not the operating tool that has entered the blind spot of the imaging means is in contact with the operating surface.

  Therefore, the present invention allows the touch state to be continued even when it becomes difficult to detect at least one of a plurality of designated positions constituting the input operation due to the influence of a part of the operating body entering the blind spot of the imaging means. The purpose is to be able to recognize that

  The present invention relates to an information processing apparatus for recognizing an input of an operation on an operation surface by an end of an operation body, wherein a distance from the operation surface in a space on the operation surface is smaller than a predetermined reference. An acquisition means for acquiring information representing a state of the operation body with respect to an imaging means for capturing the image from an image obtained by imaging a space on the operation surface; If the end of the operating body is considered to be in the blind spot of the imaging means in relation to the information acquired by the means, the end of the operating body is considered to be in the blind spot of the imaging means. Recognizing means for recognizing that the end of the operation body is inputting an operation on the operation surface, using a different standard from the case where the operation is not performed.

  According to the present invention, even when it becomes difficult to detect at least one of a plurality of designated positions constituting an operation being input due to the influence of a part of the operating body entering the blind spot of the imaging unit, the touch state is continued. It becomes possible to recognize that.

The figure showing an example of the definition of the appearance and position information of the system which uses information processor. The figure showing an example of the correspondence of the condition of the hand during a touch operation, and a hand region. 2 is a block diagram illustrating an example of a hardware configuration and a functional configuration of an information processing device. FIG. The figure showing the function part in the Example of this invention. The flowchart which shows an example of the flow of the process which recognizes touch operation. The flowchart which shows an example of the flow of a touch determination process. The flowchart which shows an example of the flow of the threshold control process of a touch determination. The figure which shows the operation example of single touch operation. The figure which shows the operation example of single touch operation. The figure which shows the operation example of multi-touch operation. The figure which shows the operation example of multi-touch operation. The figure which shows the method of the detection accuracy improvement which detects the state which the front-end | tip of an operating body is hidden. The figure explaining the relationship between the state of the user's finger | toe which is an operation body, and a system.

  Hereinafter, information processing according to an embodiment of the present invention will be described in detail with reference to the drawings. In addition, the structure described in an Example is an illustration and is not the meaning which limits the scope of the present invention to those structures.

<First Embodiment>
First, as a first embodiment, an example of processing for recognizing a touch operation performed by an operator on an item projected on a table surface of a table top interface system will be described.

[System configuration]
FIG. 1A is an example of the appearance of a tabletop interface system in which the information processing apparatus 100 according to the present embodiment is installed. The operation surface 101 is a table portion of the table top interface, and the operator can input a touch operation by touching the operation surface 101. However, in this embodiment, the touch operation means an operation that is input in a state in which a part of the operation body used by the operator is close enough to satisfy a predetermined standard with respect to a predetermined operation surface. It does not matter whether the operation surface is actually in contact with the operating body during the input of the touch operation. In the present embodiment, the distance image sensor 102 is installed above the operation surface 101 so as to look down at the operation surface. A distance image is an image in which information corresponding to the distance from the reference position (for example, the center of a lens, etc.) of the imaging means that captures the distance image to the subject surface imaged by the pixel is reflected in the value of each pixel. It is. In the present embodiment, the distance image sensor 102 captures the distance value from the distance image sensor 102 to the operation surface 101 or the object surface above the distance image. The distance image sensor 102 used in the present embodiment is a TOF sensor (Time-Of-Flight sensor). The TOF sensor receives light reflected from the surface of the subject and measures the phase lag to change the difference in time taken to reflect the light to the distance to the subject surface (distance in the depth direction). Detect as corresponding. In the distance image, distance information obtained as described above for each pixel is reflected as a pixel value. Further, the distance image sensor 102 obtains distance information of the distance in the depth direction, and therefore may be called a depth sensor. The distance image sensor 102 of the present embodiment is not limited to the TOF sensor, and for example, a distance image can be obtained by installing an infrared pattern irradiation type sensor or a stereo camera system. A distance image captured by the distance image sensor 102 is input to the information processing apparatus 100 as an input image. The information processing apparatus 100 acquires the three-dimensional position of the operator's hand 106 by analyzing the input image, and recognizes the input operation. Accordingly, the operator moves the space gesture by moving a predetermined object such as a hand within a range that can be imaged by the distance image sensor 102 in the space on the operation surface (the space between the operation surface 100 and the distance image sensor 102). An operation can be entered.

  In the present embodiment, the visible light camera 103 is installed so as to look down on the operation surface 101 from above. The information processing apparatus 100 can function as a document camera that controls the visible light camera 103 to capture an image of an object placed on the operation surface 101 and obtain a read image thereof. The information processing apparatus 100 detects and further identifies an object existing in the space on the operation surface 101 based on a visible light image obtained by the visible light camera 103 and a distance image obtained by the distance image sensor 102. The object includes, for example, an operator's hand, a paper medium, a document such as a book, and other three-dimensional objects. However, in the case of the system illustrated in FIG. 1A, the angle of view of the distance image sensor 102 and the visible light camera 103 does not include the operator's head around the table. Therefore, in the obtained input image, the image end portion intersects some part of the user's arm (portion beyond the shoulder).

  The projector 104 projects an image on the upper surface of the operation surface 101. In this system, the operator performs an operation by touch or space gesture on the item 105 included in the projected image. As described above, in the present embodiment, the distance image acquired by the distance image sensor 102 is used for detecting the hand 106 and recognizing the operation. By using the distance image, there is an advantage that even if the color of the operator's hand changes due to the projection light of the projector 104, the distance image is hardly affected. The display device of the present system can be configured by using the operation surface 101 as a liquid crystal display instead of the projector 104. In that case, even if a method of detecting a human hand from an image by detecting a skin color region from a visible light image, the hand can be detected without being affected by the projection light.

  If the image obtained by viewing the operation surface 101 from above is obtained, the distance image sensor 102 and the visible light camera 103 do not necessarily have to be installed above, for example, a mirror so as to face the operation surface. It may be installed and configured to take an image of the mirror. Similarly, in the example of FIG. 1A, the projector 104 projects onto the operation surface 101 so as to look down from diagonally above, but the projection light projected in different directions is used using a mirror or the like. It may be reflected on the operation surface 101. The same applies to the case where the operation surface 101 is installed along the vertical direction.

  In the present embodiment, the x, y, and z axes shown in FIG. 1 are defined in a three-dimensional space on the operation surface 101, and position information is handled. In the example of FIG. 1A, the point 107 is the origin of the coordinate axes. Here, as an example, the two dimensions parallel to the upper surface of the table are the xy plane, and the direction perpendicular to the table upper surface and extending upward is the positive direction of the z axis. In the present embodiment, the z-axis direction corresponds to the height direction in the world coordinate system. However, the present embodiment is also applied to a system in which a non-horizontal surface such as a whiteboard or a wall surface is used as the operation surface 101, a case where the operation surface has irregularities, or a virtual surface generated using MR. Is possible.

  FIG. 1B is a diagram illustrating the relationship between the distance image captured by the distance image sensor 102 and the system. An image 108 represents an example of the content of a distance image captured by the distance image sensor 102. In each pixel of the distance image, a value corresponding to the distance from the distance image sensor 102 to the subject is held as a pixel value. In the present embodiment, the distance image sensor 102 captures a distance image so that the operation surface 101 is looked down. Accordingly, each pixel value of the distance image includes information for obtaining position information (z coordinate) in the height direction from the operation surface 101. However, in the present embodiment, the distance image sensor 102 is installed so as to have an angle with respect to the z axis. Therefore, in this embodiment, the pixel value of the distance image is not used as it is as the z-coordinate, but the position information defined in the image is converted to the world by performing coordinate conversion using parameters according to the sensor and the installation environment. Used by converting to xyz coordinates. In the example of the distance image shown in FIG. 1B, the distance information reflected in each pixel value is omitted, and only the edge of the subject is clearly shown. The table 101 and the origin 107 correspond to FIG. A range 111 corresponds to an angle of view of the distance image sensor 102. Hereinafter, the image edge 111 is referred to.

  A hatched area 109 is an image of the operator's hand 106 in the distance image 108 (hereinafter simply referred to as the hand area 109). In the present embodiment, one designated position is detected for each detected hand. The designated position is a coordinate of a position estimated that the operator is pointing with a finger. In the present embodiment, as a premise, when the operator designates a point for a touch operation, it is prescribed to take a “pointing pose” with one finger extended. This is because a pose with one finger extended is a natural posture for many people to point to one point. Therefore, the position estimated to be the fingertip in the hand area 109 is specified as the designated position. Specifically, the hand region 109 is extracted from the distance image, and the coordinates indicating the pixel existing in the farthest position from the image end 111 in the hand region 109 are set as the designated position. Thereby, the fingertip position of the finger extended by one is specified as the designated position. In the case of the hand area 109 shown in FIG. 1B, the point farthest from the image end 111 is specified as the designated position 110. However, the method for specifying the designated position is not limited to this, and for example, the five fingers of the hand may be detected from the hand area, and the tip of a predetermined finger may be used as the designated position.

  As shown in FIG. 1B, a two-dimensional coordinate system based on the u axis and the v axis is set in the distance image. In the example of FIG. 1B, the resolution of the distance image is 640 [dot] × 480 [dot]. Assume that the position coordinate in the distance image of the designated position 110 is (u, v), and the pixel value corresponding to the distance from the distance image sensor 102 to the designated position is d. In the present embodiment, coordinate conversion based on the lens characteristics of the distance image sensor 102 and the relative positional relationship with the operation surface 101 is performed on the position information defined in the distance image in this way. As a result, the coordinates of each pixel can be mapped to a coordinate system in the real world defined in the table, and the three-dimensional position (x, y, z) in the real space can be acquired for the designated position 110. The transformation matrix used for coordinate transformation is acquired by performing adjustment work in advance when the distance image sensor 102 is installed. Note that the distance d used for calculating the three-dimensional coordinates is not limited to a single pixel value, but noise removal processing and averaging processing are performed on pixels of several pixels in the vicinity of the indicated position 110 in the hand region. And may be specified.

  In this embodiment, the center of the portion where the hand region 109 intersects the image end 111 in each frame is defined as the hand entry position. In the case of the hand region 109 in FIG. 1B, the intrusion position 112 is specified. The intrusion position 112 is also acquired as three-dimensional position information in real space by converting the position information defined in the distance image. In the present embodiment, when a plurality of hand regions are detected from the distance image, the indicated position and the intrusion position are specified for each of them. Then, the touch operation is recognized for each of the plurality of hand regions. In this specification, it is assumed that an operator's hand 106 and its finger are used as an example of an operation body and an end part thereof used for an input of a touch operation by the operator. However, in the present embodiment, the present invention can be applied not only to fingers but also instruments such as a stylus and a robot arm as the operating body. Note that the end portion of the operating body indicates a part used to point to the designated position for the touch operation. The end is not necessarily limited to the tip of the protruding portion as long as the end belongs to a part of the operation body and can be input by a touch operation.

  Here, with reference to FIG. 2, the problem which this embodiment solves is demonstrated anew. FIG. 2 shows an example of the correspondence relationship between the hand situation during the touch operation and the hand region. FIG. 2A is a view of the vicinity of the operator's hand 106 along the x-axis positive direction in the table top interface system shown in FIG. At this time, the operator's hand 106 extends the index finger and touches the operation surface 101. The information processing apparatus 100 specifies the position of the fingertip of the index finger as the designated position 200 based on the distance image captured from above, and the operator touches the operation surface 101 by threshold processing for the z coordinate of the designated position 200. It is determined that it is in a state. Specifically, when the z coordinate is smaller than a value representing the height from the operation surface 101 set as the threshold value, it is determined that the operation surface 101 is touched by the fingertip. Here, even if the operator's finger is not in physical contact with the operation surface 101, if the z coordinate is lower than the threshold value, all are considered to be in the touch state. This determination process is hereinafter referred to as touch determination. The operation surface 101 is touched by a fingertip, that is, the touch operation based on the instruction position that is the processing target of the information processing apparatus 100 is being accepted. Say there.

  In the system configuration of FIG. 1A, the distance image sensor 102 is installed obliquely upward with respect to the operation surface 101. Therefore, depending on the angle of the hand 106, as shown in FIG. Is hidden, and the fingertip enters the blind spot 201 of the distance image sensor 102. In order to reduce the size of the apparatus, it is necessary to limit the height at which the distance image sensor 102 is installed, and the installation height cannot be increased excessively. FIG. 2B shows the hand region 109 in the distance image obtained by the distance image sensor 102 capturing the state of FIG. Since the fingertip is in the blind spot, the indication position 202 identified as the point farthest from the image end 111 in the hand region 109 is near the base of the middle finger. As shown in FIG. 2A, the designated position 202 exists at a position higher than the actual fingertip position 200 by the height 203. Therefore, if the height (z coordinate) threshold used for determining whether or not the operator touches the operation surface 101 is smaller than the height 203, the operator intends to touch, but is not touched. The information processing apparatus 100 may have a problem that it cannot detect that it has been touched. In addition, even when detecting the five fingers of the hand from the hand region and specifying the tip of the predetermined finger as the designated position, the predetermined finger cannot be detected because the fingertip enters the blind spot 201, There may be a problem that the length is detected shorter than the actual length.

  Therefore, in the present embodiment, it is determined whether the fingertip is considered to be in the blind spot 201 of the distance image sensor 102 based on the state of the operating body with respect to the distance image sensor 102. In other words, it is determined whether the fingertip is considered to be detected as the designated position. And according to the result, by using different touch determination criteria, erroneous determination of touch determination is reduced even when the fingertip position 200 is in the blind spot 201 of the distance image sensor 102.

[Overview of judgment processing]
Here, based on the height of the designated position, the direction of the operator's arm in the distance image, and the direction from the imaging position to the designated position, executed by the information processing apparatus 100, the state of the operating body with respect to the distance image sensor The outline of the process of acquiring the blind spot and determining the presence or absence of the blind spot including the fingertip will be described.

  Specifically, the determination process in the present embodiment is executed according to the conditions shown in FIG. FIG. 13A shows the detected height of the designated position, the relationship between the imaging direction in which the distance image sensor 102 images the designated position and the direction of the operation body of the user as the operating body, and the determination result based on them. It is a table. In the present embodiment, first, when the height of the detected indication position is lower than the touch threshold (low enough to be considered to be in the touch state), it is considered that there is no blind spot including the fingertip. This is because, in the case where the fingertip enters the blind spot caused by another part of the same hand such as the back of the hand, which is the cause of the problem of the present embodiment, as shown in FIG. In many cases, the indicated position is detected at a position higher than the touch threshold. In other words, if the detected indication position is sufficiently low, it can be said that there is a high possibility that there is no blind spot including the fingertip.

  In the present embodiment, when the height of the detected indication position is higher than the touch threshold (not considered to be in the touch state), the fingertip is determined according to the relationship between the imaging position of the distance image sensor 102 and the direction of the operating tool. The presence or absence of blind spots that contain Specifically, the relationship between the imaging position of the distance image sensor 102 and the direction of the operating body is distinguished based on the difference in angle between the following two directions. One is a first direction that represents the direction of the operation body itself in the space on the operation surface 101. The other is a second direction representing the direction (imaging direction) when viewing the designated position from the imaging position, and is specified based on the imaging position of the distance image sensor 102 and the detected designated position. The However, the two directions are compared with the directions on the xy plane.

  FIG. 13B shows that the height of the detected pointing position is higher than the touch threshold value and the operating body has a sufficient angle difference between the first direction and the second direction. It is a figure explaining the relationship between a state and a system. It is assumed that the designated position 1301 is detected for the user's hand 1300. As an example, the center position of the image sensor 1302 is defined as an image pickup position serving as a reference in the image pickup direction 1304. As shown in FIG. 13B, in this system, when there is a sufficient angle difference between the direction 1303 (first direction) of the user's hand 1300 and the imaging direction 1304 (second direction), Obstacles due to parts other than the fingers of the hand 1300 are unlikely to occur between the user's fingertips. Therefore, it can be considered that there is no possibility that the blind spot has a fingertip.

  FIG. 13C illustrates the relationship between the state of the operating tool and the system when the height of the detected pointing position is higher than the touch threshold and the imaging direction and the direction of the operating tool itself are nearly parallel. FIG. However, the positional relationship between the imaging element 1302 and the designated position 1301 is the same as that in FIG. 13B, and the imaging direction 1304 (second direction) is also the same as in FIG. 13B. The difference between FIG. 13B and FIG. 13C is in the direction 1306 (first direction) of the user's hand 1305 as the operating tool. Also in FIG. 13C, the user's hand 1305 is in a pointing pose, but the fingertip is lower and the back of the hand is higher than that, and the image sensor 1302 is further upward, and these are substantially the same. Line up in a straight line. Accordingly, when the fingertip of the operator's hand is viewed from the imaging position, the back of the hand becomes an obstacle. As described above, in the present system, when there is no sufficient angle difference between the direction 1306 (first direction) of the operating body itself and the imaging direction 1304 (second direction), the hand is moved from the imaging position to the user's fingertip. Parts other than the 1300 finger are likely to be obstacles. Therefore, in this embodiment, in such a case, it is considered that there is a blind spot including a fingertip.

[Device configuration]
In the present embodiment, different touch determination criteria are used according to the determination results based on the rules shown in the table of FIG. Below, the structure of the information processing apparatus 100 which concerns on this embodiment is demonstrated. FIG. 3A is a hardware configuration diagram of a tabletop interface including the information processing apparatus 100 according to the present embodiment. The central processing unit (CPU) 300 is connected to the system bus 304 by using the RAM 302 as a work memory, executing an OS and a program stored in the ROM 301 and the storage device 303, performing various processing operations and logical determinations, and the like. Control each configuration. The storage device 303 is an external storage device connected by a hard disk drive or various interfaces, and stores programs and various data related to the operation recognition processing of the embodiment. The distance image sensor 102 captures a distance image of a space on the operation surface 101 including the table on which the item is displayed and the hand of the operator who operates the item under the control of the CPU 300, and the captured distance image is the system bus 304. Output to. In this embodiment, the distance image acquisition method will be described based on the TOF method that is less affected by ambient light and table surface display. However, a parallax method, an infrared pattern projection method, or the like may be used depending on the application. Is possible. The projector 104 projects and displays the image item to be operated on the table according to the control of the CPU 300.

  In the above-described system, the visible light camera 103, the distance image sensor 102, and the projector 104 are external devices connected to the information processing apparatus 100 via an input / output interface, respectively. Configure the processing system. However, these devices may be integrated with the information processing apparatus 100.

  FIG. 3B is an example of a block diagram illustrating a software configuration of the information processing apparatus 100. Each of these functional units is realized by the CPU 300 developing a program stored in the ROM 301 in the RAM 302 and executing processing according to each flowchart described below. The execution result of each process is held in the RAM 302. In addition, for example, when configuring hardware as an alternative to software processing using the CPU 300, it is only necessary to configure arithmetic units and circuits corresponding to the processing of each functional unit described here.

  The image acquisition unit 310 acquires a distance image captured by the distance image sensor 102 as an input image at regular intervals, and stores the acquired image in the RAM 302 as needed. The object acquired by the image acquisition unit 310 and exchanged with each functional unit is actually a signal corresponding to the image data, but in this specification, simply “acquire a distance image” or “acquire an input image”. ".

  The hand region extraction unit 311 performs threshold determination and noise reduction processing on each pixel of the input image acquired by the image acquisition unit 310, and extracts a hand region in the distance image. The hand region is a region in which the hand used by the operator as the operating body is shown in the input distance image. The position detection unit 312 detects the fingertip position of the operator's hand based on the contour information of the hand region extracted by the hand region extraction unit 311 and specifies the coordinate value as the designated position. In the present embodiment, coordinates indicating a pixel existing farthest from the image end 111 in the contour of the hand region are detected as the designated position. At this time, the position detection unit 312 according to the present embodiment detects the center of the portion where the hand region and the image edge 111 intersect as the entry position of the hand region.

  The state acquisition unit 313 acquires information representing the state of the operation body relative to the distance image sensor 102 from the distance image obtained by the distance image sensor 102 capturing an operation body existing in the space on the operation surface 101. In the present embodiment, information representing the direction (first direction) of the operating body in the space on the operation surface 101 and information representing the imaging direction (second direction) are acquired. The direction of the operating body itself in the space on the operation surface 101 is the direction of a straight line specified based on the intrusion position and the designated position detected by the position detection unit 312. Specifically, the designated position is determined from the intrusion position as a starting point. Vector information (first vector) as an end point is acquired. In the present embodiment, information indicating the direction of a straight line specified based on the imaging position of the distance image sensor 102 and the detected indication position is acquired as information indicating the imaging direction. Specifically, vector information (second vector) having the imaging position as the start point and the designated position as the end point is acquired. The second vector indicates the direction in which the designated position detected by the position detection unit 312 is viewed from the position defined as the imaging position of the distance image sensor 102. Note that when the distance image sensor 102 has an imaging system in which the light reflected from the subject surface is passed through one or more mirrors, the position of the mirror so that the rules shown in FIG. It is necessary to set an appropriate imaging position according to the number. 2B and 13C when at least the direction in which the pointing position is viewed from the mirror installed closest to the subject among the one or more mirrors and the direction of the operating body are nearly parallel. The same problem may occur. In order to solve the problem by adding the control of the reference for touch determination according to the present embodiment, it is necessary to determine the imaging position based on at least the mirror installed at the position closest to the subject.

  In the present embodiment, the information on the vector representing the two directions described above is handled as the information representing the state of the operating tool in the imaging range, but the state shown in FIGS. 13A and 13B is distinguished. However, the information is not limited to this as long as it can be used. For example, as the information indicating the direction, coordinates of two points that define a straight line instead of a vector, an expression, and information on a specific rotation angle with respect to a predetermined coordinate axis may be handled. Further, for example, information representing the direction of the operating body itself may be obtained based on the inertial principal axis of the hand region. In addition, information indicating the result of classifying the positional relationship between the operating tool and the distance image sensor 102 into a predefined positional relationship pattern, or coordinate information indicating each position itself may be handled.

  The determination unit 314 relates to the information representing the state of the operating body with respect to the distance image sensor 102 acquired by the state acquisition unit 313, and the portion used for the input of the touch operation in the operating body enters the blind spot of the distance image sensor 102. Determine if it is considered to be. In this embodiment, the part used for the input of touch operation among operation bodies is an edge part of an operation body. The determination unit 314 determines whether the end of the operating body is considered to be within the blind spot of the distance image sensor 102 generated by other parts other than the end of the same operating body. When the operating body is the operator's hand, the determination unit 314 is regarded as entering the blind spot of the distance image difference sensor 102 generated by a part other than the fingertip, such as the back of the hand, as shown in FIG. Determine whether. Specific processing of the determination unit 314 will be described later with reference to the flowchart of FIG.

  The reference determination unit 315 is a reference used for touch determination processing executed by the recognition unit 316 to determine whether the end of the operating body is inputting a touch operation to the operation surface 101 (whether it is in a touch state). To decide. In this embodiment, when the determination unit 314 determines that the fingertip is considered to be in the blind spot of the distance image sensor 102, it is determined that the fingertip is not considered to be included in the blind spot of the distance image sensor 102. Touch determination is performed using a different standard. The reference used for touch determination is a threshold value to be compared with the height (z coordinate) of the designated position. In this embodiment, the threshold used when determining whether the end of the operating body touches the operation surface 101 is the touch threshold, and whether the end of the operating body that has touched the operating surface 101 is separated from the operation surface 101. The threshold used for determination is called a release threshold. The reference determination unit 315 considers that the end of the operating body is in the blind spot as a reference used when it is determined that the end of the operating body is in the blind spot of the distance image sensor 102. If not, determine a threshold value that is greater than the criterion used.

  The recognizing unit 316 recognizes a touch operation input to the operation surface 101 by the operation body identified as being in the touch state by the touch determination process, and touch event information indicating the touch operation responds to an input from the display control unit 317 or the like. To the function unit or application. In the present embodiment, at least three events of “TOUCH”, “MOVE”, and “RELEASE” are issued by the recognition unit 316 and notified to the display control unit 317 according to the condition that the locus of the touched indication position satisfies. The “TOUCH” indicates that the touch of the operation body that is not in the touch state has started, “MOVE” indicates that the operation body has moved while maintaining the touch state, and “RELEASE” indicates that the touch body is in the touch state. This indicates that the operation body that was present is in a non-touch state. Note that the touch event described in the present embodiment is an example, and touch events to be notified can be added or reduced according to the use of the information processing apparatus 100 or the sensor used.

  The display control unit 317 uses the information stored in the ROM 301 or the storage device 303 to generate an image to be projected on the operation surface 101 by the projector 104 that is the display unit of the present embodiment. For example, based on the recognition result of the recognition unit 316, at least a part of the image displayed on the display unit is subjected to deformation such as movement, enlargement / reduction, rotation, and the like, and is output to the display unit. The output image is projected and displayed on the operation surface 101 by the projector 104. The projected image includes a plurality of items, and may be moved, enlarged / reduced, and rotated for each item, or the entire image may be a target of movement, enlargement / reduction, and rotation.

[Touch operation recognition process flow]
The flowchart in FIG. 4 represents the flow of touch operation recognition processing executed by the information processing apparatus 100 according to the present embodiment. In the present embodiment, it is assumed that the processing of the flowchart in FIG. 4 is started in response to the information processing apparatus 100 being activated.

  In step S <b> 400, the display control unit 317 acquires image information to be displayed on the operation surface 101 from the ROM 301 or the like and outputs the information to the projector 104. The projector 104 projects an image on the operation surface 101 based on the output information. The image projected here is, for example, an initial screen in which operation menus are listed, a screen representing a final state when the information processing apparatus 100 is operated last time, or the like. In step S <b> 401, the image acquisition unit 310 initializes a work area in the RAM 302, acquires a distance image captured by the distance image sensor 102 as an input image, and stores the input image in the RAM 302.

  In step S <b> 402, the hand region extraction unit 311 extracts one or more hand regions from the input image, and holds the information in the RAM 302. In this embodiment, by performing threshold processing on the z-direction coordinate value indicated by each pixel of the input image, an area having a coordinate value higher than the operation surface 101 is extracted as a hand area. However, the extraction method is not limited to this, and for example, the RGB value of a visible light image captured separately may be analyzed to extract a portion corresponding to the skin color region.

  In step S <b> 403, the position detection unit 312 refers to the hand region information held in the RAM 302, and identifies the point farthest from the image end 111 as the designated position for each hand region extracted in step S <b> 402. Furthermore, one point corresponding to the center of the intersection between the image edge 111 and the hand area 109 is specified as the entry position. Then, position information (u, v) in each input image is held in the RAM 302. In step S404, the position detection unit 312 converts the position information (u, v) of the designated position and the intrusion position specified in step S403 into position information (x, y, z) in the real world.

  In step S405, the determination unit 314 and the reference determination unit 315 execute processing for determining a reference used for touch determination. In the case of the present embodiment, a selection is made as to which of a plurality of patterns of touch threshold values and release threshold values held in advance in the ROM 301 is to be used. Details of the processing in step S405 will be described later with reference to the flowchart of FIG.

  In step S406, the recognition unit 316 performs touch determination according to the determined criterion. That is, it is determined whether the height (z coordinate) of the designated position is smaller than the determined touch threshold. If it is determined that the height of the indicated position is smaller than the touch threshold (Yes in step S406), the process proceeds to step S407. If it is determined that the height of the indicated position is equal to or greater than the touch threshold (No in step S406), the process proceeds to step S414. However, which determination result includes the case where the height of the designated position matches the touch threshold may be appropriately selected according to the installation status of the system.

  In step S407, the recognizing unit 316 refers to the information stored in the RAM 302 and determines whether the indicated position that has already been processed has transitioned to the touch state in the process up to immediately before. If it is determined that the touch state has not been changed (No in step S407), the process proceeds to step S408. If it is determined that the touch state has already been transitioned (Yes in step S407), the process proceeds to step S411.

  In step S408, the recognizing unit 316 changes the state of the designated position to be processed to the touch state, and causes the RAM 302 to store information indicating the transition. In step 409, the position information (x, y, z) of the designated position is held in the RAM 302 as a reference position for determining whether or not the designated position is moved in the touch state. At this time, for each touch determination criterion used in step S407, a reference position for determining the presence or absence of movement is held. In the present embodiment, as described later, there are cases where the reference for touch determination is a first touch threshold that is a default value, and a second threshold that is larger than the first threshold. Therefore, two reference positions for determining the presence or absence of movement are retained: position information associated with the first threshold value and position information associated with the second threshold value at the maximum. In step S410, the CPU 300 generates a touch event “TOUCH” and notifies each functional unit related to the response of the operation. In the present embodiment, at least the display control unit 317 is notified.

  On the other hand, if it is determined in step S407 that the state has already been changed to the touch state, in step S411, the recognition unit 316 determines whether the movement distance of the indicated position from the reference position is greater than a predetermined distance threshold. . In this embodiment, the presence / absence of movement of the designated position that is repeatedly detected at a predetermined detection cycle is determined based on the distance that the designated position has moved during the two most recent detections. The threshold of the predetermined distance is a value that is appropriate for distinguishing between movement of the indicated position that is caused by the operator's intentional movement of the indicated position and blurring caused by a detection error of the system. Is set. In step S411, the movement distance of the designated position is obtained by obtaining the distance between the position information held in the ROM 301 as the reference position and the position information of the latest designated position specified in step S403. Compare with However, the determination in step S407 is performed using a reference position associated with the reference for touch determination used in the immediately preceding process in step S407. For example, in a frame in which a touch state is detected using the first threshold as a reference for touch determination, the moving distance from the reference position associated with the first threshold to the latest designated position is compared with a predetermined distance threshold. . If it is determined that the movement distance of the indicated position is greater than the predetermined distance threshold (Yes in step S411), the process proceeds to step S412. When it is determined that the movement distance of the indicated position is equal to or less than the predetermined distance threshold (No in step S411), the process proceeds to step S417.

  In step S412, the recognition unit 316 updates the position information held as the movement reference position with the position information of the latest designated position specified in step S403. That is, in the present embodiment, the presence / absence of movement of the indicated position repeatedly detected at a predetermined detection cycle is determined based on the distance that the indicated position has moved between the two most recent detections. In step S413, the recognition unit 316 generates a touch event “MOVE” and notifies each function unit related to the response of the operation. In the present embodiment, at least the display control unit 317 is notified.

  In step S414, the recognition unit 316 determines whether the height of the designated position is greater than the release threshold. The release threshold is a height (z-coordinate value) threshold for determining that the designated position is sufficiently away from the operation surface. When the designated position is larger than the release threshold, the touch operation is terminated. It is determined that That is, the information processing apparatus 100 considers that the operator has clearly lifted his / her finger from the operation surface when the detected height of the indicated position is greater than the release threshold. In the present embodiment, it is assumed that the touch threshold value and the release threshold value are different, and the release threshold value is set to be larger than the touch threshold value. With such a setting, the operator can be clearly identified even if the height blur due to a detection error or the like may exceed or fall below the touch threshold after the designated position to be processed transitions to the touch state. The touch operation can be continuously recognized unless the finger is released from the operation surface. However, the touch threshold value and the release threshold value can be set to the same value. If it is determined that the height of the indicated position is greater than the release threshold (Yes in step S414), the process proceeds to step S415. If it is determined that the height of the indicated position is less than or equal to the release threshold (No in step S414), the process proceeds to step S417. It should be noted that which determination result includes the case where the height of the designated position matches the release threshold may be appropriately selected according to the installation status of the system.

  In step S415, the recognizing unit 316 changes the state of the designated position to be processed to the release state, and stores information indicating the state in the RAM 302. In step S416, the recognition unit 316 generates a touch event “RELEASE” and notifies each functional unit related to the response of the operation. In the present embodiment, at least the display control unit 317 is notified. In step S417, the image projected on the operation surface is updated by the display control unit 317. Specifically, a display image corresponding to the notified touch event is generated and output to the projector 104. In step S418, the CPU 300 determines whether a predetermined end operation such as turning off the power of the information processing apparatus 100 has been input. If it is determined that the end operation has been input (Yes in step S418), the information processing apparatus 100 ends all the processes. If it is determined that the end operation has not been input (No in step S418), the process returns to step S401, and the process is repeated for the newly acquired distance image.

  Next, the flow of touch determination criterion determination processing executed in step S405 will be described with reference to the flowchart of FIG. In step S <b> 500, the determination unit 314 determines whether the detected height of the indicated position is greater than the threshold value H. The threshold value H represents the position of the fingertip part of the operator who is performing the touch operation, or the position of another part (for example, the base of the finger) that is not the fingertip. This is a threshold value that is set to determine whether or not there is a high possibility. When the coordinates specified as the designated position represent the position of the fingertip portion of the operator who is performing the touch operation, it is assumed that the height of the designated position is as close as possible to the operation surface. On the other hand, as shown in FIG. 2 (a) and (b) as a problem, when the part that is not the fingertip is identified as the point farthest from the intrusion position in the hand region because the fingertip has entered the blind spot, The height is about the length of the tilted finger than the height at which the fingertip is present. The threshold value H is a height threshold value provided to identify such two situations, and an appropriate value is set in advance according to the installation environment of the information processing apparatus 100. When it is determined that the detected height of the indicated position is greater than the threshold value H (Yes in step S500), the process proceeds to step S501. Step S501 proceeds to a situation where the fingertip has entered the blind spot as shown in FIGS. 2A and 2B, or the fingertip has been detected correctly, but the fingertip is at the threshold H. It is a case where it exists in a higher position. When it is determined that the height of the detected indication position is equal to or less than the threshold value H (No in step S500), the process proceeds to step S507. The process proceeds to step S507 when there is a high possibility that the problem described in FIGS. 2A and 2B has not occurred. However, which of the determination results includes the case where the height of the designated position matches the threshold value H may be appropriately selected according to the installation status of the system.

  In step S501, the determination unit 314 indicates the direction from the intrusion position specified in step S403 to the designated position as information indicating the direction of the operating body itself used for the touch operation in the space on the operation surface 101. One vector is generated.

  In step S <b> 502, the determination unit 314 represents the direction from the imaging position to the designated position specified in step S <b> 403 as information indicating the direction of viewing the end portion used for the touch operation of the operating body from the distance image sensor 102. A second vector is generated. The imaging position is position information representing a reference of a range that is a field of view of the distance image sensor 102 at the time when the input distance image is captured, and is acquired as two-dimensional coordinates (x, y) in real space. The More specifically, it is preferably a value that specifies a position corresponding to the center of the image sensor of the distance image sensor 102 on the xy plane. Note that the second vector can be specified by a method other than using the position information of the indicated position as long as it can roughly represent the direction of viewing the portion used for the touch operation of the operating body from the imaging position. I do not care. For example, the wrist or the back of the hand may be extracted from the hand region, and the direction of viewing the position from the imaging position may be specified as the second vector. Due to the structure of the human hand, the positional relationship between the wrist and back of the hand and the finger is fixed, and the positions are close.Therefore, even when the position of the wrist or back of the hand is used, the indicated position is set as the direction of viewing the operating body from the imaging position. Information in a direction close to that used can be obtained.

  In step S503, the determination unit 314 acquires an angle difference between the first vector generated in step S501 and the second vector generated in step S502. In the present embodiment, a difference between angles (rotation angles around the z axis) generated in the xy plane is acquired. In step S504, the determination unit 314 determines whether the angle difference acquired in step S503 is smaller than the threshold value D. The threshold D is a threshold of a predetermined angle, and is a reference for determining whether the operator's arm itself can be a factor that creates a blind spot of the distance image sensor 102 in the visual field of the distance image sensor 102. is there.

  In the present embodiment, when the angle difference between the first vector and the second vector is smaller than the threshold D, that is, when the first vector and the second vector are closer to a parallel state than the reference angle, the fingertip of the operator is a distance image. It is determined that there is a high possibility that the sensor 102 is in the blind spot. As shown in FIG. 13C, the state in which the first vector and the second vector are almost parallel means that when the fingertip of the operator's hand is viewed from the imaging position, the operator's arm is the imaging position and the fingertip. It is a state that is highly likely to exist between. Furthermore, human arms and arms have joints, and when the operator lowers the height of the fingertip for the purpose of contacting the operation surface, the part other than the fingertip (for example, the back of the hand or the wrist) is relatively To be high. Therefore, when viewed from the distance image sensor 102 installed above, the posture in which the operator tries to touch the operation surface with the fingertip is a posture in which an obstacle easily occurs between the fingertip. Therefore, in the present embodiment, a state in which the operator's fingertip is highly likely to be in the blind spot of the distance image sensor 102 is detected using the above-described two vectors and the angle threshold. When it is determined that the acquired angle difference is smaller than the threshold value D (Yes in step S504), the process proceeds to step S505. When the acquired angle difference is greater than or equal to the threshold value D (No in step S504), the process proceeds to step S507. However, which determination result includes the case where the acquired angle difference matches the threshold value D may be appropriately selected according to the installation status of the system.

  In step S505, the display control unit 317 notifies the operator that the end of the operating tool is in the blind spot. Specifically, when the operation body is the operator's hand 106, the fact that the fingertip is present in the blind spot of the sensor or that the possibility is high is displayed by an alert or the like in the projected image, for example. .

  The notification is performed here to prompt the operator to change the posture. As shown in FIG. 2B, when the fingertip has entered the blind spot 201, there are a plurality of pixels having the same distance from the intrusion position 112 among the pixels constituting the edge of the hand region. Therefore, the intrusion position 112 may change from frame to frame. Further, even when the touch state is continued, the height of the hand changes when the hand posture is changed. For example, when the pointing position 202 is used, the accuracy of the touch determination process is less than when the fingertip position 200 is used. It becomes stable. Therefore, if the operator is prompted to change the posture and the fingertip can be detected from the distance image after the next frame, the touch determination can be performed stably.

  In step S506, the reference determination unit 315 changes the reference for touch determination. In the present embodiment, information serving as a reference option for touch determination is stored in advance in the ROM 301 or the storage device 303 and read out to the RAM 302. Specifically, at least a plurality of pieces of information including combinations of touch threshold values and release threshold values are stored, including combinations set as default values. In the case of the present embodiment, the default value sets a threshold value for touch determination to be used when the designated position corresponds to the position of the fingertip of the operator's hand. That is, the first touch threshold value for determining whether the height of the fingertip is considered to be a level at which the fingertip is considered to be in contact with the operation surface, and the height of the finger are considered to be sufficiently away from the operation surface. The first release threshold value for determining whether or not the degree is the reference default value for the touch determination. In step S506, a combination of thresholds to be used when the detected pointing position corresponds to the vicinity of the base of the operator's finger among a plurality of combinations of the touch threshold and the release threshold is set as a reference for touch determination. Is done. That is, when the height of the fingertip is at the height of the first touch threshold and the first release threshold, the second touch threshold and A second release threshold is set and used as a reference for touch determination. Appropriate values may be set for the values of the second touch threshold and the second release threshold based on experimental values, results of statistical analysis, and the like in advance. Note that the second touch threshold and the second release threshold need not be set at a height near the base of the finger. For example, it may be at the center of the back of the hand or near the center of the wrist.

  On the other hand, in step S500, when the height of the designated position is equal to or less than the threshold value H, in this embodiment, in step S507, the reference determination unit 315 sets a default value as a reference for touch determination. In the present embodiment, the combination of the first touch threshold and the first release threshold described above is adopted among the combinations of the plurality of touch thresholds and the release threshold.

  If the reference for touch determination is set in step S506 or step S507, the process returns to the process of the flowchart of FIG. 4 and the processes after step S406 are executed. The above is a series of flow of the touch operation recognition process in this embodiment. In the present embodiment, the reference for touch determination is based on the determination process regarding the height of the detected indication position (step S500) and the determination process regarding the direction of two vectors within the field of view of the distance image sensor 102 (step S504). Switch. The above two determinations correspond to processing for determining whether the end of the operating tool is considered to be within the blind spot of the distance image sensor 102 generated by the operating tool itself.

<Operation example 1>
Next, an example of a situation in which the table top system of FIG. In the following operation example, it is assumed that the threshold value H matches the default touch threshold value (first touch threshold value). The first touch threshold is denoted as T1, and the second touch threshold is denoted as T2. Each variable is set such that T1 = 20 mm and T2 = 100 mm, for example. The release threshold is set to a value that is 10 mm larger than each touch threshold. Further, it is assumed that the threshold value D of the angle used for the determination process in step S504 is 30 degrees.

  First, as an operation example 1, a situation where the operator slides a finger while touching the operation surface will be described with reference to FIGS. 6 and 7. 6A and 6B show a situation where the operator is touching the operation surface 101 with the index finger of the right hand and the fingertip of the operator can be imaged by the distance image sensor 102. FIG. 7 (a) and 7 (b), after the situation shown in FIGS. 6 (a) and 6 (b), the operator slides his / her finger in the negative direction of the x axis while touching the operation surface 101 with the index finger of the right hand. By doing so, it represents the situation where the fingertip of the operator has entered the blind spot of the distance image sensor 102. FIG. 6A and FIG. 7A schematically show a state of viewing the xy plane along the negative direction of the z-axis. A broken line 600 indicates a position corresponding to the image end 111 in the distance image on the xy plane. FIG. 6B and FIG. 7B schematically show a state of viewing the zy plane along the positive direction of the x axis. In FIGS. 7A and 7B, a region 700 represents a portion that is a blind spot of the distance image sensor 102.

  In the case of FIG. 6A, regarding the hand region extracted from the distance image, p1 (x11, x11, which corresponds to the center of the portion where the hand 106 and the broken line 106 intersect is obtained as the intrusion position of the hand 106 by the processing of step S403 and step S404. y11) is specified. Further, p2 (x21, y21, z21) that is the farthest point from the entry position p1 in the hand region is specified as the designated position. Further, it is assumed that p3 (x31, y31) is position information held in advance as an imaging position of the distance image sensor 102.

  As shown in FIG. 6B, the height z2 of the designated position p2 is smaller than the height 601 that is the threshold value H = T1 (No in step S500). Therefore, the reference for touch determination is the first touch threshold and the first release threshold which are default values. Since the height z2 of the designated position is smaller than the touch threshold value T1 (Yes in step S406), it is determined that the designated position p2 is in the touched state in the states of FIGS. 6A and 6B. If the touch state has not been changed by this time (No in step S407), the designated position p2 is changed to the touch state (step S408). p2 (x21, y21, z21) is held as the reference position for the moving operation (step S409), the touch event “TOUCH” is notified to the display control unit 317 (step S410), and control of the projection image is performed. (Step S417).

  Then, when the distance image is acquired, it is assumed that the situation has changed as shown in FIG. In the case of FIG. 7A after the operator's hand 106 has been moved, p4 (x12), which is the center of the portion where the hand 106 and the broken line 106 intersect, is determined as the entry position of the hand 106 by the processing of step S403 and step S404. , Y12) is specified. In the state of FIG. 7B, a blind spot 700 is generated because the field of view of the distance image sensor 102 is blocked by a part of the hand itself (the operation body itself). At this time, since the extended finger cannot be detected from the distance image as shown in FIG. 2B, the point p5 (x22, y22, z22) that is the farthest from the entry position p4 in the hand region is indicated. Identified as a location. Since p3 (x31, y31) is fixed, it does not change.

  As shown in FIG. 7B, at this time, the height z22 of the designated position p5 is larger than the height 601 that is the threshold value H = T1 (Yes in step S500). Accordingly, in the present embodiment, the first vector V1 having the entry position p4 as the start point and the designated position p5 as the end point and the vector V2 having the imaging position p3 as the start point and the designated position p5 as the end point are generated (steps S501 and S502). Assume that the angle difference between the two generated vectors is 18 ° (step S503). The acquired angle difference 18 ° is smaller than the threshold D = 30 ° (Yes in step S504). Therefore, in this embodiment, the operator is notified that the end of the finger is in the blind spot of the distance image sensor 102 (step S505). For example, characters such as “the fingertip cannot be seen from the sensor” are added to the projected image, or a red object is superimposed on the hand and projected. Then, the reference for touch determination is changed (step S506). In the present embodiment, a second touch threshold (threshold T2) and a second release threshold are set (step S506).

  The height z22 of the designated position p5 is smaller than the height 701 that is the touch determination threshold T2 (Yes in step S406). Since it is already in the touch state (Yes in Step S407), in Step S411, the distance from the reference position to the current designated position p5 is compared with the threshold value M for the move determination. However, the reference position held at this time is only the information of p2 held when the threshold T1 is used as the reference for touch determination, and the reference position associated with the threshold T2 used immediately before is held. It has not been. The distance from p2 to p5 cannot be regarded as the distance that the user's fingertip has moved. In this embodiment, the moving distance is not calculated in such a case. Therefore, in the situation of FIG. 7, since the movement larger than the threshold value M has not occurred, the determination result of step S411 is No. As a result, the “MOVE” event is not notified, and the process ends. The above processing is repeatedly executed until an operation end instruction is input to the information processing apparatus 100.

  In the present embodiment, as shown in the examples of FIGS. 6 and 7, when the fingertip enters the blind spot during the touch operation started with the fingertip being detected, the touch state is set. However, the determination of whether or not a move has been performed as an operation is avoided. This is because when the end of the operating body enters the blind spot, the user is notified of this in step S505, and it is assumed that the user who has received it takes measures such as changing the posture. . Under this premise, in this embodiment, even if several frames of the state where the indicated position is detected at a position other than the fingertip as shown in FIG. 7 occurs, it is corrected after that and the fingertip is detected again. I think that there is a high possibility of becoming. Therefore, even if the fingertip has entered the blind spot, the recognition of the touch state continues, so that the fingertip can be detected again in the next frame or later without forcing the user to redo the operation. The continuation of the operation can be recognized.

  As shown in the operation example 1, in this embodiment, when the end of the operating body enters the blind spot of the distance image sensor 102 generated by the operating body itself, the height of the indicated position detected is the actual end of the end. It is determined whether the end is considered to have entered the blind spot by using the fact that the height is higher than the height of. Further, in order to determine whether the end of the operating body has entered the blind spot of the distance image sensor 102 generated by the operating body itself, the first vector corresponding to the direction of the operating body itself and the direction in which the operating body is viewed from the imaging position The second vector corresponding to is used. In the present embodiment, when the detected pointing position is higher than a predetermined height and the angle difference between the first vector and the second vector is smaller than a predetermined angle threshold, the end of the operating body is It is determined that the blind spot of the distance image sensor 102 generated by the body itself has been entered.

  Note that the hand 106 shown in FIG. 6A exists at a position where the height z21 of the pointing position p2 corresponding to the fingertip is larger than the height 601 where the threshold value H = T1 without changing the position information on the xy plane. In this case, the processing to be executed is as follows. Note that the case where the height of the hand 106 shown in FIG. 6A is high is an example in which the fingertip is accurately detected but the determination result in step S500 is Yes. Since the determination result in step S500 is Yes, the first vector V1 in the direction from p1 to p2 is generated (step S501), and the second vector V2 in the direction from p3 to p2 is generated (step S502). It is assumed that the angle difference between V1 and V2 is acquired as 35 degrees (step S503). Since the angle difference is larger than the threshold value D (No in step S504), a default value is set as a reference for touch determination (step S507). As described above, in the present embodiment, even when the height of the detected indication position is higher than the threshold value H, the angle difference between V1 corresponding to the direction of the operating tool and V2 corresponding to the direction of viewing the operating tool from the imaging position is different. If it is sufficiently large, it is considered that the distance image sensor 102 can detect the end of the operating body. Therefore, the touch determination using the threshold value related to the height of the fingertip in the initial state without adjustment is executed. Particularly in the present embodiment, the second vector V2 generates a vector having the imaging position as a start point and the designated position as an end point.

  As described above, according to the present embodiment, even when the end of the operating body used for the touch operation enters the blind spot of the distance image sensor 102, the recognition of the touch state can be continued. In addition, by notifying the operator that the end of the operating body has entered the blind spot of the distance image sensor 102 without interrupting the recognition of the touch state, the operator can take a posture that enables more stable touch determination. Can be encouraged to take. At this time, the user can continue the input operation without changing the input again if the posture is changed. As described above, according to the present embodiment, operability is improved in recognition of a touch operation in a system that does not use a touch sensor.

[Modification 1]
So far, in the first embodiment, the touch operation that can be recognized by the information processing apparatus 100 has been described as a so-called single touch operation that is recognized based on the movement of one designated position. However, the present embodiment can also be applied to a system in which a plurality of designated positions are input by the end portions of a plurality of operating bodies and a so-called multi-touch operation is input. Hereinafter, as a first modification, an information processing apparatus 100 for recognizing a multi-touch operation in the same table top system as that in FIG.

  The hardware configuration and functional configuration of the information processing apparatus 100 according to Modification 1 are the same as those in the first embodiment described with reference to FIGS. However, when the plurality of designated positions are determined to be in the touch state, the recognizing unit 316 of Modification 1 recognizes the command for the multi-touch operation based on the distance between the plurality of designated positions and the relative movement trajectory. Then, a touch event corresponding to the recognition result is generated. For example, when the distance between a plurality of designated positions in the touch state is enlarged / reduced, it is recognized as a multi-touch operation for enlarging / reducing the projected item. Further, for example, when a plurality of designated positions in the touch state are relatively rotated, the projected item is recognized as a multi-touch for rotating according to the rotation amount. The above is an example, and the multi-touch operation recognized by the information processing apparatus 100 is not limited to this.

  Also in the case of the modification 1, the information processing apparatus 100 performs a touch operation recognition process according to the flowchart of FIG. If a plurality of hand regions are detected by the hand region extraction unit 311 in step S402, the designated position and the intrusion position are specified for each hand region in step S403, and the reference determination unit 315 for each hand region in step S405. Define the criteria for touch determination. And in the case of the modification 1, the recognition part 316 counts the number of the instruction | indication positions determined to be Yes by step S406, and determines whether the some instruction | indication position changed to the touch state simultaneously. If a plurality of designated positions have not transitioned to the touch state at the same time, the process of step S408 in the flowchart of FIG. 4 is executed. When a plurality of designated positions change to the touch state at the same time, the multi-touch operation recognition process is executed assuming that the multi-touch state is set. Note that the number of designated positions in the touched state is counted, and when there are a plurality of designated positions in the touched state, the multi-touch operation recognition process may be executed assuming that the touched state is present. In the multi-touch operation recognition process, the above-described enlargement / reduction operation and rotation operation are recognized. Since the recognition process here is a known technique and is not the subject of the present invention, the description thereof will be omitted.

<Operation example 2>
As an operation example 2, an example of a situation in which the table top system of FIG. 1A is simultaneously operated by a plurality of operator's hands (a plurality of operation bodies) will be given, and specific processing by the information processing apparatus 100 will be described. And its operation. However, portions common to the operation example 1 described with reference to FIGS. 6 and 7 are assigned the same reference numerals, and description thereof will be omitted as appropriate. 8A and 8B show a state where two operators facing each other perform touch input with their right hands (hand 800 and hand 801). At this time, all the fingertips are in contact with the operation surface 100 and two multi-touch operations are input by two people, but the fingertips of the hand 800 have entered the blind spot 700. The multi-touch operation may be input with both hands of one operator.

  In the case of the operation example 2, the information processing apparatus 100 extracts two hand regions from the acquired distance image, specifies the intrusion position and the designated position for each hand region in step S403 and step S404, and obtains position information. For the hand 800, the intrusion position p1 (x1, y1) and the designated position p2 (x2, y2, z2) are specified. For the hand 801, the entry position p4 (x4, y4) and the designated position p5 (x5, y5, z5) are specified. In the operation example 2 as well, the imaging position p3 (x3, y3) is fixed.

  In the process of step S405, first, a process of determining a reference for touch determination regarding the hand region corresponding to the hand 800 is executed. The height z2 of the designated position p2 is higher than the height 601 of the threshold value T1 (Yes in step S500). In the case of FIG. 8A, the angle difference between the first vector V1 representing the direction from the entry position p1 to the designated position p2 and the second vector V2 representing the direction from the imaging position p3 to the designated position p2 is 19 degrees. Yes, smaller than threshold D (Yes in step S504). Accordingly, the user is notified that the fingertip is in the blind spot (step S505), and the reference for touch determination is changed to the second touch threshold and the second release threshold (step S506). On the other hand, regarding the hand region corresponding to the hand 801, the height z5 of the designated position p5 is smaller than the height 601 of the threshold T1 (No in step S500). Accordingly, the first touch threshold and the first release threshold, which are default values, are set as the reference for touch determination (step S507).

  As described above, the process of step S405 is executed for each hand region, and as a result of setting the reference for touch determination, it is determined that both the indicated position p2 and the indicated position p5 are in the touch state (Yes in step S406). ). In Modification 1, since there are a plurality of touched instruction positions, multi-touch operation recognition processing is executed.

  As described above, according to the first modification, even when at least one of the plurality of operation objects used for the multi-touch operation is not detected because the end portion enters the blind spot of the distance image sensor 102, the multi-touch operation is performed. Operation recognition can be continued. Furthermore, by notifying the operator that the end of the specific operating body has entered the blind spot of the distance image sensor 102, the operator can be encouraged to take a posture that allows more stable touch determination. .

[Modification 2]
Furthermore, a modified example in which a determination process is added in order to increase the accuracy of the process of determining whether or not the end of the operating body has entered the blind spot of the distance image sensor 102 will be described. First, in the second modification, assuming that the operating body is a hand, the center position of the back of the hand is specified, and further, based on the distance from the detected position of the center position of the back of the hand, It is determined whether or not the blind spot of the image sensor 102 has been entered. In the second modification, when the hand region is extracted, a portion having the largest width is specified with the direction orthogonal to the principal axis of inertia as the width, and the midpoint of the width is set as the center position of the back of the hand. Then, when the detected indication position is included in a range within a circle having a predetermined radius R centering on the center position of the back of the hand, it is considered that the fingertip is in the blind spot of the distance image sensor 102. This is because when the fingertip is in the blind spot, as shown in FIG. 2B, the indicated position to be detected corresponds to the vicinity of the base of the finger. This is a determination method using the fact that it is often detected in a range close to the center of the back of the hand.

  FIG. 9A schematically shows a state in which the hand placed in the space on the operation surface is viewed along the xy plane along the negative direction of the z-axis. In FIG. 9A, the hand 900 has no blind spot, and the entire hand 900 can be imaged by the distance image sensor 102. For the hand 900, the three-dimensional position information of the designated position 902 is acquired based on the process of specifying the point farthest from the entry position 901 in the hand region extracted from the distance image. Then, the position 903 is specified as the center position. At this time, the designated position 902 is not included in a circle having a predetermined radius R with the center position of the back of the hand as the center. On the other hand, the hand 904 represents a state in which a blind spot 700 has occurred and the designated position 905 has been detected not near the fingertip but near the base of the index finger. In this case, the designated position 905 is included in a garden having a radius R centered on the center position 906 of the back of the hand.

  As described above, in Modification 2, it is determined that the fingertip is not in the blind spot of the distance image sensor 102 in the hand 900 and the fingertip is in the blind spot of the distance image sensor 102 in the hand 904. In the second modification, the process by the determination unit 314 that determines whether the detected indication position is included in a range within a circle having a predetermined radius R centered on the center position of the back of the hand is a step in the flowchart of FIG. Added immediately after S504. When it is determined that the detected indication position is included in a range within a circle with a predetermined radius R centering on the center position of the back of the hand, the process proceeds to step S505. If it is determined that the detected indicated position is not included in the range within the circle having the predetermined radius R centered on the center position of the back of the hand, the process proceeds to step S507. Thereby, even if the height of the indicated position is higher than the threshold value H (Yes in step S500) and the angle difference between the two vectors is smaller than the threshold value D (Yes in step S504), the touch determination criterion is the default. The value may occur.

  For example, as shown in FIG. 9B, when the operator's finger is facing upward, the fingertip is in a state that can be detected from the distance image sensor 102. However, in the first embodiment, if the pointing position is higher than the height 601 and the direction of the arm and the direction from the imaging position toward the fingertip are close to parallel, the touch threshold is set to be larger than the normal height 601. There is a possibility of judgment. However, by adding the processing of Modification 2 as described above to the first embodiment, as shown in FIG. 9B, the finger of the operator is above the threshold value H because the operator's finger is facing upward. It is possible to reduce erroneous recognition of the touch state when it becomes higher. Modification 2 may be used in combination with Modification 1.

[Modification 3]
In Modification 3, assuming that the operating body used for the touch operation is a hand in a “pointing pose” that extends only one finger, the shape of the hand region is similar to the shape model prepared in advance. Whether or not the fingertip has entered the blind spot of the distance image sensor 102 is determined based on the degree. In Modification 3, when the hand region is extracted, the midpoint of the portion with the largest width with the direction orthogonal to the principal axis of inertia as the width is the center of the back of the hand, and the portion with the smallest width is the position of the wrist. When the fingertip is in the blind spot of the distance image sensor 102, the shape of the portion of the hand area beyond the wrist is significantly different from the hand in the “pointing pose”. Specific shape comparison is performed by, for example, extracting a plurality of points on the outer periphery with the shape of the portion ahead of the wrist position in the hand region and a predetermined shape model stored in advance, and for each extracted point The distance is calculated, and the degree of coincidence of the shape is determined by the total value obtained by integrating the distances.

  FIG. 10 shows a process of comparing the shape of the portion ahead of the wrist 1002 with respect to the hand region 109 in which the fingertip is not detected, compared with the shape model 1000 in which the index finger is extended with the center 1001 of the back of the hand as a reference position. It is schematically represented. In particular, since the shape of the index finger portion is large, it can be determined that the fingertip is not detected.

  In the third modification, as described above, the process performed by the determination unit 314 that compares the shape of the hand region and the model is added immediately after step S504 in the flowchart of FIG. As a result of the comparison, when the shape of the hand region is not regarded as the “pointing pose”, the process proceeds to step S505. If it is determined that the shape of the hand area is the “pointing pose”, the process proceeds to step S507. In addition, after the process of determining the magnitude of the distance from the center position of the back of the hand to the designated position described in Modification 2, a determination process for comparing the shape of the hand region and the model of Modification 3 is further added. The determination accuracy can be improved. According to the third modification, as in the second modification, the touch state in the case where the fingertip is higher than the threshold H because the operator's finger is facing upward as shown in FIG. Misrecognition can be reduced. Note that Modification 3 may be used in combination with Modification 1 or 2.

[Modification 4]
In the above-described first embodiment and its modification example, it is considered that the indication position detected when the fingertip cannot be detected from the distance image sensor 102 does not correspond to the fingertip position. The height to be changed was changed. Thereby, even in a situation where the exact position of the fingertip is not detected, it is possible to determine whether or not the touch state is continued based on the actual change in the height of the operator's hand.

  On the other hand, in the touch operation, the locus of the position information of the designated position while in the touch state often has an important meaning in interpreting the input command. In the above-described first embodiment and its modification, when the end of the operating body that is inputting the touch operation enters the blind spot, the touch state is continued using the detected height information of the indicated position. Even if it is accurately determined whether or not there is, the xy coordinates of the indicated position are different from the intention of the operator. Therefore, in that case, the xy coordinates of the designated position are not used for the interpretation of the operation. Therefore, in the fourth modification, when the reference for touch determination is changed, the position information that is the basis for interpreting the touch operation is not the indicated position detected at that time but the information on the estimated fingertip position. Specifically, on the basis of the positional relationship between the center of the back of the hand and the indicated position in the state where the fingertip has been detected, the operating body being touched is a hand that has taken a “pointing pose” The fingertip position in a state where the fingertip is not detected is estimated, and the three-dimensional coordinate information is specified. Then, based on the estimated fingertip information, touch determination and touch operation determination are performed.

  FIG. 11 is a diagram schematically illustrating an overview of an instruction position estimation process in the information processing apparatus 100 according to the fourth modification. Also in the modified example 4, when the hand region is extracted, the center point of the portion having the largest width and the width orthogonal to the principal axis of inertia is set as the center of the back of the hand. Regarding the hand 1100, the fingertip is present at a position that can be detected by the distance image sensor 102, and the designated position P1 (X1, Y1, Z1) is properly detected. At this time, a vector 1101 from P2 to P1 is generated and held as information indicating the positional relationship between the center position P2 (X2, Y2, Z2) of the back of the hand and the designated position P1. Thereafter, when the fingertip enters the blind spot 701 as in the case of the hand 1102, the center position P3 (X3, Y3, Z3) of the hand is specified. Further, based on the vector 1101, a position P4 (X3 + (X1-X2), Y3 + (Y1-Y2), Z3 + (Z1-Z2)) estimated to be the position of the fingertip is specified. In the modified example 4, when the fingertip is in the blind spot 701, the position information specified as the indicated position is corrected with the position information of the fingertip position P4, thereby enabling touch determination linked to the height of the hand, In addition, information close to the locus intended by the operator can be obtained.

  In the modified example 4, when the reference determination unit 315 changes the touch determination reference in step S506, the touch determination is performed on the estimated fingertip position height among the held touch threshold and release threshold references. A criterion for performing is selected. Here, the third touch threshold and the third release threshold are used. The third touch threshold value and the third release threshold value may coincide with the first touch threshold value and the first release threshold value, but in Modification 4, in order to consider an error that occurs in the fingertip position estimation process, It is assumed that the first touch threshold value and the first release threshold value are set slightly larger. Further, in the fourth modification, immediately after step S506, processing for correcting the position information of the indicated position that is held by the information of the estimated fingertip position is added. Accordingly, in step S409 and step S412, the position information held as the movement reference position is the estimated fingertip position. In Modification 4, immediately after the touch determination criterion is set to the default value in step S507, the information held as the value of the designated position is reset to the information specified in step S403.

  In Modification 4, the touch determination in step S406 is performed for interpreting the touch operation by using the height of the indicated position specified based on the distance from the intrusion position as in the first embodiment. In the processes after step S409, the estimated fingertip position may be used. The estimated fingertip position is based on the positional relationship with the center position of the back of the hand at the time when the fingertip was last detected, so if the finger angle or how the joint bends subsequently changes I can not cope. Therefore, the fundamental determination of whether or not the touch operation is being performed uses information on the height of the hand acquired from the distance image obtained in real time, and only when it is determined that the touch operation is being performed. It is possible to reduce misrecognition by using information on the estimated fingertip position. Note that Modification 4 may be used in combination with Modifications 1 to 3.

[Modification 5]
In the fifth modification, the definition of a vector to be compared in order to determine whether or not the fingertip has entered the blind spot of the distance image sensor 102 when the operating body is a hand is changed. Specifically, the position of the wrist is specified from the extracted hand region, and in step S502, a vector having the wrist position as the start point and the hand center position as the end point is generated as the first vector representing the direction of the operating tool. To do. As in the third modification, the wrist position is specified as a portion having the smallest width with the width orthogonal to the principal axis of inertia when the hand region is extracted. Further, in the modified example 5, in step S503, a vector in which the imaging position is the start point and the center position of the hand or the wrist position is the end point is generated as the second vector representing the direction in which the operating body is viewed from the imaging position.

  In the fifth modification, a vector representing the direction of the operating tool is generated using the position of the wrist instead of the entry position, so that even when the user operates by bending the wrist, the vector is valid as the direction of the operating tool. Can be obtained. In addition, by using the wrist position or the center position of the hand as the direction of viewing the operating body from the imaging position instead of the pointing position, an accurate direction vector can be generated while suppressing the influence of blurring of the pointing position. be able to. Therefore, according to the modified example 5, it is possible to accurately determine whether the fingertip is hidden even when the wrist of the operator's hand as the operation body is bent.

  FIG. 12A schematically shows a state in which the hand placed in the operation space is viewed along the xy plane along the negative direction of the z axis. In FIG. 12A, the hand 1200 has no blind spot, and the entire hand 1200 can be imaged by the distance image sensor 102. The hand 1200 is bent at the wrist. The situation where the wrist is bent in this way may occur, for example, for reasons such as preventing the shadow of the hand 1200 from being superimposed on the projected image. For the hand 1200, the designated position 1202 is identified based on the process of identifying the point farthest from the entry position 1201 in the hand region extracted from the distance image. At this time, the angle difference between the first vector V1 representing the direction from the entry position 1201 to the designated position 1202 and the second vector V2 representing the direction from the imaging position 1203 to the designated position 1202 is 18 degrees, and the threshold D = Below 30 degrees. Therefore, according to the first embodiment, when the designated position is high, it is determined that the fingertip has entered the blind spot even though the fingertip can actually be detected from the imaging position. there is a possibility.

  FIG. 12B illustrates position information that is handled when processing according to the modification 5 is performed on the hand 1200 in the same state as that in FIG. In the modified example, a vector V3 in which the wrist center position 1204 is the start point and the hand center position 1205 is the end point is used as the first vector, and a vector V4 in which the imaging position is the start point and the wrist position is the end point is used as the second vector. Generated. FIG. 12B shows the same situation as FIG. 12A, but the angle difference between V3 and V4 is 45 degrees, which is larger than the threshold D = 30 degrees. Therefore, touch determination using a default reference is performed for the fingertip. As described above, according to the fourth modification, it is possible to obtain a highly accurate determination result while corresponding to the flexible shape of the human hand used as the operating tool. Modification 5 may be used in combination with Modifications 1 to 4.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

310 image acquisition unit 311 hand region extraction unit 312 position detection unit 313 state acquisition unit 314 determination unit 315 reference determination unit 316 recognition unit 317 display control unit

Claims (21)

An information processing apparatus for recognizing an input of an operation on an operation surface by an end of an operation body,
When the end of the operating body is present within a range in which the distance from the operating surface is smaller than a predetermined reference in the space on the operating surface, the operating body is obtained from an image of the space on the operating surface. Obtaining means for obtaining information representing the state of the imaging means that has captured the image;
When the end of the operating body is considered to be in the blind spot of the imaging means in relation to the information acquired by the acquiring means, the end of the operating body is in the blind spot of the imaging means Recognizing means for recognizing that an end of the operating body is being input for an operation on the operation surface, using a different standard from that not considered
An information processing apparatus comprising:   2. The information processing according to claim 1, wherein the acquisition unit is an acquisition unit that acquires information indicating a direction of the operation body in the space and information on a positional relationship between the operation body and the imaging unit. apparatus.   3. The information processing apparatus according to claim 2, wherein the information related to the positional relationship between the operation tool and the imaging unit is information representing a direction in which an end portion of the operation tool is viewed from an imaging position of the imaging unit. . Furthermore, an angular difference between the two directions specified based on information indicating the direction of the operating body in the space and information indicating the direction of viewing the end of the operating body from the imaging position of the imaging means. Relatedly, a determination unit that determines whether an end portion of the operation body is considered to be within a blind spot of the imaging unit,
The said recognition means determines the reference | standard which recognizes that the edge part of the said operation body is inputting the operation with respect to the said operation surface according to the determination result by the said determination means. The information processing apparatus described. Each pixel constituting the image reflects position information in the space,
Furthermore, based on the position information reflected in the image, provided with position detection means for detecting the position information of the end of the operating body as an indicated position pointed to by the operating body,
When the distance from the operation surface of the indicated position detected by the position detection unit is smaller than a predetermined reference, the determination unit uses the angle difference between the two directions, and the end of the operation body is The information processing apparatus according to claim 4, wherein a criterion for recognizing that it is determined whether the image is regarded as being within a blind spot of the imaging unit is determined. The recognizing unit recognizes that the end of the operating body is inputting a touch operation to the operation surface using a reference regarding the distance from the operation surface of the indicated position detected by the position detecting unit. A means of recognition,
When the end of the operating body is considered to be in the blind spot of the imaging means in relation to the information acquired by the acquiring means, the end of the operating body is in the blind spot of the imaging means The information processing apparatus according to claim 5, wherein a distance larger than a case where the distance is not regarded as a reference is determined as the reference.   The recognizing unit touches the operation surface with an end of the operating body when the distance from the operation surface of the indicated position detected by the position detection device is smaller than the reference distance. The information processing apparatus according to claim 6, wherein the information processing apparatus is recognized.   If the distance from the operation surface of the indicated position detected by the position detection unit is greater than a predetermined distance threshold, the determination unit may determine that the two directions are closer to parallel than a reference angle. The information processing apparatus according to claim 5, wherein it is determined that an end portion of the operating body is considered to be in a blind spot of the imaging unit.   When the distance from the operation surface of the indicated position detected by the position detection unit is larger than a predetermined distance threshold, the determination unit determines that the angle difference between the two directions is the difference between the angles between the two directions. Is smaller than a predetermined angle threshold, it is determined that the end of the operating body is within the blind spot of the imaging means, and the difference between the two directions is the difference between the two directions. 7. The information processing apparatus according to claim 5, wherein if it is larger than a predetermined angle threshold, it is determined that an end portion of the operation body is not considered to be within a blind spot of the imaging unit. The information indicating the direction of the operating body in the space is the indicated position detected by the position detecting means from the position where the end of the space specified by using the image intersects the operating body. Information indicating the direction of the straight line specified based on
The information indicating the direction of viewing the end of the operating body from the imaging position of the imaging means is a straight line specified based on the imaging position of the imaging means and the indicated position detected by the position detection means. The information processing apparatus according to claim 5, wherein the information processing apparatus is information representing a direction. The recognition means is
In relation to the information acquired by the acquisition means, when the end of the operating body is considered to be in a blind spot caused by a portion other than the end of the operating body, the end of the operating body 7. The method according to claim 6, further comprising: recognizing that an end of the operation body is inputting a touch operation on the operation surface using a different standard from a case where the operation object is not considered to be in the blind spot. The information processing apparatus according to any one of 10.   12. When an operator's hand is used as the operating body, the end of the operating body is a portion corresponding to a fingertip of the operator's hand. The information processing apparatus according to item. The information processing apparatus constitutes a table top interface system,
The imaging means is a distance image that is captured by a distance image sensor that constitutes the table top interface system, and position information corresponding to a height direction from the operation surface is reflected on each pixel,
The recognizing unit is configured to determine an end of the operating body based on a height of a part of the operating body specified based on the distance image and a reference regarding a height of the part of the operating body from the operating surface. The information processing apparatus according to claim 1, wherein the information processing apparatus recognizes that a part touches the operation surface.   The recognition means, when there are a plurality of operating bodies recognized as being input to the operation surface, based on the trajectories of a plurality of designated positions detected for the plurality of operating bodies by the position detection means. The information processing apparatus according to claim 6, wherein a multi-touch operation is recognized. When an operator's hand is used as the operating body, the end of the operating body corresponds to a portion corresponding to a fingertip of the operator's hand,
Furthermore, it comprises a position detection means for detecting position information that is regarded as a fingertip of the operator's hand from the image,
The recognizing means is based on the information acquired by the acquiring means and the magnitude of the distance from the image of the position detected by the position detecting means from the portion considered as the back of the hand of the operator's hand, The information processing apparatus according to claim 1, wherein it is determined whether a fingertip of the operator's hand is considered to be in a blind spot of the imaging unit. When an operator's hand is used as the operating body, the end of the operating body corresponds to a portion corresponding to a fingertip of the operator's hand,
The recognizing means is based on a result of comparing the information acquired by the acquiring means with the shape of a region in which the operating body extracted from the image is captured and the shape of a predetermined hand model. The information processing apparatus according to claim 1, wherein it is determined whether a fingertip of a hand is considered to be in a blind spot of the imaging unit. When an operator's hand is used as the operating body, the end of the operating body corresponds to a portion corresponding to a fingertip of the operator's hand,
When the fingertip of the operator's hand is considered to be in the blind spot of the imaging means, the recognition means is considered not to be in the blind spot of the imaging means when the fingertip is considered to be in the blind spot of the imaging means. The touch operation is performed using the position information of the fingertip estimated based on the position information regarded as the fingertip of the operator's hand in the image captured at the time and the position information of the center of the back of the hand of the operating body. The information processing apparatus according to claim 1, wherein the information processing apparatus is recognized. When an operator's hand is used as the operating body, the end of the operating body corresponds to a portion corresponding to a fingertip of the operator's hand,
The information indicating the direction of the operating body in the space is a straight line specified based on the position of the center of the back of the operator's hand from the position of the wrist of the operator's hand specified using the image. Information indicating the direction of
The information indicating the direction of viewing the end of the operating body from the imaging position of the imaging means is the center of the back of the operator's hand from the imaging position of the imaging means and the wrist position of the operator's hand. The information processing apparatus according to claim 3, wherein the information processing apparatus is information indicating a direction of a straight line specified based on the position. A control method for an information processing device that recognizes an input of an operation on an operation surface by an end of an operation body,
In the space on the operation surface, when the end of the operation body exists within a range in which the distance from the operation surface is smaller than a predetermined reference, an acquisition unit obtains an image of the space on the operation surface. An acquisition step of acquiring information representing a state of the operation body with respect to an imaging unit that has captured the image;
When the recognition means regards the end of the operating body as being within the blind spot of the imaging means in relation to the information acquired in the acquisition step, the end of the operating body is the blind spot of the imaging means. A recognition step for recognizing that an end of the operating body is being input for an operation on the operation surface, using a different standard than when not considered to be included,
An information processing apparatus control method comprising:   A program that causes a computer to function as the information processing apparatus according to claim 1 by being read and executed by the computer.   A computer-readable storage medium storing the program according to claim 20.

Images