JP2017102598A - Recognition device, recognition method, and recognition program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a recognition device capable of recognizing plural types of gestures, and a recognition method, and a recognition program.SOLUTION: A recognition device 100 includes: an acquisition section 110 that acquires, from a distance image sensor 50, a distance image from the distance image sensor up to an object existing in a space in a visual line direction of a user; an identification part that identifies a finger area by identifying a fingertip and finger base based on the distance between a feature point and a reference point of the object included in the distance image, and identifies a hand area based on the identified finger area; and a gesture recognition part 130 that recognizes the type of the gesture based on a differential value between the fingertip position and the finger base position which are included in the hand area.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a recognition device and the like.

  In recent years, HMD (Head Mounted Display) has become widespread. The HMD is a display device that is worn on the user's head. By showing a CG (Computer Graphics) video or a remote video to the user, the user can experience virtual reality.

  Here, since a user wearing a full-screen HMD covers his eyes with a display, it is difficult for the user to operate an input device such as a keyboard while wearing the HMD, and the viewpoint in the video is changed. It was difficult to do. For this reason, there is a demand for a technique that allows a user wearing an HMD to change a viewpoint in a video by performing a predetermined gesture or the like.

  Prior art 1 will be described. Prior art 1 recognizes a gesture based on movement of a user's arm using a fixed sensor such as Kinect (registered trademark), and enables navigation in six directions. FIG. 13 is a diagram for explaining the related art 1. For example, as shown in 1A, when the user fixes the position of the hand, the apparatus is instructed to stop moving. As shown in 1B, when the user moves the arm position to the left, the device is instructed to move leftward, and when the user moves the arm position to the right, the device is instructed to move rightward. . As shown in 1C, when the user moves the arm position forward, the device instructs the device to move forward, and when the user moves the arm position backward, the device instructs the device to move backward. As shown in 1D, when the user moves the arm position upward, the user instructs the apparatus to move upward, and when the user moves the arm position downward, the user instructs the apparatus to move downward.

  Since the conventional technique 1 is not applied to the HMD, a case where the conventional technique 1 is applied to the HMD and the viewpoint position in the video is changed will be considered. Since the camera mounted on the HMD is attached so as to capture the direction of the line of sight from the user's head, the imaging range includes the palm, but the arm is not included, so the movement of the arm cannot be detected, and the viewpoint It is difficult to indicate the direction of movement. For example, it is difficult to recognize the 1B and 1D gestures described in FIG.

  In addition, when moving an arm forward and backward like 1C of the prior art 1, an arm can be included in the imaging | photography range of a camera. However, since the physical characteristics are different for each user, the length of the hand is different for each user. For example, it is difficult to estimate the bending and stretching of the arm depending on the distance between the palm and the camera.

  Prior art 2 will be described. Prior art 2 is a technique for determining whether or not both palms of the user are present in the shooting range of the HMD camera, and moving the viewpoint position forward if both palms are present. Compared with the prior art 1, the prior art 2 only needs to determine whether or not both palms exist, so that the user can move the viewpoint position forward while wearing the HMD.

JP 2012-79177 A Japanese Patent Laid-Open No. 10-177449 JP 2014-72575 A

  However, the above-described conventional technique has a problem that it cannot recognize a plurality of types of gestures.

  For example, in the conventional technique 2, since it is only determined whether or not the palms of both hands exist, the movement direction of the operable viewpoint position is limited.

  In one aspect, an object of the present invention is to provide a recognition device, a recognition method, and a recognition program capable of recognizing a plurality of types of gestures.

  In the first proposal, the recognition device includes a distance sensor, an acquisition unit, a specification unit, and a gesture recognition unit. The distance sensor acquires distance information from the user side to an object existing in space in the direction of the user's line of sight. The acquisition unit acquires distance information from the distance sensor to an object existing in the space in the user's line-of-sight direction as a distance image. The identifying unit identifies the finger region by identifying the fingertip and the root of the finger based on the distance between the feature point of the object represented as the distance image and the reference point, and the hand region based on the identified finger region Is identified. The gesture recognition unit recognizes the type of gesture based on the difference value between the position of the fingertip included in the hand region and the base position of the finger.

  Multiple types of gestures can be recognized.

FIG. 1 is a functional block diagram illustrating the configuration of the recognition apparatus according to the present embodiment. FIG. 2 is a diagram for explaining a process of cutting out hand region candidates. FIG. 3 is a diagram for explaining the smoothing process. FIG. 4 is a diagram (1) for explaining a process of identifying a fingertip candidate point and a root candidate point. FIG. 5 is a diagram (2) for explaining the process of identifying the fingertip candidate point and the root candidate point. FIG. 6 is a diagram for explaining another process for detecting a root candidate point. FIG. 7 is a diagram for explaining processing for specifying a finger region. FIG. 8 is a diagram for explaining processing for specifying a hand region. FIG. 9 is a flowchart illustrating the processing procedure of the recognition apparatus according to the present embodiment. FIG. 10 is a flowchart showing a processing procedure for calculating a fingertip candidate point and a root candidate point. FIG. 11 is a flowchart illustrating a processing procedure for calculating a finger region from a fingertip candidate point and a finger root candidate point. FIG. 12 is a diagram illustrating an example of a computer that executes a recognition program. FIG. 13 is a diagram for explaining the related art 1.

  Hereinafter, embodiments of a recognition device, a recognition method, and a recognition program disclosed in the present application will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments.

  FIG. 1 is a functional block diagram illustrating the configuration of the recognition apparatus according to the present embodiment. The recognition apparatus 100 is incorporated in, for example, an unillustrated HMD (Head Mounted Display). The recognition device 100 is connected to a distance image sensor 50 and a display device 60 mounted on the HMD. The user wears the HMD in which the recognition device 100 is incorporated on the head.

  The distance image sensor 50 is a sensor that measures distance information from the distance image sensor 50 to an object included in the shooting range of the distance image sensor 50. The shooting direction of the distance image sensor is the user's line-of-sight direction.

  The distance image sensor 50 measures distance information from the distance image sensor 50 to an object based on the TOF (Time Of Flight) method. The distance information includes the distance from the distance image sensor 50 to each point on the object. The distance image sensor 50 outputs distance information to the recognition device 100. The distance image sensor 50 may output to the recognition device 100 information on a distance image obtained by combining an image of a shooting range captured by a camera and distance information. Further, the distance image may be a distance image in which the distance between the target object and the distance sensor is represented by a pixel value. The processing described below may also be performed using a distance image instead of distance information.

  The display device 60 is a display device that displays video information output from the recognition device 100.

  The recognition apparatus 100 includes an acquisition unit 110, a specification unit 120, a gesture recognition unit 130, and a display control unit 140.

  The acquisition unit 110 is an acquisition unit that acquires distance information from the distance image sensor 50. The acquisition unit 110 outputs the acquired distance information to the specifying unit 120.

  The identifying unit 120 identifies the finger region by identifying the user's fingertip and the root of the finger based on the distance between the point on the object included in the distance information and the reference point, and identifies the identified finger region. This is a processing unit for specifying a hand region based on the basis. The identification unit 120 outputs the hand region information to the gesture recognition unit 130.

  Below, the process of the specific | specification part 120 is demonstrated concretely. The specifying unit 120 performs processing for cutting out hand region candidates, processing for smoothing the contours of hand region candidates, processing for specifying fingertip candidate points and root candidate points, processing for specifying finger regions, and hand region candidates as hand regions The process which specifies whether or not is executed.

  A process in which the specifying unit 120 extracts a hand region candidate will be described. Based on the distance information, the specifying unit 120 cuts out an object existing in an area where the distance from the distance image sensor 50 is less than the threshold as a hand area candidate.

  FIG. 2 is a diagram for explaining a process of cutting out hand region candidates. In FIG. 2, for example, the distance information 10a includes objects 11, 12, 13, and 14. Of the objects 11 to 14, it is assumed that the object 14 exists in a region where the distance from the distance image sensor 50 is less than the threshold value. In this case, the specifying unit 120 generates the distance information 10b by cutting out the object 14 from the distance information 10a.

  A process in which the specifying unit 120 smoothes the contour of the hand region candidate will be described. The specifying unit 120 performs smoothing on the distance information obtained by cutting out the hand region candidates by executing a weighted average with adjacent contours.

  FIG. 3 is a diagram for explaining the smoothing process. The identification unit 120 smoothes the distance information by applying the smoothing filter 20 to the entire distance information. The smoothing filter is a filter that calculates the distance value of the center pixel 21i based on the equation (1).

  The distance value of the central pixel 21i = 1/9 (the distance value of the pixel 21a + the distance value of the pixel 21b + the distance value of the pixel 21c + the distance value of the pixel 21d + the distance value of the pixel 21e + the value of the pixel 21f) Distance value + distance value of pixel 21g + distance value of pixel 21h) (1)

  The process in which the specifying unit 120 specifies the fingertip candidate points and the finger root candidate points will be described. 4 and 5 are diagrams for explaining processing for specifying a fingertip candidate point and a root candidate point. The specifying unit 120 sets a reference point 15 for the hand region candidate 14 included in the smoothed distance information. As illustrated in FIG. 4, for example, the specifying unit 120 sets the barycentric position of the hand region candidate 14 as the reference point 15.

  The specifying unit 120 calculates points satisfying the following first condition and second condition for all points on the contour of the hand region candidate 14 and specifies a fingertip candidate point and a finger root candidate point. Here, as shown in FIG. 5, description will be made using points 16A, 16B, and 16C on adjacent contours. The distance between the reference point 15 and the point 16A is Da. The distance between the reference point 15 and the point 16B is Db. The distance between the reference point 15 and the point 16C is Dc.

  The specifying unit 120 specifies the point 16B as a fingertip candidate point when the following first condition is satisfied.

  First condition: Da <Db, Db> Dc

  When the second condition is satisfied, the specifying unit 120 specifies the point 16B as a root candidate point.

  Second condition: Da> Db, Db <Dc

  The identifying unit 120 identifies the fingertip candidate points 17a, 17b, 17c, 17d, 17e, 17f, and 17g, for example, by repeatedly executing the above process for all points on the contour of the hand region candidate 14 in FIG. . Further, the specifying unit 120 specifies the finger root candidate points 18a, 18b, 18c, 18d, and 18e.

  The specifying unit 120 compares the positions of the fingertip candidate points 17a to 17g and merges the fingertip candidate points whose distance between the fingertip candidate points is less than the threshold. In addition, the specifying unit 120 compares the positions of the respective finger root point candidates 18a to 18e, and merges the finger root candidate points whose distances between the finger root candidate points are less than the threshold.

  For example, in the example shown in FIG. 4, since the distance between the fingertip candidate point 17e and the fingertip candidate point 17f is less than the threshold value, the specifying unit 120 merges the fingertip candidate point 17e and the fingertip candidate point 17f. For example, the specifying unit 120 leaves the fingertip candidate point 17e and merges by deleting the fingertip candidate point 17f. The specifying unit 120 obtains an average distance between the distance between the fingertip candidate point 17e and the distance between the fingertip candidate point 17f, sets the obtained average distance as the distance between the new fingertip candidate points, and sets the fingertip candidate points 17e and 17f as the distances. You may merge by deleting.

  Here, in the above process, the detection of the finger base candidate point in the end finger becomes unstable, and the finger base candidate point may not be detected appropriately. For this reason, the specifying unit 120 may detect a root candidate point that has not been detected in the above process by further executing the following process.

  FIG. 6 is a diagram for explaining another process for detecting a root candidate point. The specifying unit 120 obtains an intersection between a straight line passing through the plurality of fingertip candidate points and the contour of the hand region candidate 14 and detects the obtained intersection as a finger root candidate point. For example, as illustrated in FIG. 6, the specifying unit 120 detects a straight line 19 passing through the finger root candidate points 18 d and 18 e and an intersection 18 f of the contour of the hand region candidate 14 as a finger root candidate point.

  Note that the specifying unit 120 may detect the intersection of the straight line passing through the root of the finger included in the hand region and the contour of the hand region candidate 14 as the root of the finger after specifying the hand region by processing to be described later. . In the following description, the finger root point specified by the intersection of the straight line passing through the finger root point and the contour of the hand region candidate 14 is referred to as a virtual finger root point.

  Processing for specifying the finger area by the specifying unit 120 will be described. FIG. 7 is a diagram for explaining processing for specifying a finger region. The specifying unit 120 specifies a region including a fingertip candidate point that satisfies the following third condition and fourth condition and two finger candidate points as a finger region.

  As an example, as shown in FIG. 7, description will be made using a fingertip candidate point 17 </ b> A and two finger root candidate points 18 </ b> B and 18 </ b> C. Here, a triangle composed of the fingertip candidate point 17A and the two finger root candidate points 18B and 18C is represented as a triangle ABC. The specifying unit 120 determines the shape of the triangle ABC, and determines the region of the triangle ABC as a finger region when the third condition is satisfied and the fourth condition is satisfied.

Third condition: The angle of the point 17A of the triangle ABC is an acute angle.
Fourth condition: The difference between the length of the side AB connecting the point 17A and the point 18B and the length of the side AC connecting the point 17A and the point 18C is less than the threshold value.

  When the triangle ABC shown in FIG. 7 satisfies the third condition and the fourth condition, the specifying unit 120 defines the area surrounded by the fingertip candidate point 17A and the two finger root candidate points 18B and 18C as a finger area. As specified. The specifying unit 120 specifies another finger region by repeatedly executing the above process for the other fingertip candidate points and the other two finger root candidate points. In the following description, the fingertip candidate points included in the finger area are referred to as fingertip points, and the finger root candidate points included in the finger area are referred to as finger root points.

  Further, the specifying unit 120 specifies a triangle including a fingertip candidate point, one finger root candidate point, and one virtual finger root point, and the specified triangle satisfies the third condition and the fourth condition. In this case, the corresponding triangle is specified as the finger area.

  A process in which the specifying unit 120 determines whether the hand region candidate is a hand region will be described. The specifying unit 120 determines whether the hand region candidate is a hand region based on the number of finger regions detected from the hand region candidates. For example, when the number of finger regions included in the hand region candidate is 5, the specifying unit 120 determines that the hand region candidate is a hand region.

  FIG. 8 is a diagram for explaining processing for specifying a hand region. For example, in the example illustrated in FIG. 8, it is assumed that the finger regions 20 a, 20 b, 20 c, 20 d, and 20 e are specified from the hand region candidate 14. Since the number of finger areas 20a to 20e is “5”, the specifying unit 120 determines that the hand area candidate 14 is a hand area.

  The identification unit 120 outputs the hand region information to the gesture recognition unit 130. The information on the hand area includes the distance between the fingertip points and the distance between the finger base points of each finger area included in the hand area. Here, the distance of the fingertip point indicates the distance from the distance image sensor 50 to the fingertip point. The distance of the finger root point indicates the distance from the distance image sensor 50 to the finger root point. The specifying unit 120 may convert the distance between the fingertip points and the distance between the finger base points into three-dimensional coordinates.

  The gesture recognition unit 130 is a processing unit that recognizes the type of gesture based on the difference value between the distance between the fingertip points included in the hand region and the distance between the finger base points. For example, the gesture recognizing unit 130 recognizes that the hand region is tilted forward when the value obtained by subtracting the finger root distance from the fingertip point distance is a “positive value”. On the other hand, when the value obtained by subtracting the distance of the finger base point from the distance of the fingertip point is a “negative value”, the gesture recognition unit 130 recognizes that the hand region is inclined backward.

  Note that the gesture recognition unit 130 may calculate a difference value between the fingertip point and the finger base point included in the finger area from each finger area, and the average value of the plurality of calculated difference values may be the above difference value. . Alternatively, the gesture recognition unit 130 may select any one finger region from among the plurality of finger regions, and calculate the difference value based on the fingertip point and the finger base point included in the selected finger region. .

  The gesture recognition unit 130 outputs information on the gesture recognition result to the display control unit 140. For example, the gesture recognition result information includes information on whether the hand region is tilted forward or the hand region is tilted backward, the distance between the fingertip points included in the hand area, and the distance between the finger root points. Information on the absolute value of the difference value is included.

  The display control unit 140 is a processing unit that displays the video in the virtual space on the display device 60. Further, the display control unit 140 moves the viewpoint position of the user on the virtual space to be displayed on the display device 60 based on the information on the gesture recognition result.

  For example, if the gesture recognition result information includes information indicating that the hand region is tilted forward, the display control unit 140 controls the virtual space image displayed on the display device 60. The user's viewpoint position in the virtual space is moved forward.

  If the gesture recognition result information includes information indicating that the hand region is tilted backward, the display control unit 140 controls the virtual space image to be displayed on the display device 60 to control the virtual space. The user's viewpoint position in space is moved backward.

  The display control unit 140 may adjust the moving speed when the user's viewpoint position is moved based on the absolute value information included in the gesture recognition result information. For example, the display control unit 140 increases the moving speed of the viewpoint position as the absolute value increases, and decreases the moving speed as the absolute value approaches 0.

  Next, an example of a processing procedure of the recognition apparatus according to the present embodiment will be described. FIG. 9 is a flowchart illustrating the processing procedure of the recognition apparatus according to the present embodiment. The process illustrated in FIG. 9 is executed every time the recognition apparatus 100 acquires distance information from the distance image sensor 50. As illustrated in FIG. 9, the specifying unit 120 of the recognition apparatus 100 cuts out hand region candidates (step S101).

  The identifying unit 120 smoothes the contour of the hand region candidate (step S102). The specifying unit 120 calculates a fingertip candidate point and a finger root candidate point included in the hand region candidate (step S103). The specifying unit 120 calculates a finger region from the fingertip candidate points and the finger root candidate points (step S104).

  The identifying unit 120 counts the number of finger areas (step S105). The identifying unit 120 determines whether or not the number of finger areas is 5 (step S106). If the number of finger areas is not 5 (No at Step S106), the specifying unit 120 ends the process.

  When the number of finger areas is five (step S106, Yes), the specifying unit 120 specifies a hand area candidate including the finger area as a hand area (step S107). The gesture recognition unit 130 of the recognition apparatus 100 recognizes a gesture based on the difference value between the distance between the fingertip point and the root point included in the hand region (step S108).

  The display control unit 140 of the recognition apparatus 100 controls the viewpoint position in the virtual space based on the gesture recognition result (step S109).

  Next, the processing procedure for calculating the fingertip candidate points and the finger root candidate points shown in step S103 of FIG. 9 will be described. FIG. 10 is a flowchart showing a processing procedure for calculating a fingertip candidate point and a root candidate point. As illustrated in FIG. 10, the specifying unit 120 of the recognition apparatus 100 determines a reference point (step S201).

  The specifying unit 120 specifies a point satisfying the first condition as a fingertip candidate point for all the points of the contour of the hand region candidate (step S202). The specifying unit 120 specifies a point satisfying the second condition as a root candidate point for all points of the contour of the hand region candidate (step S203).

  The specifying unit 120 merges adjacent fingertip candidate points (step S204). The identifying unit 120 merges adjacent root candidate points (step S205).

  Next, a processing procedure for calculating a finger region from the fingertip candidate point and the root candidate point shown in step S104 of FIG. 9 will be described. FIG. 11 is a flowchart illustrating a processing procedure for calculating a finger region from a fingertip candidate point and a finger root candidate point. As illustrated in FIG. 11, the identifying unit 120 of the recognition apparatus 100 selects a fingertip candidate point having a finger root candidate point on both sides (step S301).

  When the triangle ABC connecting the fingertip candidate point A and the root candidate points B and C on both sides of the fingertip candidate point A satisfies the third condition and the fourth condition, the specifying unit 120 determines the region of the triangle ABC. The finger area is specified (step S302).

  The specifying unit 120 specifies the intersection of the straight line passing through the finger root point belonging to the finger area and the contour of the hand area candidate as the virtual finger root point (step S303). When the triangle ABC connecting the fingertip candidate point A and the finger root candidate points (virtual finger root points) B and C adjacent to the fingertip candidate point A satisfies the third condition and the fourth condition, The area of the triangle ABC is specified as the finger area (step S304).

  Next, the effect of the recognition apparatus 100 according to the present embodiment will be described. The recognition apparatus 100 identifies the hand region based on the fingertip candidate point and the finger root candidate point of the object included in the distance information, and based on the difference between the fingertip point and the finger root point included in the hand region, Recognize the type. For this reason, according to the recognition apparatus 100, multiple types of gestures can be recognized. In addition, by using the gesture recognition result, for example, an intuitive viewpoint movement navigation is possible in a full-screen display HMD.

  The recognition apparatus 100 selects three adjacent points on the object, and selects the center point as a fingertip candidate point when the distance relationship between the selected point and the reference point satisfies the first condition. For this reason, a fingertip candidate can be appropriately identified from each point on the object.

  The recognition apparatus 100 selects three adjacent points on the object, and selects the central point as a root candidate point when the relationship between the distance between the selected point and the reference point satisfies the second condition. Therefore, it is possible to appropriately specify the root candidate from each point on the object.

  The recognizing device 100 selects a fingertip candidate point and an adjacent finger root candidate point, and when the shape of a triangle formed by each selected candidate point satisfies the third condition and the fourth condition, The candidate point area is specified as the finger area. For this reason, the finger region can be accurately identified from the object.

  When the number of specified finger areas is 5, the recognition apparatus 100 determines a hand area candidate including each finger area as a hand area. For this reason, the user can make the recognition apparatus 100 recognize the gesture by tilting the hand with the finger of the hand consciously opened. In addition, even if the user's hand is not open, even if the hand is accidentally placed in front of the line of sight, it is not recognized as a hand region, so the recognition device 100 recognizes the gesture without being conscious. Can be prevented.

  The recognizing device 100 recognizes that the hand region is tilted forward when the value obtained by subtracting the distance between the fingertip points from the distance between the fingertip points is a positive value. Further, when the value obtained by subtracting the distance between the fingertip points from the distance between the fingertip points is a negative value, the recognition apparatus 100 recognizes that the hand region is inclined backward. For this reason, the recognition apparatus 100 can recognize a user's gesture appropriately, without being influenced by a user's physical difference.

  The recognition apparatus 100 specifies an intersection of a straight line passing through a plurality of finger root points and the contour of a hand region candidate as a new finger root candidate point, and specifies a finger region using the specified new finger root candidate point To do. For this reason, even if the detection of the finger root candidate point on the end finger becomes unstable, the finger root candidate point can be appropriately detected.

  By the way, the process of the recognition apparatus 100 mentioned above is an example, and the recognition apparatus 100 may recognize a gesture by performing other processes. For example, the gesture recognition unit 130 of the recognition apparatus 100 determines to recognize that the hand region is tilted backward when the value obtained by subtracting the finger root point distance from the fingertip point distance is a “negative value”. However, it is not limited to this.

  For example, the gesture recognition unit 130 may acquire distance information from the acquisition unit 110 and may recognize that the hand region is tilted backward when the distance from the distance image sensor 50 to the object is less than a threshold value. .

  The distance image sensor 50 may be mounted on a helmet in addition to that mounted directly on the HMD. The distance image sensor 50 does not necessarily have to be mounted on the head. For example, a wearing method in which the user puts the distance image sensor 50 from the neck may be used.

  The HMD incorporating the recognition apparatus 100 according to the present embodiment may be a full screen display HMD, an optical see-through HMD, or the like. The display device 60 is not limited to the HMD, and may be projected onto the user's field of view using a projector or the like.

  Next, an example of a computer that executes a recognition program that realizes the same function as the recognition device 100 described in the above embodiment will be described. FIG. 12 is a diagram illustrating an example of a computer that executes a recognition program.

  As illustrated in FIG. 12, the computer 300 includes a CPU 301 that executes various arithmetic processes, an input device 302 that receives data input from a user, and a display 303. The computer 300 includes a reading device 304 that reads a program and the like from a storage medium, an interface device 305 that exchanges data with another computer via a network, and a distance image sensor 306. The computer 300 also includes a RAM 307 that temporarily stores various types of information and a hard disk device 308. The devices 301 to 308 are connected to the bus 309.

  The hard disk device 308 has an acquisition program 308a, a specific program 308b, and a recognition program 308c. The CPU 301 reads the acquisition program 308 a, the specific program 308 b, and the recognition program 308 c and expands them in the RAM 307.

  The acquisition program 308a functions as an acquisition process 307a. The specific program 308b functions as a specific process 307b. The recognition program 308c functions as a recognition process 307c.

  The process of the acquisition process 307a corresponds to the process of the acquisition unit 110. The processing of the specific process 307b corresponds to the processing of the specifying unit 120. The process of the recognition process 307c corresponds to the process of the gesture recognition unit 130.

  Note that the acquisition program 308a, the identification program 308b, and the recognition program 308c are not necessarily stored in the hard disk device 308 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, and an IC card inserted into the computer 300. Then, the computer 300 may read and execute each of the programs 308a to 308c.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Supplementary Note 1) A distance sensor that acquires distance information from a user side to an object that exists in space in the direction of the user's line of sight;
An acquisition unit that acquires, as a distance image, distance information from the distance sensor to an object that exists in a space in a user's line-of-sight direction;
Based on the distance between the feature point of the object represented as the distance image and the reference point, the finger region is identified by identifying the fingertip and the root of the finger, and the hand region is identified based on the identified finger region. Specific part to do,
A gesture recognition unit, comprising: a gesture recognition unit that recognizes a gesture type based on a difference value between the position of the fingertip included in the hand region and the position of the base of the finger.

(Appendix 2) The specifying unit selects three adjacent feature points on the object, a first distance between the reference point and the first feature point, a second distance between the reference point and the second feature point, When the third distance between the reference point and the third feature point is calculated, and the second distance is larger than the first distance and the second distance is larger than the third distance, the second distance is calculated. The recognition apparatus according to appendix 1, wherein the point is identified as a fingertip point.

(Supplementary Note 3) The specifying unit selects three adjacent feature points on the object, a first distance between the reference point and the first feature point, a second distance between the reference point and the second feature point, When the third distance between the reference point and the third feature point is calculated, and the second distance is smaller than the first distance and the second distance is smaller than the third distance, the second distance is calculated. The recognition apparatus according to appendix 1 or 2, wherein the point is identified as a point of a root of a finger.

(Supplementary Note 4) The specifying unit selects the fingertip, the root of the first finger adjacent to the fingertip, and the root of the second finger, the distance from the fingertip to the root of the first finger, A difference between the distance to the root of the second finger is less than a threshold value and a line segment passing through the fingertip and the first finger root and a line segment passing through the fingertip and the second finger root When the angle to be performed is less than a predetermined angle, the fingertip and a region surrounded by the root of the first finger and the root of the second finger adjacent to each other are specified as the finger region. 2. The recognition device according to 2 or 3.

(Additional remark 5) The said specific | specification part specifies the area | region of the object containing each finger area | region as a hand area, when the number of the specified finger area | regions is 5, Any one of Additional remarks 1-4 characterized by the above-mentioned. The recognition device described in 1.

(Supplementary Note 6) When the value obtained by subtracting the base position of the finger from the position of the fingertip is a positive value, the gesture recognition unit recognizes that the hand region is tilted forward, and the fingertip When the value obtained by subtracting the base position of the finger from the position is a negative value, it is recognized that the hand region is tilted backward. The recognition device described.

(Additional remark 7) The said specific | specification part specifies the intersection of the straight line which passes through the root of several fingers, and the outline of the said object as a new finger root, and specifies a finger area | region using the specified new finger root The recognition device according to any one of supplementary notes 1 to 6, wherein:

(Appendix 8) A recognition method executed by a computer,
Obtain distance information from the user side to the object that exists in the space in the user's line-of-sight direction as a distance image from the distance sensor that acquires distance information to the object in the space in the user's line-of-sight direction,
Based on the distance between the feature point of the object represented as the distance image and the reference point, the finger region is identified by identifying the fingertip and the root of the finger, and the hand region is identified based on the identified finger region. And
A recognition method, comprising: performing a process of recognizing a gesture type based on a difference value between the position of the fingertip included in the hand region and the position of the base of the finger.

(Additional remark 9) The said process to specify selects the three feature points adjacent on the said object, the 1st distance of the said reference point and a 1st feature point, and the 2nd of the said reference point and a 2nd feature point. Calculating a distance, a third distance between the reference point and the third feature point, and when the second distance is greater than the first distance and the second distance is greater than the third distance, The recognition method according to appendix 8, wherein the second point is identified as a fingertip point.

(Supplementary Note 10) The specifying process selects three adjacent feature points on the object, a first distance between the reference point and the first feature point, and a second distance between the reference point and the second feature point. Calculating a distance, a third distance between the reference point and the third feature point, and when the second distance is smaller than the first distance and the second distance is smaller than the third distance, The recognition method according to appendix 8 or 9, wherein the second point is identified as a root point.

(Additional remark 11) The process to specify selects the said fingertip, the base of the 1st finger | toe adjacent, and the base of the 2nd finger, the distance from the said fingertip to the root of the 1st finger, and the said fingertip A difference between the distance to the root of the second finger is less than a threshold and a line segment passing through the fingertip and the first finger root and a line segment passing through the fingertip and the second finger root Note that when the formed angle is less than a predetermined angle, an area surrounded by the fingertip and the base of the first finger and the base of the second finger adjacent to each other is specified as the finger area. The recognition method according to 8, 9 or 10.

(Additional remark 12) When the number of the specified finger area | regions is 5, the process to specify specifies the area | region of the object containing each finger area | region as a hand area, Any one of Additional remarks 8-11 characterized by the above-mentioned. The recognition method described in one.

(Supplementary Note 13) When the value obtained by subtracting the base position of the finger from the position of the fingertip is a positive value, the recognizing process recognizes that the hand region is tilted forward, and the fingertip If the value obtained by subtracting the base position of the finger from the position is a negative value, it is recognized that the hand region is tilted backward. The recognition method described.

(Supplementary note 14) The specifying process specifies an intersection of a straight line passing through a plurality of finger roots and an outline of the object as a new finger root, and uses the specified new finger root to specify a finger region. The recognition method according to any one of appendices 8 to 13, characterized in that the recognition method is specified.

(Supplementary note 15)
Obtain distance information from the user side to the object that exists in the space in the user's line-of-sight direction as a distance image from the distance sensor that acquires distance information to the object in the space in the user's line-of-sight direction,
Based on the distance between the feature point of the object represented as the distance image and the reference point, the finger region is identified by identifying the fingertip and the root of the finger, and the hand region is identified based on the identified finger region. And
A recognition program that executes a process of recognizing a gesture type based on a difference value between the position of the fingertip and the position of the base of the finger included in the hand region.

(Supplementary Note 16) In the specifying process, three adjacent feature points on the object are selected, a first distance between the reference point and the first feature point, and a second distance between the reference point and the second feature point. Calculating a distance, a third distance between the reference point and the third feature point, and when the second distance is greater than the first distance and the second distance is greater than the third distance, The recognition program according to appendix 15, wherein the second point is identified as a fingertip point.

(Supplementary Note 17) The specifying process selects three adjacent feature points on the object, a first distance between the reference point and the first feature point, and a second distance between the reference point and the second feature point. Calculating a distance, a third distance between the reference point and the third feature point, and when the second distance is smaller than the first distance and the second distance is smaller than the third distance, The recognition program according to appendix 15 or 16, wherein the second point is identified as a point of a root of a finger.

(Supplementary note 18) The specifying process selects the fingertip, the root of the first finger adjacent to the fingertip, and the root of the second finger, the distance from the fingertip to the root of the first finger, and the fingertip A difference between the distance to the root of the second finger is less than a threshold and a line segment passing through the fingertip and the first finger root and a line segment passing through the fingertip and the second finger root Note that when the formed angle is less than a predetermined angle, an area surrounded by the fingertip and the base of the first finger and the base of the second finger adjacent to each other is specified as the finger area. The recognition program according to 15, 16 or 17.

(Additional remark 19) When the number of the specified finger area | regions is 5, the process to specify specifies the area | region of the object containing each finger area | region as a hand area, Any one of Additional remarks 15-18 characterized by the above-mentioned. The recognition program described in 1.

(Supplementary note 20) When the value obtained by subtracting the base position of the finger from the position of the fingertip is a positive value, the recognizing process recognizes that the hand region is tilted forward, and the fingertip If the value obtained by subtracting the base position of the finger from the position is a negative value, it is recognized that the hand region is tilted backward. The recognition program described.

(Supplementary note 21) The specifying process specifies an intersection of a straight line passing through a plurality of finger roots and the contour of the object as a new finger root, and uses the specified new finger root to specify a finger region. The recognition program according to any one of supplementary notes 15 to 20, characterized in that the recognition program is specified.

DESCRIPTION OF SYMBOLS 50 Distance image sensor 60 Display apparatus 100 Recognition apparatus 110 Acquisition part 120 Identification part 130 Gesture recognition part 140 Display control part

Claims (9)

A distance sensor for acquiring distance information from the user side to an object existing in space in the direction of the user's line of sight;
An acquisition unit that acquires, as a distance image, distance information from the distance sensor to an object that exists in a space in a user's line-of-sight direction;
Based on the distance between the feature point of the object represented as the distance image and the reference point, the finger region is identified by identifying the fingertip and the root of the finger, and the hand region is identified based on the identified finger region. Specific part to do,
A gesture recognition unit, comprising: a gesture recognition unit that recognizes a gesture type based on a difference value between the position of the fingertip included in the hand region and the position of the base of the finger.   The specifying unit selects three adjacent feature points on the object, a first distance between the reference point and the first feature point, a second distance between the reference point and the second feature point, and the reference point A third distance to the third feature point is calculated, and when the second distance is greater than the first distance and the second distance is greater than the third distance, the second point is determined as a fingertip. The recognition device according to claim 1, wherein the recognition device is identified as a point.   The specifying unit selects three adjacent feature points on the object, a first distance between the reference point and the first feature point, a second distance between the reference point and the second feature point, and the reference point A third distance to the third feature point is calculated, and when the second distance is smaller than the first distance and the second distance is smaller than the third distance, the second point is determined as a root The recognition device according to claim 1, wherein the recognition device is identified as a point.   The identification unit selects the fingertip, the root of the first finger and the root of the second finger adjacent to each other, the distance from the fingertip to the root of the first finger, and the distance between the fingertip and the second finger An angle formed by a line segment passing through the roots of the fingertip and the first finger and a line segment passing through the roots of the fingertip and the second finger has a difference from a distance to the root being less than a threshold value. When the angle is less than a predetermined angle, an area surrounded by the fingertip and the root of the first finger and the root of the second finger adjacent to each other is specified as the finger area. 4. The recognition device according to 3.   The said specific | specification part specifies the area | region of the object containing each finger | toe area | region as a hand area | region, when the number of the specified finger | toe area | region is five, The one of Claim 1 characterized by the above-mentioned. Recognition device.   The gesture recognition unit recognizes that the hand region is tilted forward when the value obtained by subtracting the position of the fingertip from the position of the fingertip is a positive value, and the position of the fingertip The recognition according to any one of claims 1 to 5, wherein when the value obtained by subtracting the base position of the finger is a negative value, the hand region is recognized to be inclined backward. apparatus.   The specifying unit specifies an intersection of a straight line passing through a plurality of finger roots and the contour of the object as a new finger root, and specifies a finger region using the specified new finger root. The recognition device according to any one of claims 1 to 6. A recognition method performed by a computer,
Obtain distance information from the user side to the object that exists in the space in the user's line-of-sight direction as a distance image from the distance sensor that acquires distance information to the object in the space in the user's line-of-sight direction,
Based on the distance between the feature point of the object represented as the distance image and the reference point, the finger region is identified by identifying the fingertip and the root of the finger, and the hand region is identified based on the identified finger region. And
A recognition method, comprising: performing a process of recognizing a gesture type based on a difference value between the position of the fingertip included in the hand region and the position of the base of the finger. On the computer,
Obtain distance information from the user side to the object that exists in the space in the user's line-of-sight direction as a distance image from the distance sensor that acquires distance information to the object in the space in the user's line-of-sight direction,
Based on the distance between the feature point of the object represented as the distance image and the reference point, the finger region is identified by identifying the fingertip and the root of the finger, and the hand region is identified based on the identified finger region. And
A recognition program that executes a process of recognizing a gesture type based on a difference value between the position of the fingertip and the position of the base of the finger included in the hand region.

Images