JP2016045670A - Gesture management system, gesture management program, gesture management method and finger pointing recognition device - Google Patents

Abstract

CONSTITUTION: A gesture management system (100) includes such as a plurality of distance image sensors 12 installed on a ceiling of a space and the distance image sensor 12 outputs a distance image. For example, when a human being H gestures by pointing a finger, body characteristic points such as a head top part (HT) or two shoulder positions (Sn) are detected from each of the distance images outputted by the distance image sensors 12. Then by using such as the body characteristic points, finger pointing direction (Px) of the finger pointing gesture is detected. Among the distance image sensors 12, the distance image sensor 12 detecting the finger pointing direction as well as with the greatest plotted range indicating the human being H is specified. Then, the finger pointing direction corresponding to the specified distance image sensor 12 is registered as a recognition result of the finger pointing gesture.EFFECT: A gesture is capable of being suitably recognized even when a human being H carries out a finger pointing gesture in a randomly selected position in a space.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a gesture management system, a gesture management program, a gesture management method, and a pointing recognition device, and more particularly, to a gesture management system, a gesture management program, a gesture management method, and a pointing recognition device that recognizes a gesture using an arm, for example.

  An example of background art is disclosed in Patent Document 1. In the measuring device of this patent document 1, a person's position and head direction can be measured in real time using a plurality of three-dimensional distance measuring sensors.

  Moreover, in the gesture switch of patent document 2, the input space for receiving the input by a predetermined gesture is set in the detection target area. When a person makes a predetermined gesture in the vicinity of the input space, the gesture is recognized.

JP 2012-215555 A [G01S 17/89, G01S 17/66, G01C 3/06, G06T 7/20, G06T 7/60] JP 2006-99749 A [G06F 3/033, G06T 7/20, G01B 11/00, G01B 11/24]

  However, the measuring device of Patent Document 1 can measure a human position and the like, but cannot recognize a gesture using an arm such as a pointing finger. Further, the gesture switch of Patent Document 2 cannot recognize a predetermined gesture unless it is in the vicinity of the input space set in the detection target area. In other words, this gesture switch cannot recognize a gesture using an arm performed at an arbitrary position in a space where a human or the like moves.

  Therefore, a main object of the present invention is to provide a novel gesture management system, gesture management program, gesture management method, and pointing recognition device.

  Another object of the present invention is to provide a gesture management system, a gesture management program, and a gesture management method that can appropriately recognize a gesture using an arm performed at an arbitrary position.

  Another object of the present invention is to provide a pointing device that can properly recognize a pointing device.

  The present invention employs the following configuration in order to solve the above problems. The reference numerals in parentheses, supplementary explanations, and the like indicate the corresponding relationship with the embodiments described in order to help understanding of the present invention, and do not limit the present invention.

  The first invention includes a plurality of distance image sensors, a recognition means for recognizing a gesture using an arm for each distance image sensor, a storage means for storing a distance image sensor used for recognition of a gesture using an arm, and a storage means. Identification means for identifying the distance image sensor used when the gesture using the arm is appropriately recognized from the stored distance image sensor, and the arm recognized by using the distance image sensor identified by the identification means It is a gesture management system provided with a registration means for registering a gesture using a character as a recognition result.

  In the first invention, the gesture management system (100: reference numerals exemplifying corresponding parts in the embodiment, hereinafter the same) is used in a space where a plurality of distance image sensors (12) are provided on the ceiling, for example. . The recognition means (60, S1) recognizes a gesture using an arm in association with a plurality of distance image sensors. A memory | storage means (60, S5) memorize | stores the identification information of the distance image sensor utilized for recognition of the gesture using an arm among several distance image sensors, for example. The specifying means (60, S7) specifies the distance image sensor used when the gesture using the arm is properly recognized from the distance image sensors used for recognizing the gesture using the arm. The registration means (60, S15) registers, for example, a gesture using an arm recognized by using the specified distance image sensor as a recognition result.

  According to the first invention, for example, even if a human makes a gesture using an arm at an arbitrary position in the space, a distance image sensor that appropriately recognizes the gesture using the arm can be specified. Therefore, even if a human makes a gesture using an arm at an arbitrary position in the space, the gesture can be appropriately recognized.

  The second invention is dependent on the first invention, the distance image sensor outputs a distance image, and the specifying means is a size of a human performing a gesture using an arm in the distance image output from the distance image sensor. Based on this, the distance image sensor used when the gesture using the arm is properly recognized is specified.

  In the second invention, each of the plurality of distance image sensors outputs a distance image. Based on the human size in the output distance image, the distance image sensor used when the gesture using the arm is appropriately recognized is specified.

  The third invention is dependent on the second invention, and the size of the person is indicated by the number of pixels of the distance image.

  In the third invention, for example, the plurality of distance image sensors are provided at different positions. Therefore, in the distance image output from each distance image sensor, the number of pixels indicating the size of a person is different.

  According to the second invention or the third invention, the time until the distance image sensor is specified can be shortened by using the human size in the distance image.

  According to a fourth aspect of the present invention, there is provided a distance image sensor according to any one of the first to third aspects of the present invention, the distance image sensor capable of observing the gesture among the distance image sensors not used for the recognition of the gesture using the arm. The registration means further includes a search means, and when the distance image sensor capable of observing the gesture using the arm is not found by the search means, the registration means uses the distance image sensor specified by the specifying means to recognize the recognized arm. Register the used gesture as a recognition result.

  In the fourth invention, the retrieval means (60, S11) retrieves a distance image sensor capable of observing the gesture from the distance image sensors that are not used for recognizing the gesture using the arm. If no distance image sensor capable of observing the gesture using the arm is found by the search means, the registration means registers the gesture using the arm recognized by the specified distance image sensor as the recognition result. For example, when a distance image sensor capable of observing a gesture using an arm is found, there is a high possibility that the gesture using the arm is erroneously recognized. That is, it is possible to confirm whether or not a gesture using the arm has been erroneously recognized by the search.

  According to the fourth aspect of the present invention, it is possible to improve the accuracy of integrating the recognition results by checking whether the gesture using the arms is erroneously recognized before integrating the recognition results of the gestures using the arms. .

  The fifth invention is dependent on any one of the first to fourth inventions, and the gesture using the arm includes a pointing gesture, and the registration unit registers the pointing direction (Px) in the pointing gesture.

  According to the fifth aspect, the pointing direction can be registered as the recognition result of the pointing gesture.

  According to a sixth aspect of the present invention, there is provided a gesture management system processor having a plurality of distance image sensors, a recognition means for recognizing a gesture using an arm for each distance image sensor, and a distance image sensor used for recognizing a gesture using an arm. Storing means for storing the distance image sensor, the specifying means for specifying the distance image sensor used when the gesture using the arm is properly recognized from the distance image sensor stored by the storing means, and the distance image specified by the specifying means It is a gesture management program for causing a gesture using an arm recognized using a sensor to function as a registration unit for registering as a recognition result.

  In the sixth invention, the same effect as in the first invention can be obtained.

  A seventh invention is a gesture management method in a gesture management system having a plurality of distance image sensors, wherein a processor of the gesture management system recognizes a gesture using an arm for each distance image sensor. A storage step for storing the distance image sensor used for recognizing the used gesture, and a specification for specifying the distance image sensor used when the gesture using the arm is properly recognized from the distance image sensor stored in the storage step. This is a gesture management method for executing a registration step of registering a gesture using an arm recognized using the step and the distance image sensor specified in the specifying step as a recognition result.

  In the seventh invention, the same effect as in the first invention can be obtained.

  According to an eighth aspect of the present invention, there is provided a distance image sensor that outputs a distance image, a first extraction unit that extracts a feature point of a body based on a distance image of a space in which a human exists, a candidate point from a distance image based on the feature point When there is a region including a feature point and a candidate point when there is a region including a feature point and a candidate point, a first storage unit that stores the region as an arm region, and a feature point and a candidate point included in the arm region, A pointing device recognizing device comprising second storage means for storing a pointing direction.

  In the eighth invention, the plurality of distance image sensors (12) of the pointing finger recognition device (10) are provided, for example, on the ceiling of a space where a human is present. The first extraction means (60, S33) extracts a plurality of feature points of the human body from the distance image output by the distance image sensor, for example, when a person is in the detection range of the distance image sensor. The second extraction means (60, S35) extracts, for example, a feature point (X) that is separated from the feature point by a certain distance or more from the distance image. For example, when a feature point is included in a region resulting from combining points around the same distance as a candidate point farthest from the feature point, the first storage means (60, S51) uses the region of the combination result as an arm. Store as an area. The second storage means (60, S59) stores, for example, a vector indicated by a candidate point and a feature point farthest from the feature point included in the arm region as a pointing direction.

  According to the eighth aspect, by using the human feature point, it is possible to appropriately detect the pointing direction of the pointing gesture performed by the human.

  A ninth invention is according to the eighth invention, and the first storage means stores the area as an arm area when the area including the feature point and the candidate point has a feature as an arm. .

  In the ninth invention, the feature as an arm (region) is that, for example, the region resulting from the combination has an elongated shape and has a certain length. And when the area | region containing a feature point and a candidate point has the above-mentioned characteristic, the area | region is memorize | stored as an arm area | region.

  According to the ninth aspect, it is possible to increase the reliability when recognizing the pointing gesture by determining whether the region including the feature point and the candidate point has the arm feature.

  A tenth invention is dependent on the eighth invention or the ninth invention, and the second storage means is a candidate included in the feature point and the arm region when the angle based on the arm region and the human is within a predetermined angle range. Based on the points, the pointing direction is stored.

  In the tenth invention, for example, the predetermined angle range is a range including the angle of the arm when performing a pointing gesture. For example, when the angle based on the arm region and the person is within a predetermined angle range, the pointing direction is stored based on the feature points and the candidate points included in the arm region.

  According to the tenth aspect, by determining that the angle based on the arm region and the person is within the predetermined range angle, the possibility that the pointing gesture is erroneously recognized by the arm that is not performing the pointing gesture is reduced.

  The eleventh invention is dependent on any of the eighth to tenth inventions, and the second extraction means extracts a plurality of candidate points from the distance image based on the feature points, and the first storage means When three or more arm regions are stored, a deletion unit that deletes the pointing direction stored by the second storage unit is further provided.

  In the eleventh invention, the second extracting means extracts a plurality of candidate points from the distance image based on the feature points. When three or more arm regions are stored, the deleting means (60, S63) deletes the stored pointing direction.

  The twelfth invention is dependent on the eleventh invention, and the deleting means stores the second arm area when a new arm region is stored by the first storage means and the pointing direction is already stored by the second storage means. The pointing direction stored by the means is deleted.

  In the twelfth aspect, for example, when the pointing direction is already stored using another arm region, the pointing direction already stored is deleted.

  According to the eleventh aspect or the twelfth aspect, it is possible to cancel the recognition result of the pointing gesture that is likely to be erroneously recognized.

  A thirteenth invention is dependent on any of the eighth to twelfth inventions, wherein the feature points include a human head position and a shoulder position, and the second extracting means is a distance image based on the shoulder position. The first storage means stores the area as an arm area when there is an area including the shoulder position and the candidate point, and the second storage means stores the midpoint of the head position and the shoulder position. And the pointing direction is stored based on the candidate points included in the arm region.

  In the thirteenth invention, the feature points include a crown (HT) indicating a human head position and two shoulder positions (Sn). For example, candidate points are extracted from the distance image based on the top of the head. For example, if a shoulder position and a candidate point are included in a region where peripheral points approximately the same distance as the candidate point are combined, that region is stored as an arm region. For example, the pointing direction is stored based on the midpoints of the top and shoulder positions and the candidate points included in the arm region.

  According to the thirteenth aspect, the pointing direction can be stored using the human head position and shoulder position.

  According to the present invention, it is possible to appropriately recognize a gesture using an arm performed at an arbitrary position.

  Further, by using a human feature point, the pointing direction of the pointing gesture performed by the human can be appropriately detected.

  The above object, other objects, features, and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

FIG. 1 is an illustrative view showing an outline of a gesture management system according to an embodiment of the present invention. FIG. 2 is an illustrative view showing one example of a plan view of a space in which the gesture management system shown in FIG. 1 is used. 3 is an illustrative view showing one example of a detection range of the distance image sensor shown in FIG. FIG. 4 is an illustrative view showing one example of a configuration of the gesture management system shown in FIG. FIG. 5 is a block diagram showing an example of the electrical configuration of the distance image sensor shown in FIG. FIG. 6 is a block diagram showing an example of an electrical configuration of the central controller shown in FIG. 7 is an illustrative view showing one example of the feature points and pointing direction of the human body shown in FIG. 1, FIG. 7 (A) shows an example of the upper body of the human, and FIG. 7 (B) shows FIG. 7 (A). It is an illustration figure which shows an example of the pointing direction based on the feature point of the human body shown in FIG. FIG. 8 is an illustrative view showing one example of a distance image output from the distance image sensor shown in FIG. FIG. 9 is an illustrative view showing an example of a processing flow for detecting the pointing direction from the distance image shown in FIG. 8, and FIG. 9A shows a state in which the body feature points are detected from the distance image shown in FIG. An example is shown, and FIG. 9B is an illustrative view showing an example of the pointing direction detected based on the feature points of the body shown in FIG. FIG. 10 is an illustrative view showing an example of a distance image output from a plurality of distance image sensors shown in FIG. 1, and FIG. 10 (A) shows an example of a distance image output from a certain distance image sensor. 10 (B) shows an example of a state in which feature points of the body are detected in the distance image shown in FIG. 10 (A), and FIG. 10 (C) shows an example of a distance image output from another certain distance image sensor. 10D shows an example of a state in which the pointing direction is detected in the distance image shown in FIG. 10C, and FIG. 10E shows a distance image output from some other distance image sensor. FIG. 10F shows an example of a state in which body feature points are detected in the distance image shown in FIG. FIG. 11 is an illustrative view showing one example of an angle formed by a vector connecting the distance image sensor and the person shown in FIG. 4 and a vector in the pointing direction. FIG. 12 is an illustrative view showing one example of a memory map of a memory of the central control unit shown in FIG. FIG. 13 is a flowchart showing an example of the pointing direction management process of the processor of the central controller shown in FIG. FIG. 14 is a flowchart showing an example of a part of the pointing recognition process of the processor of the central control unit shown in FIG. FIG. 15 is an example of another part of the pointing recognition processing of the processor of the central controller shown in FIG. 6 and is a flowchart subsequent to FIG.

  With reference to FIG. 1, the gesture management system 100 according to the embodiment is used in a space (environment) such as a shopping mall, a museum, and an exhibition hall where a human H can freely come and go. For example, when the person H in the space performs a gesture using an arm such as a pointing gesture, the gesture management system 100 recognizes the gesture.

  In order to recognize a gesture using an arm, a plurality of distance image sensors 12 (12a,...) For detecting the position of the person H and the direction of the body in the space are provided on the ceiling in the space. It has been. The plurality of distance image sensors 12 are respectively connected to a central control device 10 (see FIG. 4) described later.

  Note that only one person H is present in the space of the present embodiment, but more persons H may be present in the space in other embodiments. Further, not only the human H but also a robot or the like that can act independently may be in the space.

  FIG. 2 is a plan view (map) showing the planar position of such a space. Referring to FIG. 2, a plurality of distance image sensors 12 (14 in this embodiment) are provided in the space. These distance image sensors 12 are one set except for those provided at the end of the space (12a, 12b, 12m, 12n). Referring also to FIG. 1, for example, the distance image sensor 12c and the distance image sensor 12d are set as one set. In one set of distance image sensors 12, the sensor surfaces face each other, and the sensor surface is inclined downward by a certain angle (for example, about 10 °) from the vertically downward direction. This is because the detection range of the distance image sensor 12 becomes wider when the sensor surface is tilted than when the sensor surface is directed directly below. In addition, when the sensor surface is tilted, the detection surface of one set of distance image sensors 12 is tilted toward the other party so that the detection ranges do not overlap each other and become inefficient detection. . Further, in order to prevent the detection range in one set of range image sensors 12 from overlapping with the other set of range image sensors 12 and resulting in inefficient detection, the horizontal position of each set of range image sensors 12 is determined. Is provided with a gap.

  The distance image sensor 12 provided at the end of the space is provided such that the sensor surface faces the inside of the space.

  In such a space, when the person H is substantially below the distance image sensor 12k and the distance image sensor 121, the position of the person H and the body orientation are shown on the plan view as shown in FIG. . In another embodiment, the number of distance image sensors 12 provided in the space may be 13 or less, or 15 or more.

  Referring to FIG. 3, one distance image sensor 12 is provided at a position of about 3.0 m from the ground. Further, since the sensor surface is inclined at a certain angle from the vertically downward direction, the detection range is substantially elliptical. In this detection range, the portion corresponding to the major axis of the ellipse is about 6.0 m, and the portion corresponding to the minor axis of the ellipse is about 4.5 m. In the present embodiment, the positions where the plurality of distance image sensors 12 are provided are adjusted so that the detection ranges shown in FIG.

  Referring to FIG. 4, a plurality of distance image sensors 12 are connected to central control device 10 of gesture management system 100. The central controller 10 acquires a distance image output from the distance image sensor 12 at regular time intervals. Then, the central control device 10 detects the position information of the person H from each acquired distance image. In the present embodiment, since the central control device 10 executes a process of recognizing a pointing gesture, the central control device 10 may be referred to as a “pointing recognition device”.

  FIG. 5 is a block diagram showing an electrical configuration of the distance image sensor 12. Referring to FIG. 5, the distance image sensor 12 includes a control IC 30 and the like. An A / D converter 32, a camera 36, a depth sensor 38, a depth camera 40, an I / O 42, and the like are connected to the control IC 30.

  The control IC 30 includes a cache memory and the like, and controls the operation of the distance image sensor 12. For example, the control IC 30 operates according to a command from the central control device 10 and transmits the detected result (distance image) to the central control device 10.

  A microphone 34 is connected to the A / D converter 32, and an audio signal from the microphone 34 is converted into a digital audio signal by the A / D converter 32 and input to the control IC 30. The sound collected by the microphone 34 may be used for measuring the volume of noise or the like in the space.

  The camera 36 is a camera for photographing RGB information of the space in which the distance image sensor 12 is installed, that is, a color image. In addition, the camera 36 is provided in the distance image sensor 12 so as to be able to capture a space that is substantially the same as the space that is captured by the depth camera 40 described later.

  The depth sensor 38 is an infrared projector, for example, and the depth camera 40 is an infrared camera, for example. The depth sensor 38 irradiates the front of the distance image sensor 12 with, for example, infrared laser light. A special pattern is drawn in the space by the irradiated laser light, and the depth camera 40 captures the drawn pattern. The captured image is input to the control IC 30, and the control IC 30 analyzes the image to measure depth information of the space irradiated with the laser light.

  The I / O 42 is a digital port capable of input / output control, and an audio signal, a distance image including RGB information and depth information is output from the output port, and is provided to the central controller 10. On the other hand, a control signal is output from the central controller 10 and applied to the input port.

  The distance image sensor 12 outputs RGB information and depth information, and is therefore sometimes referred to as an RGB-D sensor.

  The distance image sensor 12 of this embodiment employs a product called a Kinect (registered trademark) sensor manufactured by Microsoft (registered trademark). However, in another embodiment, Xtion manufactured by ASUS (registered trademark), D-Imager (registered trademark) manufactured by Panasonic (registered trademark), or the like may be employed as the distance image sensor 12.

  FIG. 6 is a block diagram showing an electrical configuration of the central controller 10. Referring to FIG. 6, central controller 10 includes a distance image sensor 12 and a processor 60. The processor 60 is sometimes called a microcomputer or CPU. A plurality of distance image sensors 12, a memory 62, an output device 64, an input device 66, and the like are connected to the processor 60.

  The distance image sensor 12 outputs a distance image or the like as described above. The depth information included in the distance image includes the shape of the person H in the space and the distance to the person H. For example, when the human H is sensed by the distance image sensor 12 provided on the ceiling, the head H HT (head position), the frontal head HF, and the two shoulder positions Sn, which are characteristic points of the human H body, are detected. The distance and the three-dimensional shape (for example, the shape of the head and both shoulders) of the person H including the feature point are obtained.

  Further, 14 distance image sensors 12 are installed at predetermined positions (known) in the space, and the processor 60 acquires distance images from each of them, and the position of the human H in the space (world coordinate system) ( For example, position coordinates such as the center of gravity) and direction (angle) can be calculated.

  Refer to Patent Document 1 (Japanese Patent Laid-Open No. 2012-215555) of this specification for a specific method for obtaining the feature point, three-dimensional shape, position, and orientation of the human H body.

  Further, since the distance image sensor 12 includes the microphone 34, the central control device 10 can also estimate the sound generation source from the position of the distance image sensor 12 to which sound information is input.

  In another embodiment, not the distance image sensor 12 but a three-dimensional laser range finder (LRF) may be used.

  The processor 60 includes an RTC that controls the operation of the central controller 10 and outputs date and time information. The memory 62 is also called storage means, and includes a ROM, an HDD, and a RAM. In the ROM and the HDD, a control program for controlling the operation of the central controller 10 is stored in advance. The RAM is used as a work memory or a buffer memory for the processor 60.

  The output device 64 is, for example, a display, and the input device 66 is, for example, a mouse or a keyboard. The administrator of the gesture management system 100 can confirm and use the state of the central control device 10 using the output device 64 and the input device 66.

  With reference to FIG. 7A and FIG. 7B, the pointing direction Px when the human H is performing a pointing gesture will be described. As shown in FIG. 7A, for the human H who is making a pointing gesture with the right hand, the above-described head top portion HT, frontal head portion HF, and two shoulder positions Sn are determined, and the tip of the pointing finger gesture, that is, the fingertip Determine the position of P. Next, the midpoint C of the crown position HT and the shoulder position Sn of the arm where the pointing gesture is performed are obtained. Since the midpoint C is located near the eyes of the human H, it is considered that an object pointed by the human H exists on a line connecting the midpoint C and the fingertip P. Therefore, in the present embodiment, a vector connecting the middle point C and the fingertip P is set as the pointing direction Px. Hereinafter, by obtaining these positions for the person H in the space, the gesture using the arm, that is, the pointing direction Px of the pointing gesture is recognized (detected).

  FIG. 8 is an illustrative view showing an example of a distance image output from a certain distance image sensor 12. Referring to FIG. 8, a person H who performs a pointing gesture is captured by a certain distance image sensor 12, and a region indicating the person H who performs a pointing gesture is included on the right side of the approximate center of the distance image. In this embodiment, the position of each part shown in FIG. 7 is detected from such a distance image, and the pointing direction Px of the person H is detected. In the actual distance image, black and white shading changes according to the detected distance, but in the distance image in FIG. 8, the shading indicating the distance is omitted for simplicity of illustration. .

  Referring to FIG. 9A, when a region indicating human H is detected in a distance image output from a certain distance image sensor 12, in the region indicating human H, the parietal portion HT and frontal head HF are displayed. And two shoulder positions Sn are shown respectively.

  Next, a point X that is more than a certain distance away from the crown HT is extracted and stored in the candidate point list. From the points X stored in the candidate point list, the point X that is the tip portion (fingertip P) of the pointing gesture is obtained. Specifically, first, a point X farthest from the top HT is specified, and points around the same distance as the farthest point X are combined (clustered). When the combination of the surrounding points is completed, it is determined whether one of the shoulder positions Sn is included in the combination result.

  When the joining result including the shoulder position Sn is obtained, it is determined whether the joining result has the feature of the arm region. As described above, by determining whether or not the region indicating the arm on which the pointing gesture is performed has the characteristics of the arm region, it is possible to improve the reliability when the pointing gesture is recognized. The feature of the arm region is that the region has an elongated shape and has a certain length. If the combined region has the above-described arm region characteristics, it is determined whether two or more other arm regions have not been detected. That is, it is determined whether or not the arm is erroneously recognized.

  If the arm is not misrecognized, the angle based on the arm region and the human H, that is, the angle between the line segment of the fingertip P and the shoulder position Sn included in the arm region and the vertical segment of the human H (hereinafter referred to as the human segment H). It is determined whether the angle of the arm is within a predetermined angle range (for example, 60 ° to 180 °). In other words, it is determined whether the angle of the arm is within a range that is assumed as an arm that performs a pointing gesture. If the other finger pointing direction Px is not detected when the arm angle is within the predetermined angle range, the point X included in the arm region, that is, the vector connecting the fingertip P and the midpoint C described above is It is detected as the pointing direction Px. As described above, by determining that the angle of the arm is within the predetermined range angle, the possibility that the pointing gesture is erroneously recognized by the arm not performing the pointing gesture is reduced.

  With reference to FIG. 9B, the point X in the arm region is set to the fingertip P, and the midpoint C between the crown HT and the shoulder position Sn connected to the fingertip P is obtained as described above. Then, the direction of the fingertip P with respect to the middle point C is detected as the pointing direction (pointing vector) Px, and the pointing direction Px is stored together with the identification information of the distance image sensor 12.

  Further, when the joint result does not include the shoulder position Sn, when the joint result does not have the characteristics of the arm region, or when the arm angle is out of the predetermined angle range, the pointing direction Px is detected until the pointing direction Px is detected. The point X is selected again and the above process is repeated. However, if the pointing direction Px cannot be detected at all the points X stored in the candidate point list, there is a high possibility that the pointing gesture is not performed. finish.

  Further, when two or more other arm regions are detected or the pointing direction Px is already stored, the stored pointing direction Px and the arm region are deleted and the points included in the candidate point list All Xs are also deleted.

  For example, when the third arm is detected, there is a high possibility that the pointing gesture for the human H is not correctly recognized. Similarly, when the pointing direction Px is newly detected in the state where the pointing direction Px is stored, it is not normally considered that the human H points to two places at the same time, so the human H performs a pointing gesture. Most likely not. Therefore, by deleting the stored pointing direction Px and the arm region, the recognition result of the pointing gesture that is highly likely to be erroneously recognized is canceled.

  As can be understood from the above description, the pointing direction of the pointing gesture performed by the human H by using the characteristic points such as the head position and the shoulder position Sn of the human H in particular for detecting the position and direction of the human H. Px can be detected appropriately.

  In FIGS. 9A and 9B, in the area indicating the person H, shades indicating the distance are drawn. Further, the region indicating the person H in the distance image is detected by combining (clustering) surrounding points that are substantially the same distance as the body feature points. However, in another embodiment, a region indicating the person H may be detected from the distance image using another method.

  As described above, when the pointing gesture is recognized using the distance image sensors 12, the recognition results are integrated. Specifically, the distance image sensor 12 used when the finger pointing gesture is most appropriately recognized is specified from the distance image sensors 12 that have recognized the finger pointing gesture. When the distance image sensor 12 is specified, the pointing direction Px detected using the specified distance image sensor 12 is registered as a recognition result.

  First, a procedure for identifying the distance image sensor 12 used when the finger pointing gesture is most appropriately recognized will be described. When there are a plurality of distance image sensors 12 used for recognizing the pointing gesture, one distance image sensor 12 is specified based on the size of the person H who performs the pointing gesture in the distance image. In this embodiment, the distance image sensor 12 outputting the distance image having the largest number of pixels constituting the region indicating the person H is used as the distance image sensor 12 used when the pointing gesture is most appropriately recognized. Identified. Thus, by using the size of the person H in the distance image, it is possible to shorten the time until the distance image sensor 12 is specified.

  For example, FIG. 10A to FIG. 10F show distance images output from the distance image sensors 12a, 12e, and 12m when the human H shown in FIG. 2 is making a pointing gesture. FIG. 10A shows a distance image output from the distance image sensor 12b, and FIG. 10B shows a state where body feature points are detected in the distance image output from the distance image sensor 12b. Further, as can be seen from the distance image of FIG. 10B, this distance image does not include the arm region where the pointing gesture is performed, and the pointing direction Px is not detected.

  FIG. 10C shows a distance image output from the distance image sensor 12e, and FIG. 10D shows a state where body feature points are detected in the distance image output from the distance image sensor 12e. Further, as can be seen from the distance image in FIG. 10D, this distance image includes an arm region where a pointing gesture is performed, and the pointing direction Px is detected.

  Further, FIG. 10E shows a distance image output from the distance image sensor 12a, and FIG. 10F shows a state where body feature points are detected in the distance image output from the distance image sensor 12a. Further, as can be seen from the distance image in FIG. 10F, the distance image does not include the arm region where the pointing gesture is performed, and the pointing direction Px is not detected.

  Then, if the pointing gesture is not recognized by the other distance image sensors 12, the distance image sensor 12e that outputs the distance images of FIGS. 10C and 10D is specified. However, when there is another distance image sensor 12 that recognizes the pointing gesture, the distance image sensor 12 is specified based on the size of the region indicating the person H in the distance image.

  Next, when the distance image sensor 12 is specified, it is confirmed whether or not the pointing gesture is erroneously recognized. Specifically, a distance image sensor 12 that can observe the pointing direction Px (arm) of the pointing gesture performed by the person H is searched in the distance image sensor 12 that is not used for recognition of the pointing gesture.

  Referring to FIG. 11, distance image sensor 12 capable of observing pointing direction Px has a predetermined angle θ formed by vector SH connecting the position of distance image sensor 12 and human H and a vector indicating pointing direction Px. In the case of a value (for example, 15 °) or less, it is determined that the distance image sensor 12 cannot observe the pointing direction Px. That is, when the angle θ formed by the two vectors is larger than the predetermined value, it is determined that the distance image sensor 12 can observe the pointing direction Px.

  If no distance image sensor 12 that can observe the pointing direction Px is found in the distance image sensor 12 that is not used for recognizing the pointing gesture, the pointing direction Px is registered in the pointing direction table. That is, the detected pointing direction Px is the recognition result of the pointing gesture performed by the person H. As described above, it is possible to improve the accuracy of integrating the recognition results by confirming whether or not the pointing gesture is erroneously recognized before integrating the recognition results of the pointing gesture. Further, the pointing direction Px can be registered as a recognition result of the pointing gesture.

  When the pointing direction Px is registered, the pointing direction Px is used. For example, when a robot for guiding is installed in the space, it is possible to specify an object or the like that the human H is pointing with the pointing gesture using the pointing direction Px of the human H. In addition, when a robot capable of performing a pointing gesture is installed in the space, by recognizing the pointing direction Px of the pointing gesture by the robot, it is possible to determine whether the robot is pointing correctly to the object by the pointing gesture. It can be confirmed.

  As can be understood from the above description, even if the human H performs a pointing gesture at an arbitrary position in the space, the distance image sensor 12 that appropriately recognizes the pointing gesture can be specified. Therefore, even if the human H performs a gesture at an arbitrary position in the space, the gesture can be appropriately recognized.

  The features of the present embodiment have been outlined above. Hereinafter, the present embodiment will be described in detail with reference to the memory map of the memory 62 of the central control apparatus 10 shown in FIG. 12 and the flowcharts shown in FIGS.

  FIG. 12 is an illustrative view showing one example of a memory map of the memory 62 of the central controller 10 shown in FIG. As shown in FIG. 12, the memory 62 includes a program storage area 302 and a data storage area 304. In the program storage area 302, a pointing direction management program for integrating recognition results of pointing gestures (gestures using arms) recognized by the plurality of distance image sensors 12 as a program for operating the central control device 10. A finger pointing recognition program 312 for recognizing a finger pointing gesture by 310 and each distance image sensor 12 is stored. Although illustration is omitted, the program for operating the central control device 10 includes a program for editing the plan view data 340 and the like.

  In the data storage area 304, a pointing direction buffer 330, a candidate point list buffer 332, an arm candidate point list buffer 334, an arm area buffer 336, a temporary pointing direction buffer 338, and the like are provided. The data storage area 304 stores plan view data 340, a distance image sensor table 342, a pointing direction table 344, and the like.

  The pointing direction buffer 330 temporarily stores the pointing direction Px of the pointing gesture recognized by the specified distance image sensor 12. The candidate point list buffer 332 temporarily stores a candidate point list including a point X that is separated from the top HT by a certain distance or more. The arm candidate point list buffer 334 temporarily stores an arm candidate point list including a result of combining points around the point X. The arm area buffer 336 temporarily stores information indicating the arm area. The temporary pointing direction buffer 338 temporarily stores the pointing direction Px associated with the identification information of each distance image sensor 12.

  The plan view data 340 is data indicating the plan view shown in FIG. 2, for example. The distance image sensor table 342 is a table in which identification information indicating a distance image in which the pointing gesture is recognized is stored. The pointing direction table 344 is a table in which the pointing direction Px of the pointing gesture recognized by the specified distance image sensor 12 is stored (registered) in chronological order. Note that the plurality of pointing directions Px stored in the pointing direction table 344 represent changes in the pointing direction Px of the pointing gesture performed by the person H.

  Although not shown, the data storage area 304 is also provided with a buffer for temporarily storing various calculation results, and other counters and flags necessary for the operation of the central controller 10.

  The processor 60 of the central control device 10 includes a pointing direction management process shown in FIG. 13 and a pointing recognition process shown in FIGS. 14 and 15 under the control of a Linux (registered trademark) -based OS or other OS. Handle multiple tasks.

  FIG. 13 is a flowchart of the pointing direction management process. When the central controller 10 is turned on and an execution command for the pointing direction management process is issued, the pointing direction management process is executed. The pointing direction management process execution command is issued at regular intervals.

  When the pointing direction management process is executed, the processor 60 executes the pointing recognition process for each distance image sensor 12 in step S1. That is, a finger pointing recognition process described later is added to the distance image output from each distance image sensor 12. The processor 60 that executes the process of step S1 functions as a recognition unit.

  Subsequently, in step S3, the processor 60 determines whether or not there is the distance image sensor 12 that has recognized the pointing gesture. For example, it is determined whether the pointing direction Px is stored in the temporary pointing direction buffer 338. If “NO” in the step S3, that is, if there is no distance image sensor 12 used for recognizing the pointing gesture, the processor 60 ends the pointing direction management process.

  On the other hand, if “YES” in the step S3, as shown in FIG. 10D, for example, if the pointing direction Px is detected from the distance image output by the distance image sensor 12e, the processor 60 in the step S5, The distance image sensor 12 that recognizes the pointing gesture is stored in the distance image sensor table 342. That is, the processor 60 stores the pointing direction Px stored in the temporary pointing direction buffer 338 and associated with the identification information of the distance image sensor 12 in the distance image sensor table 342. The processor 60 that executes the process of step S5 functions as a storage unit.

  Subsequently, in step S7, the processor 60 specifies the distance image sensor 12 that most appropriately recognizes the pointing gesture. That is, in the distance image in which the pointing direction Px is detected, the distance image in which the area corresponding to the person H is drawn is specified. Then, the distance image sensor 12 corresponding to the identified distance image is set as the identified distance image sensor 12 in step S7. The processor 60 that executes the process of step S7 functions as a specifying unit.

  Subsequently, in step S <b> 9, the processor 60 stores the pointing direction Px corresponding to the identified distance image sensor 12 in the pointing direction buffer 330. For example, when the distance image sensor 12 e is specified, the pointing direction Px associated with the identification information of the distance image sensor 12 e is read from the temporary pointing direction buffer 338 and stored in the pointing direction buffer 330.

  Subsequently, in step S <b> 11, the processor 60 searches for the distance image sensor 12 that can observe the pointing direction Px from the distance image sensor 12 that has not recognized the pointing gesture. That is, it is confirmed whether or not the pointing gesture is misrecognized. The processor 60 that executes the process of step S11 functions as a search unit.

  Subsequently, in step S13, the processor 60 determines whether or not the observable distance image sensor 12 has been found. That is, it is determined whether the pointing gesture is not recognized by the distance image sensor 12 capable of recognizing the pointing gesture. If “YES” in the step S13, that is, if there is a possibility that the pointing gesture is erroneously recognized, the processor 60 ends the pointing direction management process.

  On the other hand, if “NO” in the step S13, that is, if there is no possibility that the pointing gesture is erroneously recognized, the processor 60 in the step S15 points the pointing direction Px stored in the pointing direction buffer 330. Register at 344. That is, the detected pointing direction Px is a recognition result of the pointing gesture. Then, when the process of step S15 ends, the processor 60 ends the pointing direction management process.

  The processor 60 that executes the process of step S15 functions as a registration unit. Further, when the process of step S15 is completed, each buffer is initialized.

  14 and 15 are flowcharts of the pointing finger recognition process. When step S1 for executing the pointing recognition process for each distance image sensor 12 is executed in the pointing management process shown in FIG. 13, the pointing recognition process is executed. In addition, after this recognition process is performed, it is completed in about 1/30 seconds, for example.

  For example, when a pointing recognition process is executed corresponding to the distance image sensor 12e, the processor 60 acquires a distance image in step S31. For example, the distance image output from the distance image sensor 12e is acquired. Subsequently, in step S33, the processor 60 extracts body feature points. That is, from the acquired distance image, the parietal region HT, the frontal region HF, and the two shoulder positions Sn are extracted as body feature points. Subsequently, in step S35, the processor 60 extracts a point X that is a predetermined distance or more from the head position on the plane coordinates. For example, a point X that is separated from the top of the head HT by a certain distance or more and is approximately the same distance as the human H is extracted. The processor 60 that executes the process of step S33 functions as a first extraction unit. The processor 60 that executes the process of step S35 functions as a second extraction unit.

  Subsequently, in step S37, the processor 60 stores the extracted point X in the candidate point list. That is, the point X extracted by the process of step S35 is stored in the candidate point list stored in the candidate point list buffer 332. If the point X has not been extracted, nothing is stored in the candidate point list.

  Subsequently, in step S39, the processor 60 determines whether or not the point X is stored in the candidate point list. For example, it is determined whether an area that is a candidate arm exists near the top HT of the human H. If “NO” in the step S39, for example, if the point X is not extracted in the process of the step S35, or if all the points X are deleted from the candidate point list in the process described later, the processor 60 ends the pointing recognition process. .

  On the other hand, if “YES” in the step S39, for example, if the point X extracted in the process of the step S35 is stored in the candidate point list, the processor 60 selects the point X farthest from the head position in the step S41. Read from the candidate point list. For example, the point X farthest from the top HT is read from the candidate point list. Subsequently, in step S43, the processor 60 combines the peripheral points that are approximately the same distance as the read point X. That is, at a point X farthest from the head position, points around the same distance are combined (clustered). Subsequently, in step S45, the processor 60 stores the combination result in the arm candidate point list. That is, all the points (regions) combined in the process of step S43 are stored in the arm candidate point list stored in the arm candidate point list buffer 334.

  Subsequently, in step S47, the processor 60 determines whether or not the shoulder position Sn is included in the combination result. That is, it is determined whether any one of the two shoulder positions Sn of the person H is included in the arm candidate point list. If “YES” in the step S47, that is, if the shoulder position Sn is included in the region of the joining result, the processor 60 determines whether or not the joining result has the feature of the arm region in a step S49. to decide. For example, it is determined whether the combined result area stored in the arm candidate point list has an elongated shape and a certain length. If “YES” in the step S49, that is, if the joining result has the characteristics of the arm region, the processor 60 stores the joining result as the arm region in a step S51. That is, an arm region is defined based on each point X stored in the arm candidate point list, and the arm region is stored in the arm region buffer 336. The processor 60 that executes the process of step S51 functions as a first storage unit.

  Subsequently, in step S53, the processor 60 determines whether or not there are three stored arm regions. That is, it is determined whether the arm is erroneously recognized. If “NO” in the step S53, that is, if the number of recognized arms is two or less, the processor 60 determines whether or not the arm angle is within a predetermined angle range in a step S55. That is, it is determined whether the angle of the human H's arm is an angle that is considered to be a pointing gesture. If “NO” in the step S55, that is, if the angle of the arm of the human H is an angle at which it cannot be considered that a pointing gesture is performed, the processor 60 proceeds to a process of step S61. That is, since it is determined that the finger pointing gesture is not performed with the recognized arm, the processor 60 proceeds to the process of step S61 in order to search for an arm region based on another point X.

  Similarly, the shoulder position Sn is not included in the group (area) of the combined points and “NO” is determined in step S47, or the combined result does not have the feature of the arm area and “NO” is determined in step S49. "If so, the processor 60 proceeds to the process of step S61. That is, even when it is determined that the combined region is not the arm of the human H, the processor 60 proceeds to the process of step S61 in order to search for an arm region based on another point X.

  If “YES” in the step S55, that is, if the arm angle is considered to be a pointing gesture, the processor 60 determines whether or not the pointing direction Px is already stored in the step S57. to decide. That is, it is determined whether the finger pointing gesture performed by another arm has already been recognized. Specifically, it is determined whether the pointing direction Px is already stored in the temporary pointing direction buffer 338.

  If “NO” in the step S57, that is, if the pointing gesture by another arm is not recognized, the processor 60 stores the pointing direction Px based on the body feature point and the arm region in a step S59. That is, the point X in the arm region is set as the fingertip P and the midpoint C described above is obtained, and the direction of the fingertip P relative to the midpoint C is stored (detected) as the pointing direction Px. The processor 60 that executes the process of step S59 functions as a second storage unit.

  Subsequently, in step S61, the processor 60 deletes a point corresponding to the combination result from the candidate point list. That is, the combination result (arm region) stored in the arm candidate point list is deleted from the candidate point list. Then, when the process of step S61 ends, the processor 60 returns to the process of step S39. In another embodiment, the pointing recognition process may be terminated when the pointing direction Px is detected (stored).

  On the other hand, if the arm region is detected and “YES” is determined in step S53, or if the pointing direction Px is already detected and “YES” is determined in step S57, the processor 60 stores in step S63. In step S65, all the points X stored in the candidate point list are deleted. That is, since there is a high possibility that the pointing gesture is misrecognized, the recognized arm region and the detected pointing direction Px are deleted. Further, since it is not necessary to detect other arm regions, all the points X are deleted from the candidate point list. Then, when the process of step S65 ends, the process returns to the process of the processor 39. At this time, since “NO” is determined in the step S39, the processor 60 ends the pointing recognition process. Further, the processor 60 that executes the process of step S63 functions as a deleting unit.

  In another embodiment, when the distance image sensor 12 that appropriately recognizes the pointing gesture is specified, the distance image sensor 12 may be determined based on the distance between the person H and the distance image sensor 12.

  In another embodiment, the gesture using the arm includes not only a pointing gesture but also a gesture that extends a palm and points to an object.

  In still another embodiment, it may be determined whether the gesture is performed correctly using the time during which the gesture is performed using the arm. For example, it is determined that the gesture using the arm is being performed when the time during which the gesture using the arm is performed is longer than a threshold (for example, 2 seconds).

  Further, in this embodiment, even if the robot has substantially the same appearance as the human H, the feature points of the body can be extracted.

  In the above-described embodiments, words such as “greater than” are used for the threshold (predetermined value) and the like, but “greater than the threshold” includes the meaning of “greater than or equal to the threshold”. Similarly, “smaller than a threshold” includes the meanings “below the threshold” and “below the threshold”.

  In addition, the plurality of programs described in the present embodiment may be stored in the HDD of a data distribution server and distributed to a system having the same configuration as that of the present embodiment via a network. In addition, storage programs such as CDs, DVDs, and BDs (Blu-ray (registered trademark) Disc) are sold or distributed with these programs stored in storage media such as USB memory and memory card. May be. When the plurality of programs downloaded through the server and storage medium described above are applied to a system having the same configuration as that of this embodiment, the same effect as that of this embodiment can be obtained.

  The specific numerical values given in this specification are merely examples, and can be appropriately changed according to a change in product specifications.

DESCRIPTION OF SYMBOLS 10 ... Central controller 12a-12n ... Distance image sensor 60 ... Processor 62 ... Memory 100 ... Gesture management system

Claims (13)

Multiple distance image sensors,
Recognition means for recognizing gestures using arms for each distance image sensor,
Storage means for storing a distance image sensor used for recognizing a gesture using the arm;
Using the distance image sensor specified by the specifying means, the specifying means for specifying the distance image sensor used when the gesture using the arm is properly recognized from the distance image sensor stored by the storage means A gesture management system comprising registration means for registering a gesture using an arm recognized as a recognition result. The distance image sensor outputs a distance image;
The distance image sensor used when the identification unit appropriately recognizes the gesture using the arm based on the size of the person performing the gesture using the arm in the distance image output from the distance image sensor. The gesture management system according to claim 1, wherein:   The gesture management system according to claim 2, wherein the human size is indicated by the number of pixels of the distance image. Searching means for searching for a distance image sensor capable of observing the gesture among distance image sensors that are not used for recognizing the gesture using the arm,
When the distance image sensor capable of observing the gesture using the arm is not found by the search unit, the registration unit performs a gesture using the arm recognized by using the distance image sensor specified by the specifying unit. The gesture management system according to claim 1, wherein the gesture management system is registered as a recognition result. The gesture using the arm includes a pointing gesture,
The gesture management system according to claim 1, wherein the registration unit registers a pointing direction in the pointing gesture. A gesture management system processor having a plurality of range image sensors;
Recognition means for recognizing gestures using arms for each distance image sensor,
Storage means for storing a distance image sensor used for recognizing a gesture using the arm;
Using the distance image sensor specified by the specifying means, the specifying means for specifying the distance image sensor used when the gesture using the arm is properly recognized from the distance image sensor stored by the storage means A gesture management program that causes a gesture using an arm recognized as a recognition means to function as a registration means for registering the result as a recognition result. A gesture management method in a gesture management system having a plurality of range image sensors, wherein the processor of the gesture management system includes:
A recognition step for recognizing a gesture using an arm for each distance image sensor;
Storing a distance image sensor used for recognizing the gesture using the arm;
A specifying step of specifying a distance image sensor used when the gesture using the arm is appropriately recognized from the distance image sensor stored in the storing step, and using the distance image sensor specified in the specifying step A gesture management method for executing a registration step of registering a gesture using an arm recognized as a recognition result. A distance image sensor that outputs a distance image,
First extraction means for extracting feature points of a body based on a distance image of a space in which a human exists;
Second extraction means for extracting candidate points from the distance image based on the feature points;
When there is an area including the feature point and the candidate point, a first storage unit that stores the area as an arm area, and stores a pointing direction based on the feature point and the candidate point included in the arm area A pointing device recognizing device comprising second storage means.   The pointing device according to claim 8, wherein the first storage unit stores the region as an arm region when the region including the feature point and the candidate point has a feature as an arm.   The said 2nd memory | storage means memorize | stores a pointing direction based on the said feature point and the candidate point contained in the said arm area | region when the angle based on the said arm area | region and the said person is in a predetermined angle range. A pointing device recognizing device according to 8 or 9. The second extraction means extracts a plurality of candidate points from the distance image based on the feature points,
11. The apparatus according to claim 8, further comprising a deletion unit that deletes the pointing direction stored by the second storage unit when three or more arm regions are stored by the first storage unit. Pointer recognition device.   The deletion unit deletes the pointing direction stored by the second storage unit when a new arm region is stored by the first storage unit and the pointing direction is already stored by the second storage unit. The pointing finger recognition apparatus according to claim 11. The feature points include the human head position and shoulder position,
The second extraction means extracts candidate points from the distance image based on the head position,
When there is an area including the shoulder position and the candidate point, the first storage means stores the area as an arm area,
The said 2nd memory | storage means memorize | stores a pointing direction based on the midpoint of the said head position and the said shoulder position, and the candidate point contained in the said arm area | region. Pointer recognition device.

Images