JP2014182662A - Operation apparatus and operation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an operation apparatus and an operation method capable of recognizing a shape and movement of a hand accurately at high speed, while reducing a processing load.SOLUTION: In hand area extraction processing S10, a hand area is extracted from image information obtained by imaging means. In fingertip shape recognition processing S20, a plurality of fingertip shapes are recognized from the hand area. In hand shape recognition processing S30, there is calculated, as an angle between fingers, an angle made by neighboring straight lines connecting a specific point within a palm or back of the hand in the hand area, and a specific point within the fingertip shapes. In pointing information calculation processing S40, an operation command is created for operating equipment according to a change of the angle between fingers. In command transmission processing S50, the operation command is transmitted to the equipment.

Description

  The present invention relates to an operating device and an operating method for operating a device such as a monitor device in accordance with the shape and movement of a human hand.

  2. Description of the Related Art Conventionally, human interface devices for operating devices such as a pointer and a robot in a non-contact manner according to hand gestures and actions are known.

  For example, in the human interface device described in Patent Document 1, the imaging unit images a finger of a hand at a predetermined time interval. The imaging means includes a plurality of LEDs and a fisheye lens, detects light reflected when the finger is illuminated with the LEDs, and acquires three-dimensional position information of each finger. The calculation of the distance to the finger utilizes the property that the light intensity reflected by the object decreases as the distance between the object and the LED increases (paragraph 0028, FIG. 2).

  In the user interface device described in Patent Document 2, the shape interpretation unit sequentially captures a two-dimensional image of the target object (user's hand) output from the image input unit every predetermined time. The shape interpretation unit extracts a predetermined feature amount (distance information or object region) from the two-dimensional image, and recognizes the shape of the target object based on the interpretation rules (paragraphs 0038 to 0042, FIG. 1).

JP 2011-18129 A Japanese Patent Laid-Open No. 10-207618

  However, in the device of Patent Document 1, the distance of light cannot be obtained accurately even if measured with the same device due to individual differences in light reflectivity due to gender, skin color, etc., and calibration is performed for each person. Had to do.

  Further, in the apparatus of Patent Document 2, it is necessary to prepare various interpretation rules depending on the shape, position, direction, skin color, etc. of the hand, and the control burden and time required for matching processing have become enormous. .

  The present invention has been made in view of such circumstances, and provides an operation device and an operation method capable of recognizing the shape and movement of a hand accurately and at high speed while reducing the processing burden. Objective.

  An operation device according to the present invention includes an imaging unit that captures a human hand in an imaging region and acquires the image information as image information, an image processing unit that recognizes a hand motion based on the image information acquired by the imaging unit, An operation device comprising command processing means for sending an operation command for operating a device in accordance with hand movement, wherein the image processing means extracts a hand area from the image information. A fingertip shape recognizing means for recognizing a plurality of fingertip shapes from the hand area extracted by the hand area extracting means, a specific point in the palm or back of the hand of the hand area, and the fingertip shape recognizing means. And an inter-finger angle calculating unit that calculates an angle between adjacent straight lines connecting specific points in each fingertip shape as the inter-finger angle, and the command processing unit is configured to change the inter-finger angle based on the change in the inter-finger angle. Operation command And command generating means for generating, characterized in that it has a command transmitting means for transmitting the operation command created by the command generating means to said device.

  According to the operating device of the present invention, when a human hand enters the imaging region, the imaging unit acquires this as image information. The hand region extracting unit extracts a hand region from the image information, and the fingertip shape recognizing unit extracts, for example, a portion having a predetermined curvature from the extracted hand region, and recognizes a plurality of fingertip shapes.

  At this time, a straight line connecting a specific point in the palm or back of the hand and a specific point in the fingertip shape in the hand region is generated by the number of fingers (up to five) recognized by the fingertip shape recognition means. Since the inter-finger angle calculating means calculates the angle formed by the adjacent straight lines as the inter-finger angle, it is possible to easily recognize a hand gesture or the like.

  The command creation means creates an operation command for operating a device (for example, a robot or a pointer in a monitor) based on the change of the finger-to-finger angle (the hand moves), and the command transmission means outputs the operation command. Send to the device. Therefore, an operation device that can recognize the movement of the hand and operate the device is realized. Since many conditions are not required for recognizing the hand movement, the hand shape and movement can be recognized accurately and at high speed while reducing the control load of image processing.

  In the operating device of the present invention, the specific point in the palm or back of the hand is preferably the center of gravity of the palm or back of the hand, and the specific point in the fingertip shape is preferably the center of gravity or the tip of the fingertip shape.

  According to the present invention, since the specific point in the palm or back of the hand is the center of gravity of the palm or back of the hand, the center of gravity can be easily calculated by extracting the area. Further, since the specific point in the fingertip shape is the center of gravity of the fingertip shape, for example, the center of gravity coordinates can be easily calculated from the coordinates of the pixels constituting the fingertip shape. The specific point in the fingertip shape may be the tip of the fingertip shape. As a result, a straight line connecting both barycentric coordinates can be obtained, and the inter-finger angle can be easily calculated.

  The operation device of the present invention includes a finger type determining unit that determines a finger type, and the finger type determining unit investigates the inter-finger angle calculated by the inter-finger angle calculating unit and a large number of hand samples. It is preferable to determine the finger type according to the database constructed by this.

  In the present invention, the finger type determining means determines the finger type by applying the calculated inter-finger angle to, for example, a model in the database. As a result, the movement of a specific finger can be tracked to recognize a fine hand movement.

  In the operation device according to the present invention, the command creation means creates the operation command when a thumb and an index finger are determined, and an angle between fingers between the thumb and the index finger is smaller than a predetermined angle threshold. It is preferable to do.

  According to the present invention, when the finger-to-finger angle between the thumb and the index finger becomes smaller than a predetermined angle threshold, for example, an operation command corresponding to a mouse button down is created. An operating device to be operated can be realized.

  An operation method according to the present invention is an operation method in which a person's hand in an imaging region is captured and acquired as image information, the movement of the hand is recognized based on the image information, and the device is operated according to the movement of the hand. A hand region extracting step for extracting a hand region from the image information, a fingertip shape recognition step for recognizing a plurality of fingertip shapes from the hand region extracted in the hand region extracting step, and the hand A finger angle calculation step of calculating an angle between adjacent straight lines connecting a specific point in the palm or the back of the hand in the region and a specific point in each fingertip shape recognized in the fingertip shape recognition step; A command creation step of creating an operation command for operating the device based on a change in the angle between the fingers, and a command transmission step of sending the operation command created in the command creation step to the device. Features .

  According to the operation method of the present invention, when a human hand enters the imaging region, it is first acquired as image information. In the hand region extraction step, a hand region is extracted from the image information. In the fingertip shape recognition step, for example, a plurality of fingertip shapes are recognized by extracting a portion of a predetermined curvature from the extracted hand region. .

  At this time, a straight line connecting a specific point in the palm or back of the hand and a specific point in the fingertip shape in the hand region is generated by the number of fingers (up to 5) recognized in the fingertip shape recognition step. In the inter-finger angle calculating step, the angle formed by the adjacent straight lines is calculated as the inter-finger angle, so that a hand gesture or the like can be easily recognized.

  In the command creation process, an operation command for operating a device (for example, a robot or a pointer in a monitor) is created based on the change of the finger-to-finger angle (hand movement). Transmitted to the device. Therefore, it is possible to recognize the movement of the hand and operate the device. Since many conditions are not required for recognizing the hand movement, the hand shape and movement can be recognized accurately and at high speed while reducing the control load of image processing.

  As described above, according to the present invention, the shape and movement of the hand can be recognized accurately and at high speed while reducing the processing load.

The figure explaining the positional relationship of a person and a camera, and the schematic of the structure of this invention. The flowchart of the whole process. The figure explaining hand area extraction. The figure explaining outline extraction. The flowchart of a finger shape recognition process. The figure explaining calculation of a curvature. The figure explaining fingertip candidate point extraction. The flowchart of a hand shape recognition process. The flowchart of a finger classification determination process. The figure explaining calculation of the angle between fingers. The figure explaining an inter-finger angle model. The figure explaining the final output of pointing information. The figure explaining recognition of a click gesture. The figure explaining the final output of pointing information (modification).

  First, with reference to FIG. 1, the positional relationship between a person and a camera and the overall configuration of the operating device according to the present invention will be described.

  The operating device 1 includes a camera 2 that is an example of an imaging unit, and a computer 3 that executes each process described below. The camera 2 is disposed below the front of the person (operator) H as shown on the left side of FIG. 1A or above the front of the person H as shown on the right side. Regardless of the position of the camera 2, the imageable range is expanded in proportion to the distance from the camera 2, and an object within the range can be imaged.

  The camera 2 is connected to the computer 3 by a cable. Since the image information captured by the camera 2 is transmitted to the computer 3, the image information can be confirmed on the monitor M of the computer 3. Further, the computer 3 can move the pointer (cursor) in the monitor M by the movement of the hand of the person H, for example, by analyzing the image information.

  FIG.1 (b) is a figure when the state of Fig.1 (a) is seen from upper direction. The area S is an area in which the camera 2 can image the hand of the person H (hereinafter referred to as an imaging area or simply an area). In the area S, for example, the person H moves the hand to the right (in the positive x-axis direction). The computer 3 analyzes the image and recognizes that the hand has moved to the right. Thereby, in the above example, the pointer in the monitor M moves to the right. Similarly, when the person H moves his / her hand in the forward direction (y-axis positive direction), the computer 3 recognizes that the hand has moved in that direction, and the pointer in the monitor M moves downward.

  Also, the person H is not always in the same direction with respect to the camera 2, and sometimes moves to change the standing position. Accordingly, the hand of the person H may be inserted from the lateral direction or the oblique direction of the region S. However, the present invention can recognize the shape and movement of the hand regardless of the direction in which the hand is inserted even in such a case.

  Next, as illustrated in FIG. 1C, when the hand of the person H is imaged by the camera 2 in the controller device 1, the image information is transmitted to the image processing unit 31 in the computer 3. In accordance with the processing result in the image processing unit 31, the command processing unit 32 creates an operation command for operating the monitor M as an example of the operation target device.

  Here, the image processing unit 31 corresponds to the “hand region extraction unit”, “fingertip shape recognition unit”, and “inter-finger angle calculation unit” of the present invention, and the command processing unit corresponds to the “command creation unit” of the present invention. Corresponds to “command transmission means”. In FIG. 1 (c), the configuration of the computer 3 is shown as a function realizing unit of the present invention divided into an image processing unit 31 and a command processing unit 32. As will be described in detail later, these processes are performed. The function of this part is executed by the CPU of the computer 3.

  The operation command created by the command processing unit 32 is transmitted to the monitor M, and thereby the pointer in the monitor M can be moved. In addition to the monitor M, the device to be operated according to the present invention may be any device that can be operated by the action of a human hand, such as switches and robots of various devices.

  FIG. 2 is a flowchart of the overall processing of the embodiment. In the following, each process will be described by taking as an example a case where a human hand is inserted into the imaging region S.

  First, hand region extraction processing (“hand region extraction means (process)” of the present invention) is performed (step S10). This is processing for extracting a hand region from an image captured by the camera 2 (“image information” of the present invention). In the embodiment, the hand is imaged by a range image camera. When using the distance image camera, the hand region can be easily extracted by setting the distance between the hand inserted in the imaging region and the distance image camera in a predetermined range.

  FIG. 3A is an image when the hand region 4 is extracted by the hand region extraction process. As shown in the figure, the palm of the image region R is at the center and the wrist is at the bottom of the center. The palm area 4 'and palm coordinates F in the figure will be described later.

  There are several hand region extraction methods other than those described above. For example, when an ordinary camera is used, the hand region can be extracted by extracting information corresponding to the skin color from the RGB information. In addition, when there is no possibility that an object other than the hand appears in the camera, the hand region can be easily extracted by the background subtraction method. In addition, a stereo camera that can calculate the distance to the object by arranging the two lenses so as to generate parallax may be used.

  Returning to FIG. 2, after the hand region extraction process is completed, the process proceeds to the fingertip shape recognition process (“fingertip shape recognition means (step)” of the present invention) (step S20). Hereinafter, a flowchart of the fingertip shape recognition process will be described with reference to FIG.

  First, outline extraction processing is performed (step S201). This is a process for extracting a hand outline from an image including the hand region 4. FIG. 3B is an image when the edge portion of the hand region 4 is detected and the contour line 5 is extracted. Since there are several methods for extracting the contour line in addition to the above method, any method may be used.

  When the contour line extraction process ends, a curvature calculation process is performed (FIG. 4: step S202). The curvature is an angle formed by one point on a contour line that is a curve and two points that are equidistant from the point. Hereinafter, the curvature calculation method will be described with reference to FIG.

When there are a point A and a point C that are equidistant from the point B on the curve, the angle θ _i formed by the straight line AB and the straight line BC is the curvature to be calculated. Several methods of calculating the curvature are conceivable. In the embodiment, the following method is used. Here, the coordinates of each point are assumed to be point A (x _i−1 , y _i−1 ), point B (x _i , y _i ), and point C (x _{i + 1} , y _{i + 1} ).

At this time, assuming that the angle between the straight line AB and the x axis is θ _{i−1 → i} and the angle between the straight line BC and the x axis is θ _{i → i + 1} , θ _i is given by the following equation (1). It is done.

Θ _{i−1 → i} and θ _{i → i + 1} are given by the following equations (2) and (3), respectively.
Thus, the curvature θ _i can be calculated from the coordinates of the three points.

  When the curvature calculation process ends, a fingertip candidate point extraction process is performed (FIG. 4: step S203). This is a process of extracting a part that satisfies the candidate fingertip from the contour 5 of the hand. Hereinafter, the fingertip candidate point extraction process will be described with reference to FIG.

Here, the curvature θ _i of each point constituting the contour line 5 calculated in the curvature calculation process (step S202) is compared with the angle threshold value θ _th, and the one having an angle θ _th or less is extracted. By this processing, fingertip candidate points 6 (1) to 6 (5) are extracted. In addition, since it deletes about the part (dashed line part in a figure) which was not determined with a fingertip candidate point, it showed with the broken line.

  When the fingertip candidate point extraction process ends, a fingertip coordinate calculation process is performed (FIG. 4: step S204). This is a process of calculating fingertip coordinates for a plurality of fingertip candidate points extracted in the fingertip candidate point extraction process (step S203).

Again referring to FIG. 6, an enlarged view of the fingertip candidate point 6 (4) extracted by the fingertip candidate point extraction process is shown in the upper right of the figure. At this time, the fingertip coordinates of the index finger are calculated by obtaining an average value of a plurality of pixels constituting the fingertip candidate point. That is, the fingertip coordinates become the coordinates of the center of gravity of these pixels, and the fingertip coordinates G (x _g4 , y _g4 ) are determined.

  Similarly, for the fingertip candidate points 6 (1) to 6 (3) and 6 (5), the barycentric coordinates are obtained and used as the fingertip coordinates of each fingertip. When the fingertip coordinate calculation process ends, the fingertip shape recognition process also ends.

  Returning to FIG. 2 again, after the fingertip shape recognition process is completed, the process proceeds to the hand shape recognition process (step S30). Hereinafter, a flowchart of the hand shape recognition process will be described with reference to FIG.

  First, an object distance measurement process is performed (step S301). Here, the distance between the target hand and the camera is measured. In the embodiment, the distance difference between the infrared rays output from the light emitting element of the distance image camera (TOF method) is measured by converting the phase difference until it is reflected by the hand and returns to the camera. For distance measurement, a pattern irradiation type distance image camera may be used.

  When the object distance measurement processing is completed, palm coordinate calculation processing is performed (FIG. 7: Step S302). This is a process of calculating palm coordinates from the image of the hand area 4 extracted in the above-described hand area extraction process (FIG. 2: step S10).

  There are several methods for extracting the palm region from the image of the hand region 4, but in the embodiment, a method based on the Opening process is used. The opening process is a process for removing fine patterns and small patterns by repeatedly performing the erosion process (shrinkage process) and the dilation process (expansion process) for the same number of images.

  Referring to FIG. 3A again, when the opening process is performed on the image region R, the finger and wrist patterns are removed from the hand region 4 and only the palm region 4 ′ remains. Further, the center of gravity of the palm region is calculated and used as palm coordinates F. Of course, when the extracted hand region 4 is on the back of the hand, the back of the hand coordinates are calculated instead of the palm coordinates.

  When the palm coordinate calculation process ends, a finger type determination process is performed (FIG. 7: Step S303). This is a process of labeling to which finger the fingertip shape obtained by the fingertip shape recognition process (FIG. 2: step S20) corresponds. Hereinafter, the flowchart of the finger type determination process will be described with reference to FIG.

  First, it is determined whether or not R_Flag is 0 (step S311). “R_Flag” is a flag indicating completion of fingertip labeling. If R_Flag is “0”, the process proceeds to step S312. If R_Flag is “1”, the process proceeds to step S320.

  First, the case where the determination in step S311 is “YES” will be described. In this case, it is determined whether or not the number of detected fingertips is 5 (step S312). Basically, the number of detected fingertips matches the number of fingertip coordinates calculated in the fingertip coordinate calculation process (FIG. 4: step S204). If the number of detected fingertips is “5”, the process proceeds to step S313. If the number of detected fingertips is not “5”, the process proceeds to step S318.

  When two fingertips are detected, the thumb and index finger used for a click gesture, which will be described later, may be determined, so the condition of “number of detected fingertips” may be changed to 2 or more.

  If the determination in step S312 is “YES”, an inter-finger angle calculation process (“inter-finger angle calculating means (step)” of the present invention) is performed (step S313). The inter-finger angle is an angle formed by an adjacent straight line connecting the fingertip coordinate and the palm coordinate. Here, since five fingers are detected, four inter-finger angles are calculated. Hereinafter, calculation of the inter-finger angle will be described with reference to FIG.

As shown in the figure, the palm coordinates calculated in the palm coordinate calculation process (FIG. 7: step S302) are assumed to be F (x _f , y _f ). Further, labels (1) to (5) are attached in order from the fingertip coordinates having the smallest x coordinate (initial labeling), and the first fingertip coordinates G (x _g1 , y _g1 ) to the fifth fingertip coordinates G (x _g5 , ( _yg5 ) is determined (part of the coordinate names in the figure are omitted). At this stage, the outline 5 is all deleted and is shown by a broken line.

In addition, since a vector from the palm coordinate F to each fingertip coordinate is generated, a vector from the palm coordinate F to the first fingertip coordinate is set as a vector x ₁ . Similarly, a vector x ₅ from the palm coordinate F to the fifth fingertip coordinate is determined. At this time, the angle between the vector x ₁ and the vector x _{2 is} the inter-finger angle φ _1-2, and similarly, the angle between the vector x ₄ and the vector x _{5 is} the inter-finger angle φ _4-5 (the angle in the figure). Are omitted).

In the embodiment, in particular, since the fact that the thumb is closed in the direction of the index finger is recognized as a mouse button-down gesture, the inter-finger angle φ _4-5 must be detected. At this time, the inter-finger angle φ _4-5 is given by the following equation (4).
Thus, the inter-finger angle φ _4-5 can be calculated from two vectors.

  When the inter-finger angle calculation process ends, a finger type determination process (“finger type determination unit” of the present invention) is performed (FIG. 8: Step S314). An inter-finger angle model is used to determine the finger type. The inter-finger angle model is the distance between the camera and the hand obtained from the image of the hand of a different person (approximately 200 images) taken at a certain distance from the camera and the distance between the camera and the hand. (See FIG. 10).

  As a procedure for determining the finger type, first, when the camera picks up a hand, an inter-finger angle model corresponding to the distance between the camera and the hand is cut out (for example, ☆ in the figure). Next, the finger angle calculated from the acquired image is applied to the finger angle model, and the finger type is determined from the closest angle information.

According to this inter-finger angle model, the inter-finger angle φ _4-5 between the thumb and forefinger is the largest (the average inter-finger angle in the effective distance range is about 40 degrees) and can be easily distinguished from other inter-finger angles. Therefore, it is optimal as a combination of fingers used for recognizing a gesture or the like. Note that σ in the figure indicates a standard deviation, and means that there is a variation of ± 5 degrees in the case of an angle φ _4-5 .

  The inter-finger angle is accurately calculated even when the hand is inserted from the lateral direction or the oblique direction with respect to the imaging region S, and at least, there is a low possibility that the thumb and the index finger are erroneously labeled. This completes the final labeling of the fingertip.

  Next, it is determined whether or not the thumb and index finger have been determined (FIG. 8: step S315). If the thumb and index finger are determined in the above processing, the process proceeds to step S316. On the other hand, for example, if the thumb or index finger cannot be determined because there is almost no inter-finger angle, the process proceeds to step S319.

  If the determination in step S315 is “YES”, R_Flag is set to 1 (step S316). R_Flag being “1” means that fingertip labeling has been completed. Thereafter, the process proceeds to step S317.

  In step S317, it is determined whether the thumb and index finger are being tracked. This is to track the movement of the thumb and index finger recognized by the image processing unit, but immediately after the R_Flag is set to “1”, the tracking is not in progress, so the process returns to step S311. On the other hand, if tracking is in progress, the process proceeds to step S323.

  Here, a case where the determination in step S312 is “NO” will be described. In this case, it is determined whether or not the number of detected fingertips is greater than 1 (step S318). If the number of detected fingertips is greater than “1”, the process proceeds to step S319. If the number of detected fingertips is “1” or less, the finger type determination process ends. When the number of detected fingertips is “1” or less, the finger-to-finger angle determination process is terminated because the inter-finger angle cannot be calculated.

  Next, when the determination in step S318 is “YES” or when the determination in step S315 is “NO”, a warning process is performed (step S319). This is, for example, a process of warning that the hand is opened by a display on the monitor or by voice when there is almost no inter-finger angle and it does not apply to any of the inter-finger angle models described above. After the warning process is performed, the fingertip type determination process is terminated.

  Next, the case where the determination in step S311 is “NO” will be described. When R_Flag is set to “1”, the process proceeds to step S320, and it is determined whether or not the number of detected fingertips is not five. This is a check process for checking whether the number of detected fingertips has changed from “5” due to an error or the like.

  If R_Flag is “1” and the number of detected fingertips is other than “5”, it is necessary to perform relabeling, and the process proceeds to step S321. On the other hand, if the number of detected fingertips is “5”, the process proceeds to step S322.

  If the determination in step S320 is “YES”, R_Flag is set to 0 (step S321). Thereafter, since the finger type determination process is ended, fingertip labeling is performed in the next finger type determination process.

  On the other hand, if the determination in step S320 is “NO”, tracking of the thumb and index finger is started (step S322). Thereafter, the process proceeds to step S317, but since the determination is “YES” this time, the process proceeds to step S323.

  Finally, R_Flag is set to 0 (step S323). This is a process of resetting R_Flag, and fingertip labeling is first performed in the next finger type determination process. Thereafter, the finger type determination process is terminated, and the hand shape recognition process is also terminated (see FIG. 7).

  Returning to FIG. 2 again, after the hand shape recognition process is completed, the process proceeds to the pointing information calculation process ("command creation means (process)" of the present invention) (step S40). Although the details of the pointing information calculation process will be described later, it is a process of creating an operation command for operating, for example, a pointer in the monitor by performing coordinate conversion.

  After the pointing information calculation process is completed, the process proceeds to a command transmission process (“command transmission means (step)” of the present invention) (step S50). This is a process of transmitting the operation command obtained by the series of image processing to the output device. As described above, a non-contact type operating device that recognizes the shape and movement of the hand and operates the device can be realized.

  Hereinafter, with reference to FIGS. 11 and 12, the final output of pointing information, which will be described later, will be described.

  First, FIG. 11A is a view of a state in which a human hand 7 is inserted into the imaging region S as viewed from above. Since the image processing unit tracks the moving direction of the index finger P captured by the camera 2, for example, the image processing is performed when the hand 7 moves in the positive x-axis direction, and the image processing unit performs on the monitor (hereinafter, the monitor region M ′ or the region M). '), The pointer P' moves in the positive direction of the X axis (see FIG. 11C). Here, the right direction of the region M ′ is the X axis positive direction.

  FIG. 11B is a view of the state in which the human hand 7 is inserted into the imaging region S as viewed from the lateral direction. Since the camera 2 can measure the distance L between the hand 7 and the camera 2, for example, when the hand 7 moves in the positive z-axis direction, image processing is performed by the image processing unit. The pointer moves in the direction (see FIG. 11C). Here, the upward direction of the region M ′ is defined as the positive Y-axis direction.

  Since the image processing unit tracks the movement of the thumb in addition to the index finger, it is possible to recognize a gesture with these two fingers. In the following, the recognition of the click gesture will be described in particular.

FIG. 12A is a diagram of a state in which a human hand 7 is inserted into the imaging region S as viewed from above. As shown in the figure, a vector x ₄ directed from the palm coordinates F (x _f , y _f ) to the fingertip coordinates G (x _g4 , y _g4 ) of the index finger and a vector directed to the fingertip coordinates G (x _g5 , y _g5 ) of the thumb x ₅ is shown, it forms finger angle between the vectors x ₄ and the vector x ₅ is the angle phi _4-5.

Here, when the inter-finger angle φ _4-5 decreases and falls below the angle threshold φ _th , it is recognized that the button has been pressed down by the image processing unit (see FIG. 12B). Thereafter, when the inter-finger angle φ _4-5 increases and exceeds the angle threshold φ _th , it is recognized that the button has been pushed up by the image processing unit. This is an operation corresponding to one click of the mouse.

  As described above, for example, the person can move the pointer over the icon in the region M ′. In this state, by performing the operation of closing and opening the thumb in the direction of the forefinger twice, it is recognized that the icon has been double-clicked, so that the application can be activated.

  Finally, a modification of pointing information output will be described with reference to FIG. In the example shown in FIG. 11, the movement of the hand 7 in the horizontal direction (x-axis direction) and the height direction (z direction) are converted into the movement of the pointer in the monitor area M ′. An operation in the (y-axis direction) can be added (see FIG. 13A).

  Thereby, the pointer of the region M ′ can be moved three-dimensionally. When three-dimensional movement is not necessary, it may be used for other operations in the region M ′. For example, it can be used for operations for enlarging / reducing the screen, rotating figures and solids, and switching to the next slide or image.

  The above-described embodiment is an example of the present invention, and various modifications can be considered. Since various methods are known for extracting a hand region from a hand image, extracting a contour line from a hand region, and calculating a curvature, any method may be used. Also, the fingertip coordinates can be the tip coordinates of the fingertips instead of the barycentric coordinates, or the wrist coordinates instead of the palm coordinates.

  In the above embodiment, the motion of the thumb and forefinger is recognized and the pointer in the monitor area is moved. However, the motion of another finger such as the motion of the index finger and the middle finger may be recognized. Further, the “finger movement” includes an operation of bending or tapping each finger.

  In the finger type determination process (FIG. 8), it can be changed to “number of detected fingertips ≧ 2?” In step S312. Thereby, when at least two fingertip shapes are detected, the process proceeds to the inter-finger angle calculation process (step S313). When the thumb and index finger are determined (step S315 / YES), the click gesture is recognized. It becomes possible. In this case, step S318 is not necessary, and if the determination “number of detected fingertips ≧ 2?” Is “NO”, the process proceeds to step S319.

1 Operating device 2 Camera (imaging means)
3 Computer 4 Hand region 4 'Palm region 5 Contour lines 6 (1) to 6 (5) Fingertip candidate points 7 Hand region 31 Image processing unit (image processing means)
32 Command processing part (command processing means)
H person M monitor M 'monitor area P fingertip P' pointer R image area S imaging area

Claims (5)

Imaging means for imaging a human hand in an imaging area and acquiring it as image information, image processing means for recognizing the movement of the hand based on the image information acquired by the imaging means, and equipment corresponding to the movement of the hand An operation device comprising command processing means for sending an operation command for operating
The image processing means includes
Hand region extraction means for extracting a hand region from the image information;
Fingertip shape recognition means for recognizing a plurality of fingertip shapes from the hand area extracted by the hand area extraction means;
Inter-finger angle calculation that calculates an angle between adjacent straight lines connecting a specific point in the palm or back of the hand of the hand region and a specific point in each fingertip shape recognized by the fingertip shape recognition means as an inter-finger angle. Means,
The command processing means includes
Command creation means for creating the operation command based on a change in the inter-finger angle;
An operation device comprising command transmission means for transmitting the operation command created by the command creation means to the device. The operating device according to claim 1,
The operating device characterized in that the specific point in the palm or back of the hand is the center of gravity of the palm or back of the hand, and the specific point in the fingertip shape is the center of gravity or the tip of the fingertip shape. The operating device according to claim 1 or 2,
A finger type determining means for determining a finger type;
The finger type determining means determines the finger type from the inter-finger angle calculated by the inter-finger angle calculating means and a database constructed by examining a large number of hand samples. apparatus. The operating device according to claim 3,
The command creating means creates the operation command when the thumb and the index finger are determined, and the angle between the fingers formed by the thumb and the index finger is smaller than a predetermined angle threshold value. apparatus. An operation method for capturing an image of a person's hand in an imaging region and acquiring it as image information, recognizing the movement of the hand based on the image information, and operating the device according to the movement of the hand
A hand region extraction step of extracting a hand region from the image information;
A fingertip shape recognition step for recognizing a plurality of fingertip shapes from the hand region extracted in the hand region extraction step;
Inter-finger angle calculation that calculates an angle between adjacent straight lines connecting a specific point in the palm or back of the hand of the hand region and a specific point in each fingertip shape recognized in the fingertip shape recognition step as a finger-to-finger angle Process,
A command creating step for creating an operation command for operating the device based on a change in the inter-finger angle;
A command transmission step of transmitting the operation command created in the command creation step to the device.