JP2014154049A - Fingertip detection method and fingertip detection user interface module - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily and accurately detect the top end of the finger.SOLUTION: Image data are written via a memory I/F 11 or a camera I/F 12 in a frame buffer 13. A skin color extraction/binarization processing part 14 reads the image data from the frame buffer 13, and extracts only the skin color portion, and rewrites data obtained by binarizing the data of the extracted portion in the frame buffer 13. A fingertip neighborhood data extraction processing part 15 reads the binary data of the skin color portion stored in the frame buffer 13, and extracts pixel data in the neighborhood of the fingertip by predetermined filter processing. A finger-categorized processing part 16 processes a plurality of pixel data in the neighborhood of the fingertip, and classifies the pixel data as data for each finger. A finger direction calculation part 17 calculates the direction of each finger and the position of the top end of the finger on the basis of the obtained data of each finger. Information acquired by the finger direction calculation part 17 is transmitted via an external I/F 18 to an external CPU.

Description

  The present invention relates to a fingertip detection method and a fingertip detection user interface module, and in particular, an information device user forms at least one finger in a specific form so as to give specific information to the information device. Included in the fingertip detection method for detecting the fingertip of the user based on the image information obtained by imaging the specific form by the imaging means that is presented to the device and the information device for realizing the method The present invention relates to a fingertip detection user interface module.

  With the advance of technology, the way of supplying control information such as user operations / instructions to various information devices, that is, the user interface (hereinafter referred to as UI) has been highly advanced. For example, like a so-called smart phone that is rapidly spreading recently, a UI by a user touching an information display unit of an information device with a fingertip is no longer used as common sense. Furthermore, in the UI, by making it possible to identify the tracing mode as one mode of touching, it is possible to supply a variety of control information. Further, with the development of hardware elements, it has become possible to identify control information further developed. For example, Patent Document 1 discloses a technique that can recognize the direction of the touched fingertip based on distance inclination information from the proximity sensor by employing a proximity sensor in addition to the touch sensor.

  In addition, as another UI, the information device is provided with imaging means such as a camera (hereinafter referred to as “CMOS camera”) employing a CMOS image sensor, and images the whole or part of the user's body as necessary. However, UIs that recognize control information such as operations / instructions intended by the user from the image information obtained thereby are also in the development stage.

JP 2012-194692 A

  Here, in the UI using the imaging means as described above, the most typical method is that the user expresses his / her upper limbs, especially operations / instructions by hand, and the imaging means on the information equipment side captures the images. Then, the information device obtains the operation / instruction intended by the user by performing the discrimination processing on the image data. At this time, it is easy for the user to express the operation / instructions by the number of fingers and the direction of the finger. Examples of a user exchanging control information with an information device using fingers of a hand include various cases such as specifying a numerical value by the number of fingers or specifying selection information by the direction of one finger. is there. Further, in a gaming machine or the like, for example, there may be a case where a player and a gaming machine play a game. However, on the information equipment side, it is often relatively difficult to determine the fingertip portion from the captured image data. Furthermore, there are cases where the user gives control information by “movement”, for example, by moving the index finger up and down and left and right with respect to the imaging means (so-called drag operation is intended). In the ready state, the motion must be further detected.

  The present invention has been made for the above-mentioned circumstances, and an object of the present invention is to obtain an information device having an imaging means and imaging the whole or a part of the user's body as necessary. An object of the present invention is to provide a fingertip detection method and a fingertip detection user interface module that can easily and accurately detect the tip of a finger when recognizing control information such as an operation / instruction intended by a user from the image information.

  In order to achieve the above object, the fingertip detection method according to the present invention is characterized in that a user of an information device forms at least one finger in a specific form so as to give specific information to the information device. A fingertip detection method in which the information device detects the fingertip of the user based on image information obtained by imaging the specific form by an imaging means, the sharpness of the fingertip A fan having a central angle corresponding to the filter is used as a filter, and the center of a circle including the fan is made to correspond to each pixel of the image information, and the fan is rotated while rotating the fan, The process of obtaining the change in the overlap area is performed for all the pixels by moving the filter to each pixel, and a pixel point obtained by setting the angle range where the overlap area substantially matches the fan area to be equal to or less than a predetermined threshold is set near the fingertip. Pixel And summarized in that the detected.

  Furthermore, a plurality of pixel points in the vicinity of the detected fingertip are classified as pixel points for each finger.

  Here, whether or not two neighboring pixels are data of the same finger is based on whether or not the peak angle range corresponding to the distance between the two has a predetermined difference and the angle range. It is a feature that it is discriminated based on whether or not the central value has a predetermined difference.

  Further, for each finger, it is preferable to obtain the position of the tip of the finger from the positions of a predetermined number of pixels in the vicinity of the fingertip by interpolation.

  For each finger, the direction of the finger is preferably obtained by taking a vector sum of the direction of the central value angle of the angle range for a plurality of pixels near the fingertip.

  Moreover, it is preferable that the radius of the fan is determined according to a distance from the finger to the imaging unit.

  Further, the overlapping area between the flesh-colored portion of the image information and the fan is obtained while rotating the fan at every predetermined feed angle.

  Here, it is preferable that the predetermined feed angle is determined according to the definition of the image information with respect to the radius of the fan.

  In order to achieve the above object, the fingertip detection user interface module of the present invention forms at least one finger in a specific form so that a user of an information device gives specific control information to the information device. Included in the information device, wherein the fingertip of the user is detected based on image information obtained by imaging the specific form by an imaging unit provided in the information device. A fingertip detection user interface module, wherein a fan having a central angle corresponding to the sharpness of the fingertip is used as a filter, and the fan is detected in a state where the center of a circle including the fan is associated with each pixel of the image information. Rotating the filter, the process of obtaining the change in the overlapping area between the flesh-colored portion of the image information and the fan is performed for all the pixels by moving the filter to each pixel. And summarized in that the angular range in which the overlapping area is substantially the same as the area of the fan is provided with a fingertip proximate data extraction processing unit for detecting the pixel points obtained as a predetermined threshold value or less as a pixel of a fingertip vicinity.

  According to the present invention, the information device includes an imaging unit, images the whole or a part of the user's body as necessary, and the operation / instruction intended by the user from the image information obtained thereby. When recognizing the control information, the tip of the finger can be detected easily and accurately. Furthermore, the direction of the finger can be detected easily and accurately. Also, a plurality of fingers can be detected separately.

It is a schematic block diagram of one embodiment of the fingertip detection user interface module of the present invention. It is a flowchart of one Embodiment in the fingertip detection method of this invention. It is a figure for demonstrating a fingertip detection filter and the filter process by it. It is a figure which shows the example for demonstrating the overlapping area according to the rotation angle of the fan of a fingertip detection filter, and the skin color part. It is a figure which shows the example for demonstrating the overlapping area according to the rotation angle of the fan of a fingertip detection filter, and the skin color part. It is a figure which shows the example for demonstrating the overlapping area according to the rotation angle of the fan of a fingertip detection filter, and the skin color part. It is a figure which shows the example for demonstrating the overlapping area according to the rotation angle of the fan of a fingertip detection filter, and the skin color part. It is a figure which shows the pixel point extracted as a fingertip vicinity. It is a figure which shows the example which designed the fingertip detection filter containing a fan as a collection of pixels. It is a figure which shows the example of the relationship of the number of overlapping pixels with respect to the rotation angle of a fan when a feed angle is 15 degrees and the number of pixels of a fan is 33. It is a figure for demonstrating the viewpoint for classifying each finger | toe. It is a figure which shows the fingertip vicinity data which could be classified as each finger data. It is a figure for demonstrating the process when calculating | requiring the direction of a finger | toe.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a schematic block diagram of an embodiment of a fingertip detection user interface (hereinafter referred to as “UI”) module of the present invention. The fingertip detection UI module 1 shown in the figure detects the fingertip portion from the image of the person imaged by a CMOS camera or the like, and recognizes the number of fingers and the direction of the finger, thereby presenting information presented by the person's hand. It is a UI module that makes it possible to recognize. In the following description, the description is based on recognizing the above-mentioned information from the information of one still image. However, in view of the fact that still images are temporally linked, moving images are configured. It will be obvious to those skilled in the art that if the process is performed for each still image to be performed, the movement of the hand and its finger can be recognized for the moving image.

  Therefore, the fingertip detection UI module 1 shown in FIG. 1 includes a memory interface (hereinafter referred to as “I / F”) 11, a camera I / F 12, a frame buffer 13, and a skin color detection / binarization processing unit 14. A fingertip vicinity data extraction processing unit 15, each finger classification processing unit 16, each finger direction calculation unit 17, and an external I / F (for example, an I2C (Inter-Integrated Circuit) I / F) 18. Yes.

  In FIG. 1, the image data stored in the image memory in the device is written to the frame buffer 13 for each frame via the memory I / F 11, and the image data captured by the CMOS camera is also the camera I / F. Writing to the frame buffer 13 is possible via F12. That is, any image data can be processed. The skin color extraction / binarization processing unit 14 reads out the image data from the frame buffer 13, extracts only the skin color portion from the image data, and further binarizes the extracted portion of the data to the frame buffer 13. Write back to The fingertip vicinity data extraction processing unit 15 reads the binarized data of the skin color portion stored in the frame buffer 13 in a predetermined unit, and extracts pixel data in the vicinity of the fingertip by a filtering process described later. Each finger classification processing unit 16 processes a plurality of pixel data in the vicinity of the fingertip obtained by the fingertip vicinity data extraction processing unit 15 and classifies the data as data for each finger. Each finger direction calculation unit 17 calculates the direction of each finger and the position of the tip of the finger based on the data for each finger obtained by each finger classification processing unit 16. Information obtained by each finger direction calculation unit 17 is sent to the external CPU via the external I / F 18.

  Next, a method for detecting a fingertip by the fingertip detection UI module 1 will be described in detail with reference to FIGS. FIG. 2 is a flowchart of an embodiment of the fingertip detection method of the present invention. 3 to 13 are diagrams for specifically explaining a fingertip detection method.

  In FIG. 2, first, image data stored in an image memory or data related to an image captured by a CMOS camera is input for each frame via the memory I / F 11 or the camera I / F 12, and the frame buffer 13 (Step S11). Next, the skin color extraction / binarization processing unit 14 reads out the image data from the frame buffer 13, extracts only the skin color portion from the image data, and further binarizes the data of the extracted portion. Write back to the frame buffer 13 (step S12). In order to detect the fingertip of the hand, first, since it is necessary to extract a skin color portion (including a face and the like) including the fingertip from the entire image, this processing is performed as preprocessing. This specific method is obvious to those skilled in the art.

  Next, the fingertip vicinity data extraction processing unit 15 reads out the binarized data of the skin color portion stored in the frame buffer 13, and extracts pixel data in the vicinity of the fingertip by the filter process (step S13). The details of this processing will be described below with reference to FIGS. First, in this fingertip vicinity pixel data extraction process, taking into account the generally sharp shape of the fingertip, a two-dimensional filter including a fan as shown in FIG. 3A (hereinafter referred to as “fingertip detection filter FF”). ). Then, for each pixel point of the binarized data, the pixel point and the center of a circle including a part of the pixel point are matched, and in this state, the fingertip detection filter FF including the value 1 fan is rotated. On the other hand, by taking the logical product of the fingertip detection filter FF and the binarized data representing the skin color portion, the overlapping area (number of overlapping pixels) of the fan and the skin color portion at each rotation position of the fan at each pixel point ) As shown in FIG. 3B, this is sequentially performed while shifting the fingertip detection filter FF to each pixel point arranged two-dimensionally.

  With reference to FIG. 4 thru | or FIG. 7, about the overlap area according to the rotation angle of the fan of a fingertip detection filter FF and a skin color part, the typical positional relationship of the fingertip detection filter FF and a skin color part is given and demonstrated. To do. FIG. 4 shows a case where the positional relationship between the two is such that no overlap with the flesh-colored portion occurs even when the fan is rotated. In this case, as shown in FIG. 5B, there is no overlapping area over one rotation (360 degrees). On the other hand, FIG. 5 shows the case of the positional relationship between the two so that the whole overlaps with the skin color portion regardless of the rotation of the fan. In this case, the overlapping area is 0 to 360 degrees (one rotation), which is the fan area. Next, FIG. 6 shows the case where the center of the circle including the fan is at the boundary between the skin color and the part that is not (the theoretical explanation assumes that the pixel point is on the boundary). In this case, although it depends on the central angle θ of the fan (FIG. 3A), when the central angle θ is approximately 90 degrees, as shown in the schematic diagram of FIG. The apparent shape over the angle becomes a trapezoidal shape in which the overlapping area is the area of the fan within a predetermined angle range (theoretically, the central angle θ = 90 degrees and 90 degrees). FIG. 7 shows a case where the center of the circle including the fan is at the boundary of the tip of the finger. In this case, as shown in the schematic diagram of FIG. 5B, the apparent shape over the rotation angle of the overlap area is a narrow range where the overlap area is less than a predetermined value and the overlap area is a fan. The shape is such that the angle range of the area is very small. Therefore, in other words, if the apparent shape over the rotation angle of the overlapping area appears as a shape as shown in FIG. 7B, the pixel point is a pixel point near the tip of the finger. It can be judged.

  As a condition, it is necessary that the central angle θ of the fan substantially matches the sharpness of the fingertip. When the fan central angle θ is smaller than the fingertip sharpness, there will be some angle range where the overlapping area becomes the fan area, while the fan central angle θ is larger than the fingertip sharpness In this case, the overlapping area does not reach the fan area. In the case of FIG. 6, for example, if the center angle θ of the fan is set to 180 degrees, the apparent shape over the rotation angle of the overlapping area is a large triangle having a vertex (the overlapping area is generated over all angles). turn into.

  From the above, as a whole, as a criterion for judgment, under the precondition that the central angle θ of the fan substantially matches the sharpness of the fingertip, “the angle range where the overlapping area substantially matches the fan area is a predetermined threshold value. It is only necessary to adopt the standard of “being below”. In that case, the area of the apparent shape indicated by the overlapping area is also a predetermined value or less. Therefore, when the threshold value is set appropriately and determined based on the reference, pixel points as shown in FIG. 8 can be extracted as pixels near the fingertip. In FIG. 8, the extracted pixel points are schematically arranged on the boundary line. However, since each pixel point is actually a point on the two-dimensional lattice, pixel points on the lattice near the boundary are obtained. It will be. From the above, it is preferable to adopt the sharpness of a standard human fingertip as the central angle θ of the fan.

  Note that the fan of the fingertip detection filter FF described in step S13 described above is theoretical, and when actually configured with a hardware circuit, it is configured to process as discrete data according to image definition. There is a need. FIG. 9 is a diagram showing an example thereof. Here, as described above, it is optimal that the central angle θ of the fan coincides with the sharpness of the fingertip. Therefore, the optimum angle may be set in advance with the sharpness of the standard fingertip in mind. In FIG. 9, it is 90 degrees for convenience of illustration. Further, the radius r of the fan (see FIG. 3A) is determined in terms of how many pixels the size of the user's finger to be imaged corresponds to in the usage situation assumed by the user. That's fine. That is, if the definition of the image and the distance from the camera to the finger are determined, how much the finger occupies on the pixel data is determined. 9A to 9C show the discrete shapes of the fans when the fans are rotated by 45 degrees. In this case, considering the rotation of the fan, the fingertip detection filter FF can be composed of 15 × 15 pixels. In addition, the process may be performed by rotating the fan in units (hereinafter, this angle unit is referred to as “feed angle”), depending on the definition of the image occupied by the fan. The finer the image relative to the fan, the finer the feed angle. Note that the number of pixels corresponding to the shape of the fan is preferably constant regardless of the rotational position of the fan. What is necessary is just to determine beforehand according to an angle (for example, 0 degree, 15 degrees, 30 degrees, 45 degrees, ...). In the case shown in FIG. 9, there are 33 pixels.

  FIG. 10 is a diagram illustrating an example of the relationship between the number of overlapping pixels and the rotation angle of the fan when the feed angle is 15 degrees and the number of pixels of the fan is 33. Here, FIG. 10A corresponds to FIG. FIG. 10B corresponds to FIG. 7, and when the number of overlapping pixels is detected as shown in FIG. 10, pixels near the fingertip can be determined from the viewpoint of the above threshold. A person skilled in the art can easily understand that if the rotation center of the fan (the pixel to be processed) is an edge pixel included in the finger image, an overlap pixel of one pixel appears over all angles. It will be possible.

  Regarding the reading of the binarized data from the frame buffer 13 in the fingertip vicinity data extraction processing unit 15, if it is a fingertip detection filter FF of 15 × 15 pixels as described above, it is preferably read in units of 15 lines. For each 15 lines, the fingertip detection filter FF of 15 × 15 pixels may be processed while being sequentially shifted in the line direction.

  The pixel point extracted as the vicinity of the fingertip obtained in step S13 described above may be a pixel point for only one finger depending on the presentation mode of the user (user), and FIG. As shown, there may be pixel points for multiple fingers. Accordingly, as the next process, each finger classification processing unit 16 classifies each of the plurality of pixel points near the fingertip obtained in step S13 as data for each finger (step S14). This process is equivalent to recognizing the number of fingers. However, in the case of typical finger relationships as shown in FIG. 8, separation is easy, but depending on the form of each finger revealed by the user, each finger approaches and separates the data of each finger. May not be easy. Thus, from the viewpoint as shown in FIG. 11, if each pixel extracted as the vicinity of the fingertip is scrutinized, it can be classified for each finger. Specifically, FIG. 11 is a diagram showing the relationship between the rotation angle of the fan and the overlapping area when a filter process is performed on pixels near the boundary in one finger. Five points are shown. As shown in the figure, the following relationship can be found for only one finger. That is, first, at the boundary of the tip of the finger, as described above, the apparent shape over the rotation angle of the overlapping area is a small area shape having a peak in a narrow angle range, whereas the pixel point is As the lateral direction of each finger increases, the peak angular range gradually increases and the center value of the angular range gradually shifts to the right or left. Therefore, in the case of data of one finger, the relationship between the apparent shape over the rotation angle of the overlapping area for a certain boundary pixel and the apparent shape over the rotation angle of the overlapping area for other boundary pixels is: It depends on the distance between the two pixels. From such a viewpoint, whether or not two neighboring pixels are the same finger data is based on whether or not both have a predetermined difference in the peak angle range corresponding to the distance between the two, and It can be determined whether the central value of the angle range has a predetermined difference. FIG. 12 shows the finger [1] data and the finger [2] data thus sorted.

  Returning to FIG. 2, next, for each finger sorted as in step S14, each finger direction calculation unit 17 obtains the tip of each finger and the direction of each finger (step S15). As for the tip of the finger, in the simplest case, the position of the pixel having the narrowest angular range in which the overlapping area is equal to the fan area, that is, the overlapping area is the smallest, among the data near each fingertip of the finger may be adopted. . Or you may obtain | require by interpolation from the position of several pixels among the pixels of each fingertip vicinity of the said finger | toe. As for the direction of the finger, the simplest is that of the data near each fingertip of the finger, the overlapping area is equal to the fan area, i.e., the data having the narrowest angular range with the largest overlapping area, The center value may be adopted as the angle in the finger direction. However, it is a theoretical story. Actually, the finger shape is not uniform and distorted, and discrete data is handled. Therefore, as shown in FIG. If the vector sum in the direction of the central value angle of each of the above angle ranges is taken, variations can be absorbed.

  Returning to FIG. 2, finally, for all the fingers, it is determined whether or not the finger tips and finger directions have been obtained (step S16). If unprocessed, the process returns to step S15 to continue processing. When the process is completed for all fingers, the entire process is terminated.

  As described above, according to the above-described embodiment, the user in the form of the hand portion including the finger is obtained by imaging at least the hand portion of the user of the information device using the CMOS camera attached to the information device. When the information device determines the control intention, the information device can appropriately and accurately detect the position of the tip of one or more fingers and the direction of each finger.

  In the above-described embodiment, the UI module 1 supplies information on the position of the tip of one or more fingers and the direction of each finger to the external CPU. Without providing the above, and without obtaining the position of the tip of the finger as a specific position, the output information of each finger classification processing unit 16 is supplied with a plurality of fingertip neighborhood data sorted for each finger The calculation of the position of the tip of each finger and the direction of each finger may be left to an external CPU. Furthermore, the information from the fingertip vicinity data extraction processing unit 15 may be directly supplied to the external CPU without separating each finger.

  The fingertip detection method and fingertip detection user interface module of the present invention are used for non-contact operations such as information terminals and portable terminals, remote control operations such as televisions and air conditioners, and for game machines such as game machines and pachinko machines. Adopted.

DESCRIPTION OF SYMBOLS 1 User interface module 11 Memory interface 12 Camera interface 13 Frame buffer 14 Skin color extraction / binarization process part 15 Fingertip vicinity data extraction process part 16 Each finger classification process part 17 Each finger direction calculation part 18 External interface

Claims (9)

When the user of the information device forms at least one finger in a specific form and presents it to the information device in order to give specific control information to the information device, the information device uses the imaging means to A fingertip detection method for detecting a fingertip of the user based on image information obtained by imaging a specific form,
Using a fan having a central angle corresponding to the sharpness of the fingertip as a filter, the flesh-colored portion of the image information while rotating the fan with the center of the circle including the fan corresponding to each pixel of the image information The process of obtaining the change in the overlapping area between the fan and the fan is performed for all the pixels by shifting the filter to each pixel,
A fingertip detection method, wherein a pixel point obtained by setting an angle range in which the overlapping area substantially coincides with the fan area is equal to or less than a predetermined threshold is detected as a pixel in the vicinity of the fingertip.   The fingertip detection method according to claim 1, wherein a plurality of detected pixel points in the vicinity of the fingertip are classified as pixel points for each finger.   Whether or not the two pixels near the boundary are data of the same finger is based on whether or not the peak angle range corresponding to the distance between the two has a predetermined difference and the center value of the angle range The fingertip detection method according to claim 2, wherein it is determined based on whether or not has a predetermined difference.   4. The fingertip detection method according to claim 3, wherein, for each finger, the position of the tip of the finger is obtained by interpolation from the positions of a predetermined number of pixels in the vicinity of the fingertip.   The fingertip detection method according to claim 3, wherein the finger direction is obtained for each finger by taking a vector sum of the direction of the central value angle of the angle range for a plurality of pixels near the fingertip.   The fingertip detection method according to claim 1, wherein the radius of the fan is determined according to a distance from the finger to the imaging unit.   The fingertip detection method according to claim 1, wherein an overlapping area between a flesh-colored portion of the image information and the fan is obtained while rotating the fan for each predetermined feed angle.   The fingertip detection method according to claim 1, wherein the predetermined feed angle is determined according to a definition of the image information with respect to the radius of the fan. An imaging device provided in the information device when a user of the information device forms and presents at least one finger in a specific form to give specific control information to the information device. A fingertip detection user interface module included in the information device for detecting the fingertip of the user based on image information obtained by imaging the specific form according to
Using a fan having a central angle corresponding to the sharpness of the fingertip as a filter, the flesh-colored portion of the image information while rotating the fan with the center of the circle including the fan corresponding to each pixel of the image information The process of obtaining the change in the overlap area between the fan and the fan is performed for all the pixels by moving the filter to each pixel, and the angle range in which the overlap area substantially coincides with the fan area is obtained as a predetermined threshold value or less. A fingertip detection user interface module comprising a fingertip vicinity data extraction processing unit for detecting a point as a pixel near the fingertip.

Images