JP2021517281A - Multi-gesture fine division method for smart home scenes - Google Patents

Abstract

The present invention discloses a multi-gesture fine division method for a smart home scene, and includes the following steps. S1) Gesture image Image0 is preprocessed to obtain image Image1; S2) Preprocessed image Image1 is skin-colored and processed to obtain image Image4; S3) Image 4 is the minimum image. Construct a combined rectangular MBR; S4) Exclude the non-gesture area in the image Image4 by the non-gesture area exclusion criteria to obtain the gesture image Image5; S5) Image Image5 by the arm redundancy removal algorithm based on the shape feature of the hand. By processing, the removal of arm redundancy is completed. The present invention can intelligently divide gestures locally, is quick and accurate as a whole process, and significantly improves the usability of gesture-based human-computer interaction systems. [Selection diagram] Fig. 3

Description

The present invention relates to a smart recognition method, and specifically relates to a multi-gesture fine division method for a smart home scene belonging to the field of smart home.

Gesture division refers to a technique for dividing gesture information from a complex image background. The quality of gesture partitioning (accuracy, integrity, redundancy) has a significant impact on the accuracy of gesture-based human-computer interaction systems identification and detection.

Real-time gesture splitting for home scenes is more complex. User gestures are not only more complex and diverse, but also susceptible to factors such as background, lighting and shooting angle. In the field of computer vision today, there is no adaptable gesture splitting algorithm. Some of the current typical gesture splitting methods rely primarily on external devices or require special processing in the hands of the user. These technologies limit the scope of human activity and are difficult to achieve large-scale adoption in practical use due to reasons such as hardware equipment setup, manufacturing costs, and their high costs.

In response to technological development, smart home devices with gesture splitting capabilities are rare in the current market, and most gesture splitting products can only split the skin completely and accurately. No, the split effect is not ideal. In addition, these devices often depend on cloud servers or networks too much, and cannot operate without a network.

Summarizing the above, it is for the current traders to propose a multi-gesture fine division method for new smart home scenes based on the conventional technology and to realize large-scale spread and application of gesture division technology in smart home devices. It is a problem that needs to be resolved.

An object of the present invention is to propose a multi-gesture fine division method for a smart home scene in view of the above deficiencies existing in the prior art. The method includes the following steps.
S1) Preprocessing is performed on the gesture image Image0 to obtain the image Image1;
S2) Skin color division is performed on the image Image 1 after the pretreatment to obtain the image Image 4 after the treatment;
S3) Build a minimal-coupled rectangular MBR of images in image Image4;
S4) The non-gesture area in the image Image 4 is excluded by the non-gesture area exclusion criterion, and the gesture image Image 5 is acquired;
S5) The removal of the arm redundancy is completed by processing the image Image 5 by the arm redundancy removal algorithm based on the shape feature of the hand.

The preprocessing in S1 preferably includes at least gesture image noise removal, gesture image binarization, and morphology processing.

Specifically, S2 preferably includes the following steps.
S21) The Image1 image is converted from the RGB color space to the YCbCr color space to obtain the image Image2, and further, the binarized image Image3 is obtained by comparing with the threshold value for each pixel by the global fixed threshold binarization method;
S22) The holes and gaps in the binarized image Image3 are removed by using the expansion corrosion calculation in morphology, and the binarized image is processed by using a median filter to obtain the image Image4.

Specifically, S3 stores the contour information of the binarized gesture image obtained in S2 in the list contours, and obtains the coordinates of the four vertices of the circumscribing rectangle as top_left, top_right, bottom_left, and bottom_right from the coordinate information, respectively. Is preferably included.

The non-gesture area exclusion criteria in S4 specifically preferably include:
1) When an image size of 640 * 480 is acquired, if the area of the circumscribing rectangle is less than 2500, it is recognized as a non-gesture area;
2) If the ratio of the length to the width of the circumscribing rectangle is greater than 5, it is recognized as a non-gesture area;
3) When the ratio of the number of points of the pixel value 255 in the circumscribing rectangle to the rectangular area is larger than 0.8 or smaller than 0.4, it is recognized as a non-gesture area.

The arm redundancy removal algorithm based on the shape feature of the hand in S5 specifically includes a hand width distribution histogram in which the maximum value of the width and its corresponding coordinates are the thumb carpometacarpal joint for the image Image6. , The coordinates of the division line of the wrist are preferably determined by the values after searching for the thumb carpometacarpal joint point in the hand gradient distribution histogram, including statistic with the hand gradient distribution histogram.

The coordinates of the division line of the wrist in step S5 are determined by determining that the gradient of the current point is 0 and the gradient of the next point is 0 or more, and the thumb carpometacarpal joint point in the hand gradient distribution histogram is used. It is preferably determined by the value after searching.

The advantages of the present invention as compared with the prior art are mainly as follows.

The multi-gesture fine division method for smart home scenes according to the present invention can intelligently divide gestures locally, overcomes the deficiency of the prior art that it depends too much on the network, and the device to which this method is applied is connected to the network. It can work normally without it.

The present invention converts a gesture picture from an RGB color space to a YCbCr color space, and further completes skin color division by a global fixed threshold binarization method. Then, excluding the non-gesture area, construct the MBR and MABR of the gesture contour, rotate the gesture image to stat the hand width, build the width distribution histogram and the gradient distribution histogram based on the width, and divide the wrist. Determine the line. Finally, the removal of arm redundancy is completed and a complete gesture image is obtained. The present invention makes it possible to quickly and accurately divide gestures in a home environment image, significantly improving the comfort of using a gesture-based human-computer interaction system and improving user satisfaction.

In addition, the present invention can be used as a reference for other related problems in the field, can be extended based on the present invention, can be applied to other technical proposals related to gesture division, and has a sufficiently wide application. Has a range.

Hereinafter, specific embodiments of the present invention will be described in more detail with reference to the drawings of Examples in order to facilitate understanding and understanding of the technical idea of the present invention.

It is a flowchart of the step which performs the skin color division with respect to the gesture image according to this invention. It is a flowchart of the step which removes the arm redundancy with respect to the gesture image according to this invention. It is a flowchart of the whole step of the multi-gesture fine division method for smart home scenes by this invention.

The present invention discloses a multi-gesture fine division method for smart home scenes. The method is based on a skin color division algorithm for the YCbCr color space, a non-gesture area exclusion criterion, and an arm redundancy removal algorithm based on hand shape features. The method of the present invention includes the following steps. S1) Gesture image Image0 is preprocessed to obtain image Image1; S2) Preprocessed image Image1 is skin-colored and processed to obtain image Image4; S3) Image 4 is the minimum image. Construct a combined rectangular MBR; S4) exclude the non-gesture area in the image Image 4 by the non-gesture area exclusion criteria to obtain the gesture image Image 5; S5) image Image 5 by the arm redundancy removal algorithm based on the shape feature of the hand. By processing, the removal of arm redundancy is completed.

As is clear from the expression of the above steps, the method of the present invention mainly includes two major aspects of skin color division and removal of arm redundancy.

Hereinafter, the method of the present invention will be specifically described with reference to the drawings. FIG. 1 shows a skin color division method for a gesture image according to an embodiment of the present invention. The steps of the method mainly include the following steps.

In S1, the gesture image Image0 is preprocessed to obtain the image Image1.

Since the gesture image inevitably has noise interference at the time of acquisition and has a great influence on the division and recognition of the gesture image, the preprocessing for the image before the gesture division is particularly important. The preprocessing includes at least gesture image noise removal, gesture image binarization, and morphology processing.

Here, the gesture image noise removal mainly uses a Gaussian filter belonging to a linear filter. The pixel values of the filter window follow a Gaussian distribution that decreases as the distance from the template center increases, and the two-dimensional Gaussian function is:

Here, h (x, y) represents a value on the (x, y) coordinates in the Gaussian filter, and σ represents a standard deviation.

In S21, the Image1 image is converted from the RGB color space to the YCbCr color space to obtain the image Image2, and further, the binarized image Image3 is obtained by comparing with the threshold value for each pixel by the global fixed threshold binarization method.

The YCbCr color space is a color space often used in video and digital images. Cb, which represents the brightness of an image and represents the difference between the Y (luma) component, which takes a value from the range of 0 to 255, and the blue component in the RGB color space and the brightness value in the RGB color space, and takes a value from the range of 0 to 255. It contains three components, a component, and a Cr component that represents the difference between the value of the red component in the RGB color space and the brightness in the RGB color space and takes a value from the range of 0 to 255. Here, the Cb component and the Cr component are independent of each other and can be efficiently separated from the Y component.

The conversion formula from the RGB color space to the YCbCr color space is as follows.

It is converted to the following matrix format.

The comparison with the threshold value for each pixel described in the above step is based on the assumption that the Y, Cb, and Cr values of the human skin color are about "0: 256, 130: 174, 77: 128" in the image. If the YCbCr value of the pixel belongs to this section, the pixel value is set to 255, and if not, the binarized image Image3 can be obtained.

In S22, the holes and gaps in the binarized image Image3 are removed using the expansion corrosion calculation in morphology, and the binarized image is processed with a median filter to obtain the image Image4.

The gesture image may have gaps, chips, or the like in the obtained image by undergoing the binarization process. The role of morphology is to remove isolated small dots, burrs, fill small holes, fill small gaps, etc., and there are four main types of morphology operations as follows.
1. Expansion. The expansion operation process in the morphology operation integrates the background points in contact with the object into the object. As a result, the area of the object becomes large. This means filling the cavities and gaps that exist in the target area.
2, corrosion. The corrosion calculation process in the morphology calculation erases all the boundary points of the object. As a result, the area of the object becomes smaller. This means removing small, meaningless isolated points that exist in the target area.
3, open operation. In the open calculation process in the morphology calculation, the corrosion calculation is first performed on the binarized image, and then the expansion calculation is performed. The meaning is to erase meaningless points such as small isolated points and burrs existing in the target area (corrosion calculation) and fill cavities and gaps (expansion calculation).
4, closed operation. In the closing calculation process in the morphology calculation, the expansion calculation is first performed on the binarized image, and then the corrosion calculation is performed. The meaning is to fill the cavities and gaps existing in the target area (expansion calculation) and to eliminate meaningless points such as small isolated points and burrs (corrosion calculation).

The median filter belongs to a non-linear filter, and mainly performs statistics and sorting of pixel points around the current point, and extracts the median value as the pixel value of the current point to remove isolated noise points. Gesture binarization mainly with a median filter The burr of the image edge is smoothed, the edge is smoothed, and the influence on the search of the dividing line of the wrist is reduced.

FIG. 2 shows a method of removing arm redundancy with respect to a gesture image according to an embodiment of the present invention, and the steps of the method mainly include the following steps.

In S3, the minimally coupled rectangular MBR of the gesture image is constructed.

The minimally coupled rectangular MBR of the gesture image is constructed in the image Image 4, and the vertex coordinate information thereof is as follows.

The MABR of the image is constructed based on the MBR, and on the premise that the contour is known, the convex hull of the gesture contour is obtained by the Graham scan method, and the figure is β within the range of 90 degrees with the center of the MBR as the origin. It can be rotated at equal intervals on the scale. When the MBR area of the pattern corresponding to the rotation angle is recorded at the same time, the MBR corresponding to the smallest MBR area in the recording is the required MABR.

In S4, the non-gesture area in the image Image 4 is excluded by the non-gesture area exclusion criterion, and the gesture image Image 5 is acquired.

The non-gesture area exclusion criteria specifically include:
1) When an image size of 640 * 480 is acquired, if the area of the circumscribing rectangle is less than 2500, it is recognized as a non-gesture area;
2) If the ratio of the length to the width of the circumscribing rectangle is greater than 5, it is recognized as a non-gesture area;
3) When the ratio of the number of points of the pixel value 255 in the circumscribing rectangle to the rectangular area is larger than 0.8 or smaller than 0.4, it is recognized as a non-gesture area.

After that, when the binarized gesture image is rotated counterclockwise, a rotation angle corresponding to MABR is obtained in the above step, and when the gesture image is rotated counterclockwise, the gesture direction can be made vertical.

In S5, the removal of arm redundancy is completed by processing the image Image 5 by the arm redundancy removal algorithm based on the shape feature of the hand.

The arm redundancy removal algorithm based on the shape feature of the hand in S5 specifically includes a hand width distribution histogram in which the maximum value of the width and its corresponding coordinates are the thumb carpometacarpal joint for the image Image6. The coordinates of the division line of the wrist are determined by the values after looking for the thumb carpometacarpal joint point in the hand gradient distribution histogram, including statistic with the hand gradient distribution histogram.

The calculation procedure of the width histogram is as follows.
rows = image.shape [0]
x_label = []
width = []
step = 3
for index in range (step, rows, step):
x_label.append (index)
width.appned (sum (sum (binary [index-3: index] / 255)) / 3)

The procedure for calculating the gradient histogram is as follows.
gradient = [0]
for index in range (1, len (width)):
gradient.append (width [index]-width [index-1])

After that, the dividing line of the wrist is determined. Since the maximum value of the gesture width in the hand width distribution histogram and its corresponding coordinates are the thumb carpometacarpal joint, search for the thumb carpometacarpal joint point in the hand gradient distribution histogram as the coordinates of the wrist dividing line. It is determined by the value after that, and as the determination method, the gradient of the current point is 0 and the gradient of the next point is 0 or more.

Finally, the removal of arm redundancy is completed. In the above step, the coordinate information of the wrist dividing line is obtained, and the pixel value below the wrist dividing line is 0, that is, only the upper gesture image is left, and the arm portion is removed.

Hereinafter, with reference to FIG. 3, a specific example of the multi-gesture fine division method for the smart home scene will be proposed. The embodiment mainly includes the following steps.

In step S301, an image is acquired.

The home image is acquired mainly by a 2D camera.

In step S302, preprocessing is performed on the acquired image.

The image is subjected to filter processing, morphology processing, binarization processing, and the like.

In step S303, skin color division is performed on the image.

In the YCbCr color space, binarization processing is performed using the global fixed threshold method, and contour information of each region is acquired by the 8-nearest neighbor method.

In step S304, the non-gesture area is filtered.

By performing non-gesture area filtering on the gesture image cut out in step S303, first, an MBR of the gesture image is constructed, areas that do not meet the conditions are filtered, and those that meet the conditions are subjected to gesture division processing. Do.

In step S305, gesture division is performed on the image.

By constructing MABR based on the gesture image MBR, acquiring the deflection angle of the gesture image, and analyzing the hand width distribution histogram and the hand gradient distribution histogram, the wrist dividing line of the gesture is acquired and the arm area is filtered. To do.

In step S306, a complete gesture image is acquired.

Gesture splitting produces multiple gestures from zero. Gestures in the image are extracted for subsequent needs and used for gesture-based human-computer interaction systems, allowing control of the home device from humans through gestures.

The multi-gesture fine division method for smart home scenes according to the present invention can intelligently divide gestures locally, overcomes the deficiency of the prior art that it depends too much on the network, and the device to which this method is applied is connected to the network. It can work normally without it.

The present invention converts a gesture picture from an RGB color space to a YCbCr color space, and further completes skin color division by a global fixed threshold binarization method. Then, excluding the non-gesture area, construct the MBR and MABR of the gesture contour, rotate the gesture image to stat the hand width, build the width distribution histogram and the gradient distribution histogram based on the width, and divide the wrist. Determine the line. Finally, the removal of arm redundancy is completed and a complete gesture image is obtained. The present invention makes it possible to quickly and accurately divide gestures in a home environment image, significantly improving the comfort of using a gesture-based human-computer interaction system and improving user satisfaction.

In addition, the present invention can be used as a reference for other related problems in the field, can be extended based on the present invention, can be applied to other technical proposals related to gesture division, and has a sufficiently wide application. Has a range.

The present invention is not limited to the details of the above-mentioned exemplary examples, and can be carried out in other specific forms without departing from the gist of the present invention or the basic configuration. It is self-evident to those skilled in the art. Therefore, in all respects, the examples should be considered exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the description above. Therefore, the meaning of the equivalents in the claims and all modifications within the scope are intended to be incorporated within the invention, and any reference code in the claims shall cover the relevant claims. It should not be considered a limitation.

Moreover, although described herein according to embodiments, each embodiment does not include only independent technical solutions. Such a description scheme herein is for clarity only. It should be understood by those skilled in the art that, as a whole, the technical solutions in each embodiment may be appropriately combined to form other embodiments that can be understood by those skilled in the art. Is.

(Additional note)
(Appendix 1)
A multi-gesture fine division method for smart home scenes that includes the following steps.
S1) Preprocessing is performed on the gesture image Image0 to obtain the image Image1;
S2) Skin color division is performed on the image Image 1 after the pretreatment to obtain the image Image 4 after the treatment;
S3) Build a minimal-coupled rectangular MBR of images in image Image4;
S4) The non-gesture area in the image Image 4 is excluded by the non-gesture area exclusion criterion, and the gesture image Image 5 is acquired;
S5) The removal of the arm redundancy is completed by processing the image Image 5 by the arm redundancy removal algorithm based on the shape feature of the hand.

(Appendix 2)
The multi-gesture fine division method for a smart home scene according to Appendix 1, wherein the pre-processing in S1 includes at least gesture image noise removal, gesture image binarization, and morphology processing.

(Appendix 3)
S2 is a multi-gesture fine division method for a smart home scene according to Appendix 1, which specifically includes the following steps.
S21) The Image1 image is converted from the RGB color space to the YCbCr color space to obtain the image Image2, and further, the binarized image Image3 is obtained by comparing with the threshold value for each pixel by the global fixed threshold binarization method;
S22) The holes and gaps in the binarized image Image3 are removed by using the expansion corrosion calculation in morphology, and the binarized image is processed by using a median filter to obtain the image Image4.

(Appendix 4)
Specifically, S3 stores the contour information of the binarized gesture image obtained in S2 in the list contours, and obtains the coordinates of the four vertices of the circumscribing rectangle as top_left, top_right, bottom_left, and bottom_right from the coordinate information, respectively. The multi-gesture fine division method for the smart home scene according to Appendix 1, which comprises.

(Appendix 5)
The multi-gesture fine division method for a smart home scene according to Appendix 1, wherein the non-gesture area exclusion criterion in S4 specifically includes the following.
1) When an image size of 640 * 480 is acquired, if the area of the circumscribing rectangle is less than 2500, it is recognized as a non-gesture area;
2) If the ratio of the length to the width of the circumscribing rectangle is greater than 5, it is recognized as a non-gesture area;
3) When the ratio of the number of points of the pixel value 255 in the circumscribing rectangle to the rectangular area is larger than 0.8 or smaller than 0.4, it is recognized as a non-gesture area.

(Appendix 6)
The arm redundancy removal algorithm based on the shape feature of the hand in S5 specifically includes a hand width distribution histogram in which the maximum value of the width and its corresponding coordinates are the thumb carpometacarpal joint for the image Image6. Includes statistics on the hand gradient distribution histogram
The multi-gesture fine division for the smart home scene according to Appendix 1, wherein the coordinates of the division line of the wrist are determined by the value after searching the thumb-carpometacarpal joint point in the gradient distribution histogram of the hand. Method.

(Appendix 7)
The coordinates of the division line of the wrist in step S5 are the thumb carpometacarpal joint points in the hand gradient distribution histogram based on the fact that the gradient of the current point is 0 and the gradient of the next point is 0 or more. The multi-gesture fine division method for a smart home scene according to Appendix 6, which is determined by a value after searching.

Claims (7)

A multi-gesture fine division method for smart home scenes that includes the following steps.
S1) Preprocessing is performed on the gesture image Image0 to obtain the image Image1;
S2) Skin color division is performed on the image Image 1 after the pretreatment to obtain the image Image 4 after the treatment;
S3) Build a minimal-coupled rectangular MBR of images in image Image4;
S4) The non-gesture area in the image Image 4 is excluded by the non-gesture area exclusion criterion, and the gesture image Image 5 is acquired;
S5) The removal of the arm redundancy is completed by processing the image Image 5 by the arm redundancy removal algorithm based on the shape feature of the hand. The multi-gesture fine division method for a smart home scene according to claim 1, wherein the pre-processing in S1 includes at least gesture image noise removal, gesture image binarization, and morphology processing. The multi-gesture fine division method for a smart home scene according to claim 1, wherein S2 specifically includes the following steps.
S21) The Image1 image is converted from the RGB color space to the YCbCr color space to obtain the image Image2, and further, the binarized image Image3 is obtained by comparing with the threshold value for each pixel by the global fixed threshold binarization method;
S22) The holes and gaps in the binarized image Image3 are removed by using the expansion corrosion calculation in morphology, and the binarized image is processed by using a median filter to obtain the image Image4. Specifically, S3 stores the contour information of the binarized gesture image obtained in S2 in the list contours, and obtains the coordinates of the four vertices of the circumscribing rectangle as top_left, top_right, bottom_left, and bottom_right from the coordinate information, respectively. The multi-gesture fine division method for a smart home scene according to claim 1, wherein the method includes. The multi-gesture fine division method for a smart home scene according to claim 1, wherein the non-gesture area exclusion criterion in S4 specifically includes the following.
1) When an image size of 640 * 480 is acquired, if the area of the circumscribing rectangle is less than 2500, it is recognized as a non-gesture area;
2) If the ratio of the length to the width of the circumscribing rectangle is greater than 5, it is recognized as a non-gesture area;
3) When the ratio of the number of points of the pixel value 255 in the circumscribing rectangle to the rectangular area is larger than 0.8 or smaller than 0.4, it is recognized as a non-gesture area. The arm redundancy removal algorithm based on the shape feature of the hand in S5 specifically includes a hand width distribution histogram in which the maximum value of the width and its corresponding coordinates are the thumb carpometacarpal joint for the image Image6. Includes statistics on the hand gradient distribution histogram
The multi-gesture precision for the smart home scene according to claim 1, wherein the coordinates of the division line of the wrist are determined by the value after searching the thumb-carpometacarpal joint point in the gradient distribution histogram of the hand. How to divide. The coordinates of the division line of the wrist in step S5 are the thumb carpometacarpal joint points in the hand gradient distribution histogram based on the fact that the gradient of the current point is 0 and the gradient of the next point is 0 or more. The multi-gesture fine division method for a smart home scene according to claim 6, wherein the value is determined after searching.

Images