CN111553891A - Handheld object existence detection method - Google Patents

Abstract

The invention belongs to the technical field of visual identification, and relates to a method for detecting existence of a handheld object. The sensor adopted by the method is a camera sensor integrating a color camera and a depth camera. The method comprises the steps of simultaneously acquiring color, depth and human skeleton information through a sensor, mapping human hand joint coordinate points to a depth image, extracting a hand mask region through a region growing method, mapping the hand mask region to a color image, and judging hand skin proportion based on an HSV threshold segmentation method so as to determine whether a held object exists. The invention judges whether a holding object exists or not in a visual identification mode, thereby providing a basis for judging the intention of human-computer interaction; the detection precision of the robot for human intentions can be greatly improved, and misjudgments are reduced.

Description

A method for detecting the presence of a hand-held object

technical field

The invention belongs to the technical field of visual recognition, and relates to a method for detecting the existence of a hand-held object.

Background technique

Since the birth of the first industrial robot 150 years ago, people have been working on using robots to replace the heavy work of humans. From the perspective of development history, it has roughly gone through three stages. In the early stage, the first-generation robot is called a teaching robot, which is mainly taught by the operator, and the robot is repeatedly taught to repeat the teaching action; the second-generation robot is called a robot that can perceive external information, and it mainly configures various sensors by configuring various sensors. , to perceive information such as sight, touch, force, etc. The third-generation robot is called intelligent robot, which is also a stage currently being explored. It can independently judge task requirements according to external environmental information and freely interact with humans.

Intelligent human-machine object transfer is an important part of intelligent robots. In order for humans to be able to carry out the transfer process with robots, it is necessary for the robot to be able to judge the intention of the human transmitter, and to detect whether there is an object in the human hand can greatly improve the robot's detection accuracy of human intention, reduce misjudgment. At present, the domestic research in this area is still blank.

SUMMARY OF THE INVENTION

In view of the problems existing in the prior art, the present invention proposes a method for detecting the existence of a handheld object.

The technical scheme adopted in the present invention is:

A method for detecting the existence of a hand-held object, the sensor used in the method is a camera sensor integrating a color camera and a depth camera. Simultaneously acquire color, depth and human skeleton information through the sensor, map the joint coordinates of the human hand to the depth image, extract the hand mask area by the region growing method, and then map it to the color image, based on HSV threshold segmentation The method judges the proportion of the skin of the hand, so as to confirm whether there is a holding object. Specifically include the following steps:

(1) Obtain the conversion relationship between the depth camera and the color camera image.

Use Zhang Zhengyou's calibration method to obtain the internal parameters of the color camera, the depth camera and the external parameters of the corresponding checkerboard image. In this way, the pixel coordinate system-camera coordinate system-world coordinate system of the two cameras is connected to each other to prepare for subsequent image alignment.

与相机坐标系下点

的转接关系如公式(1)所示。For optical imaging systems, there are image pixels

point under the camera coordinate system

The transfer relationship of , is shown in formula (1).

z·p=K·P (1)

dx和dy代表每一列和每一行的像素点与实际单位mm的转换关系；f为相机焦距；f_x＝f/dx和f_y＝f/dy分别表示相机在水平和竖直两个方向上的尺度因子；u₀和v₀分别代表相机光心与像素坐标系原点在水平及竖直方向上的偏移量。Among them, K is the camera intrinsic parameter matrix,

dx and dy represent the conversion relationship between the pixels of each column and each row and the actual unit mm; f is the focal length of the camera; f _x =f/dx and f _y =f/dy represent the camera in the horizontal and vertical directions, respectively The scale factor of ; u ₀ and v ₀ represent the offset of the camera optical center and the origin of the pixel coordinate system in the horizontal and vertical directions, respectively.

From the formula (1), the conversion relationship between the pixel coordinate point p _rgb of the color camera and the coordinate point P _rgb of the color camera coordinate system can be obtained as shown in the formula (2):

z _rgb · p _rgb = K _rgb · P _rgb (2)

For the same reason, the conversion relationship between the image pixel coordinate point p _dept h of the depth camera and the coordinate point P _{dept h} of the depth camera coordinate system can be obtained by formula (1), as shown in formula (3):

z _{dept h} ·p _{dept h} =K _{dept h} ·P _depth (3)

For the same checkerboard image, the extrinsic parameters R _CO and T _CO of the color camera and the extrinsic parameters R _DO and T _DO of the depth camera can be obtained, and the relationship between the two can be obtained as follows:

T _CD =T _CO -R _CD ·T _DO (5)

For the coordinate points P _rgb and P _{dept h} in the respective camera coordinate systems in the inhomogeneous coordinate system, the relationship is as follows:

P _rgb = R _CD · P _{dept h} + T _CD (6)

Combine formula (2), formula (3), and formula (6) to get:

z _rgb · p _rgb = K _rgb · R _CD · K _{dept h} ^-1 · z _{dept h} · p _{dept h} +K _rgb · T _CD (7)

where z _rgb =z _{dept h} . Then the formula (7) is the conversion relationship between the depth and the pixel coordinate system corresponding to the color image.

(2) The optical axes of the two cameras are parallel to the ground, and the two cameras are installed on the robot platform. The distance between the human body and the camera is within 1-2.5m. Let the camera look directly at the position of the hand. Part occlusion, collect color image and depth image data.

(3) Image preprocessing. Gaussian filtering is performed on the depth image data to fill in the missing depth points. Convert the color image to the HSV color space, obtain the HSV image, and perform Gaussian filtering on it.

(4) Use the skeleton recognition program to read the recognized human hand joints, and obtain the hand coordinates P _hand = (u, v, z), where u, v represent the coordinates of the hand coordinates in the pixel coordinate system of the depth camera , z represents the depth corresponding to the joint.

(5) Set P _hand as the seed point, and use the region growing method in the depth image to iteratively traverse the coordinate points whose depth values are in the range of [zT _l , z+T _r ], where T _l is the upper bound of the segmentation threshold, T _r is the segmentation threshold lower bound. And record all the growth coordinate points to obtain the hand-related area mask. The segmentation threshold is set by manual adjustment to ensure that the hand-related region mask can clearly segment the hand region.

(6) the hand-related area mask obtained in step (5) is mapped on the HSV image, obtains the HSV image of the hand-related area, traverses the area and integrates to obtain the area area S _all ; Simultaneously according to the skin color specific situation Set the HSV color threshold of the hand skin, traverse the HSV images of the relevant areas of the hand, and integrate the part in the skin color threshold range to obtain the skin integral area S _skin .

根据预设的比例阈值S进行手持物体存在判断，当s<S时视为存在手持物体，反之则不存在。所述的比例阈值S范围[0.4,0.7]，具体阈值需根据实际效果调整。(7) Obtain the scale factor of hand skin

According to the preset ratio threshold S, the existence of the hand-held object is judged. When s<S, it is considered that the hand-held object exists, otherwise it does not exist. The proportional threshold S is in the range of [0.4, 0.7], and the specific threshold needs to be adjusted according to the actual effect.

Further, in the step (6), there are various ways of setting the HSV color threshold of the skin of the hand, and the default threshold can be directly set, or set according to the recognized human face skin color interval as a basis, or use Special color gloves are used to limit the color of the hand to improve the recognition accuracy.

The beneficial effects of the present invention are as follows: in order to fill the existing research gap, a method for detecting the existence of a hand-held object is innovatively proposed, and the existence of a held object is judged by means of visual recognition, so as to provide a basis for the judgment of intention in human-computer interaction .

Description of drawings

Figure 1 is a flow chart of the program.

Figure 2 is a hand region mask image.

Figure 3 is a hand segmented image.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and examples.

The specific implementation process of the present invention adopts a camera sensor integrating a color camera and a depth camera, which can acquire color and depth images in real time, and has a built-in program to extract bone coordinate points. Its effective viewing angle is 70° in the horizontal direction and 60° in the vertical direction, the effective depth range is 0.5-4.5m, the frame rate is 30FPS, and the depth image resolution is 512*424.

A method for detecting the presence of a handheld object, the main process is shown in Figure 1, including the following steps:

与相机坐标系下点

的转接关系如公式(1)所示。(1) Obtain the conversion relationship between the depth camera and the color camera image. Use Zhang Zhengyou's calibration method to obtain the internal parameters of the color camera, the depth camera and the external parameters of the corresponding checkerboard image. In this way, the pixel coordinate system of the two cameras—the camera coordinate system—the world coordinate system is connected to each other to prepare for subsequent image alignment. For optical imaging systems, there are image pixels

point under the camera coordinate system

The transfer relationship of , is shown in formula (1).

z·p=K·P (1)

dx和dy代表每一列和每一行的像素点与实际单位mm的转换关系，f为相机焦距，f_x＝f/dx和f_y＝f/dy分别表示相机在水平和竖直两个方向上的尺度因子，u₀和v₀分别代表相机光心与像素坐标系原点在水平及竖直方向上的偏移量。Among them, K is the camera intrinsic parameter matrix,

dx and dy represent the conversion relationship between the pixels of each column and each row and the actual unit mm, f is the focal length of the camera, f _x =f/dx and f _y =f/dy represent the camera in the horizontal and vertical directions, respectively The scale factor of , u ₀ and v ₀ represent the offset of the camera optical center and the origin of the pixel coordinate system in the horizontal and vertical directions, respectively.

From (1), the conversion relationship between the pixel coordinate point p _rgb of the color camera and the point P _rgb in the camera coordinate system of the color camera can be obtained as shown in (2).

z _rgb · p _rgb = K _rgb · P _rgb (2)

For the same reason (1), the conversion relationship between the image pixel coordinate point p _{dept h} of the depth camera and the camera coordinate system coordinate point P _{dept h} of the depth camera can be obtained as shown in (3).

z _{dept h} ·p _{dept h} =K _{dept h} ·P _{dept h} (3)

For the same checkerboard image, the extrinsic parameters R _CO and T _CO of the color camera and the extrinsic parameters R _DO and T _DO of the depth camera can be obtained. The relationship between the two can be obtained as follows:

T _CD =T _CO -R _CD ·T _DO (5)

For the coordinate points P _rgb and P _{dept h} in the respective camera coordinate systems in the inhomogeneous coordinate system, the relationship is as follows:

P _rgb = R _CD · P _{dept h} + T _CD (6)

Simultaneous equations (2)(3)(6) are:

z _rgb · p _rgb = K _rgb · R _CD · K _{dept h} ^-1 · z _{dept h} · p _{dept h} +K _rgb · T _CD (7)

where z _rgb =z _{dept h} . Then the formula (7) is the conversion relationship between the depth and the pixel coordinate system corresponding to the color image.

(2) The optical axis of the camera is parallel to the ground, installed on the robot platform, and the human body is within 2m from the camera. Let the camera look directly at the position of the hand, pay attention not to be blocked by other parts of the body, and collect color image and depth image data.

(3) Image preprocessing. Perform Gaussian filtering on the depth image, select a 5×5 Gaussian kernel, and fill in the missing depth points. Convert the color image to the HSV color space to obtain the HSV image, and select a 3×3 Gaussian kernel to perform Gaussian filtering on the HSV image.

(4) Use the bone recognition program to read the recognized human hand joints, and obtain the hand coordinates P _hand = (u, v, z), where u, v represent the coordinates of the hand coordinates in the pixel coordinate system of the depth camera, z represents the depth corresponding to this joint.

(5) Set P _hand as the seed point, and use the region growing method in the depth image to iteratively traverse the coordinate points whose depth values are in the range of [zT _l , z+T _r ], where T _l =20mm, T _r =20mm . And record all the growth coordinate points to obtain the hand-related area mask, the mask image is shown in Figure 2.

(6) this hand-related area mask is mapped on the HSV image, obtain the HSV image of the hand-related area, traverse this area and carry out integration, obtain the area area S _all ; Set hand skin according to yellow race skin color simultaneously The HSV color threshold of the hand is traversed through the HSV image of the relevant area of the hand, and the part in the skin color threshold range is integrated to obtain the skin integral area S _skin .

根据预设的比例阈值S＝0.55进行手持物体存在判断，当s<S时视为存在手持物体，反之则不存在。(7) Obtain the scale factor of hand skin

According to the preset ratio threshold value S=0.55, the existence of the hand-held object is judged, and when s<S, it is regarded as the existence of the hand-held object, otherwise it does not exist.

The above-mentioned embodiments only represent the embodiments of the present invention, but should not be construed as a limitation on the scope of the present invention. It should be pointed out that for those skilled in the art, without departing from the concept of the present invention, Several modifications and improvements can also be made, which all belong to the protection scope of the present invention.

Claims (3)

1. A method for detecting the presence of a hand-held object, comprising the steps of:

(1) acquiring a conversion relation between a depth camera and a color camera image;

firstly, acquiring internal parameters of a color camera and a depth camera and external parameters of a corresponding checkerboard image, and further establishing a relationship between a pixel coordinate system, a camera coordinate system and a world coordinate system of the two cameras;

for optical imaging systems, there are image pixels

And the lower point of the camera coordinate system

The switching relationship of (a) is shown in formula (1);

z·p＝K·P (1)

wherein K is a camera internal reference matrix,

dx and dy represent the conversion relationship of the pixel points of each column and each row to the actual unit mm; f is the focal length of the camera; f. of_xF/dx and f_yF/dy represents the scale factor of the camera in both horizontal and vertical directions, respectively; u. of₀And v₀Respectively representing the offset of the optical center of the camera and the origin of the pixel coordinate system in the horizontal and vertical directions;

obtaining the image pixel coordinate point p of the color camera by the formula (1)_rgbCoordinate point P of color camera coordinate system_rgbThe conversion relationship of (c) is shown in equation (2):

z_rgb·p_rgb＝K_rgb·P_rgb(2)

the image pixel coordinate point P of the depth camera obtained by the formula (1) is similar to the image pixel coordinate point P_depthCoordinate point P of coordinate system of depth camera_depthThe conversion relationship of (c) is shown in equation (3):

z_depth·p_depth＝K_depth·P_depth(3)

obtaining the external parameter R of the color camera for the same checkerboard image_COAnd T_COAnd outer reference R of depth camera_DOAnd T_DOFurther, the following relationship is obtained:

T_CD＝T_CO-R_CD·T_DO(5)

for coordinate points P under respective camera coordinate systems under the nonhomogeneous coordinate system_rgbAnd P_depthThere is a relationship as follows:

P_rgb＝R_CD·P_depth+T_CD(6)

simultaneous formula (2), formula (3), formula (6) yields:

z_rgb·p_rgb＝K_rgb·R_CD·K_depth ^-1·z_depth·p_depth+K_rgb·T_CD(7)

wherein z is_rgb＝z_depth(ii) a The formula (7) is the conversion relation between the depth and the pixel coordinate system corresponding to the color image;

(2) optical axes of the two cameras are parallel to the ground, the two cameras are installed on a robot platform, the distance between a human body and the cameras is 1-2.5m, the cameras are enabled to look directly at the position of a hand, the hand is not shielded by other parts, and color images and depth image data are collected;

(3) preprocessing an image; carrying out Gaussian filtering on the depth image data, and filling lost depth points; converting the color image into an HSV color space, acquiring an HSV image, and performing Gaussian filtering processing on the HSV image;

(4) reading the recognized hand joints of the human body by using a skeleton recognition program, and acquiring hand coordinates P_handThe joint is divided into a plurality of joints, wherein the joints are located in the same pixel coordinate system of the depth camera, and the joints are located in the same pixel coordinate system of the depth camera;

(5) setting P_handFor the seed point, the depth value is iteratively traversed in the depth image by adopting a region growing method in the z-T_l，z+T_r]Coordinate points within a range, wherein T_lIs the upper boundary of the segmentation threshold, T_rIs the lower boundary of the segmentation threshold; recording all growth coordinate points to obtain a hand related area mask; the segmentation threshold is set through manual adjustment, so that the hand region can be clearly segmented by the hand related region mask;

(6) mapping the mask of the hand related region obtained in the step (5) onto an HSV image to obtain the HSV image of the hand related region, traversing the region and integrating to obtain the region area S_all(ii) a Meanwhile, the HSV color threshold of the hand skin is set according to the specific skin color condition, HSV images of the hand relevant area are traversed, the part in the skin color threshold interval is integrated, and the skin integral area S is obtained_skin；

(7) Calculating hand skin scale factor

Judging the existence of the handheld object according to a preset proportion threshold value S, and judging when S is<And S is regarded as the existence of the handheld object, and otherwise, the handheld object does not exist.

2. A method as claimed in claim 1, wherein said scale threshold S is in the range [0.4,0.7 ].

3. A hand-held object presence detection method according to claim 1 or 2, wherein the HSV hand color threshold in step (6) is set by: the default threshold value is directly set, or the default threshold value is set according to the recognized human face skin color interval, or the hand color is limited by using gloves with special colors so as to improve the recognition accuracy.

Images