CN115089212A

CN115089212A - Three-dimensional vision-guided automatic neck ultrasonic scanning method and system for mechanical arm

Info

Publication number: CN115089212A
Application number: CN202210496051.2A
Authority: CN
Inventors: 何昊; 李�浩; 孟玲; 肖振; 李文科; 毛昕
Original assignee: Hunan Feique Medical Technology Co ltd; Second Xiangya Hospital of Central South University
Current assignee: Hunan Feique Medical Technology Co ltd; Second Xiangya Hospital of Central South University
Priority date: 2022-05-08
Filing date: 2022-05-08
Publication date: 2022-09-23

Abstract

The invention discloses a three-dimensional vision-guided automatic neck ultrasonic scanning method and a system of a mechanical arm. The method comprises the following steps: s1, performing three-dimensional reconstruction on the neck of the human body; s2, performing neck key point detection on the three-dimensional reconstructed image according to a preset key point detection network to obtain neck key point information; s3, planning a path according to the neck key point information and the set matching rule; and S4, guiding the mechanical arm to carry out neck ultrasonic scanning according to the path planning. According to the invention, the characteristic points are extracted according to the three-dimensional visual information of the patient shot by the camera, the path planning of the mechanical arm is automatically planned, and the mechanical arm is guided to carry out neck ultrasonic scanning through the path planning, so that the thyroid diagnosis of the human body is facilitated.

Description

Three-dimensional vision-guided automatic neck ultrasonic scanning method and system for mechanical arm

Technical Field

The invention relates to the technical field of ultrasound, in particular to an automatic neck ultrasonic scanning method and system of a mechanical arm guided by three-dimensional vision.

Background

The health state of the thyroid part can be detected by using ultrasonic equipment so as to judge the health state of the thyroid and part of the body of a patient, and the pathological change condition of the thyroid part is mainly detected by ultrasonic color ultrasonography at present. The technology of adopting the mechanical arm to replace manpower to produce and live is developed and applied relatively rapidly in recent years, and particularly in the industrial field and the medical care field, the mechanical arm is popularized to a certain extent, and human resources are greatly liberated. The existing mechanical arm in the medical field is mainly applied to the nursing industry, such as massage, health care and maintenance, and the like, but has few directly related technical applications for diagnosis and treatment assistance, medicine research and development and the like.

Disclosure of Invention

The invention aims to provide a three-dimensional visual-guided automatic neck ultrasonic scanning method and system of a mechanical arm, which are used for overcoming the defects in the prior art.

In order to achieve the purpose, the technical scheme adopted by the invention is as follows:

a three-dimensional visual guidance automatic neck ultrasonic scanning method of a mechanical arm comprises the following steps:

s1, carrying out three-dimensional reconstruction on the neck of the human body,

s2, performing neck key point detection on the three-dimensional reconstructed image according to a preset key point detection network to obtain neck key point information;

s3, planning a path according to the neck key point information and the set matching rule;

and S4, guiding the mechanical arm to carry out neck ultrasonic scanning according to the path plan.

Further, the step S1 specifically includes:

s11, shooting the neck of the human body by using a binocular camera to obtain a color image of left and right visual fields;

s12, calculating three-dimensional information of each pixel point in the left view image based on a three-dimensional measurement principle;

s13, segmenting the color image according to a preset semantic segmentation network to obtain pixel point coordinates belonging to a human body part;

and S14, constructing a mask according to the pixel point coordinates obtained in the step S13, indexing the mask in the three-dimensional information matrix of the pixel point obtained in the step S12 by using the mask to obtain the space coordinates and normal vectors of each point of the human body, and constructing a point cloud.

Further, the preset key point detection network is a centret key point detection network, the centret key point detection network adopts a ResNet50 or Hourglass network as a main feature extraction network to obtain feature information in an image, and thermodynamic diagram prediction is performed by using the feature information obtained by the network to realize the detection of the key point.

Further, the specific steps of detecting the neck key points in step S2 include:

s21, inputting the obtained color image into a CenterNet key point detection network, and obtaining a pixel coordinate of a neck key point;

s22, indexing the three-dimensional information matrix by using the pixel coordinates and acquiring the three-dimensional coordinates of each key point;

and S23, acquiring position indexes of all key points in the point cloud according to the three-dimensional coordinates to obtain the spatial position and normal vector information of each key point.

Further, the neck keypoints comprise a bottom center point, a bottom left point, a bottom right point, a top left point, and a top right point.

Further, the step S3 specifically includes:

s31, using the obtained spatial position of the key point and the normal vector information as nodes on the motion path of the tail end of the mechanical arm, and interpolating between the nodes to obtain a smooth and complete path;

s32, calculating the Euclidean distance between each point and the interpolation point in the point cloud, setting the minimum Euclidean distance as the matching point of the interpolation point, and acquiring the spatial position and the normal vector information of the matching point from the point cloud.

Further, the step S4 specifically includes:

s41, constructing a Cartesian three-dimensional coordinate system Oxyz by taking the normal vector as the positive direction of the z axis, and enabling the coordinate system to pass through a rotation matrix T _m Conversion to the camera coordinate System Oxyz _camera Rotation matrix T using C + + Eigen library _m Converting into quaternion data;

s42, converting the quaternion data in the camera coordinate system into the data in the base coordinate system according to the following formula,

T _{object-in-base} ＝T _tool-in-base *T _cam-in-too *T _{object-in-can}

wherein, T _tool-in-base As the end joint coordinate system, T _cam-in-too Is a tool end coordinate system, T _{object-in-can} As camera coordinate system, T _object-in- Is a base coordinate system;

and S43, calculating the angle of each joint according to the known space position and attitude of the tail end of the mechanical arm.

Further, the step S43 is specifically: and deducing a transformation matrix from the terminal coordinate system of the tool to the base coordinate system by a D-H parameter calibration method, performing inverse transformation on the transformation matrix of each joint, and performing equation matching and variable separation to obtain the angle of each joint.

Further, in the process of the neck ultrasonic scanning by the robot arm in step S4, the force fed back by the end force sensor is directly fed back to the initial position by using a proportional coefficient by using a direct force-contact feedback control algorithm, and the initial position is corrected as a position correction amount.

The invention also provides a system of the robot automatic neck ultrasonic scanning method based on the three-dimensional visual guidance, which comprises the following steps:

a three-dimensional reconstruction module for three-dimensionally reconstructing the neck of a human body,

the key point detection module is used for detecting key points of the neck of the three-dimensional reconstructed image according to a preset key point detection network to obtain key point information of the neck;

the path planning module is used for planning paths according to the neck key point information and the set matching rule;

and the guiding module is used for guiding the mechanical arm to carry out neck ultrasonic scanning according to the path planning.

Compared with the prior art, the invention has the advantages that: according to the three-dimensional visual guidance mechanical arm automatic neck ultrasonic scanning method and system, the characteristic points are extracted according to the three-dimensional visual information of a patient shot by a camera, the path planning of the mechanical arm is automatically planned, and the mechanical arm is guided to carry out neck ultrasonic scanning through the path planning, so that the thyroid diagnosis of a human body is facilitated.

Drawings

In order to more clearly illustrate the embodiments or technical solutions of the present invention, the drawings used in the embodiments or technical descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a flow chart of the three-dimensional vision-guided robotic arm automated neck ultrasound scanning method of the present invention.

Fig. 2 is a diagram showing the result of human body segmentation in the present invention.

FIG. 3 is a graph showing the detection results of key points in the present invention.

FIG. 4 is a rendering effect diagram of each path point in the human body point cloud.

Fig. 5 is a schematic diagram of the hand-eye calibration of the present invention.

FIG. 6 is a schematic view of the joint angle of the robot arm of the present invention.

FIG. 7 is a direct force-touch feedback control algorithm diagram in accordance with the present invention.

FIG. 8 is a block diagram of the three-dimensional vision guided robotic neck ultrasound scanning system of the present invention.

Detailed Description

The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings so that the advantages and features of the present invention can be more readily understood by those skilled in the art, and the scope of the present invention will be more clearly and clearly defined.

Referring to fig. 1, the embodiment discloses a three-dimensional vision-guided robot automatic neck ultrasonic scanning method, which includes the following steps:

step S1, carrying out three-dimensional reconstruction on the neck of the human body, which specifically comprises the following steps:

and step S11, shooting the neck of the human body by using a binocular camera to obtain a color image of left and right visual fields.

And step S12, calculating the three-dimensional information of each pixel point in the left view image based on the three-dimensional measurement principle.

And S13, segmenting the color image according to a preset deep LabV3+ semantic segmentation network to obtain pixel point coordinates belonging to a human body part. The DeepLabV3+ semantic segmentation network mainly comprises an encoding part and a decoding part and can realize the segmentation of the pixel level of a target object, and the input of the DeepLabV3+ semantic segmentation network is an RGB color image and the output is the segmentation result of the target object in the image. And (3) sending the color image into a DeepLabV3+ semantic segmentation network to realize accurate segmentation of the human body, as shown in figure 2.

And S14, constructing a mask according to the pixel point coordinates obtained in the step S13, indexing the mask in the three-dimensional information matrix of the pixel points obtained in the step S12 by using the mask to obtain the space coordinates and normal vectors of each point of the human body, constructing point cloud, and realizing three-dimensional reconstruction of the human body.

And step S2, performing neck key point detection on the three-dimensional reconstructed image according to a preset key point detection network to obtain neck key point information.

Specifically, the movement path of the robot arm may be different due to different neck sizes of different users. However, by observing the scanning technique and the motion trail of the professional physician, it is found that these paths always pass through several key points of the neck. Therefore, the detection of human neck key points is crucial to the path planning of subsequent mechanical arms, and 5 neck key points are preset in the embodiment, which are respectively as follows: bottom center point (down), bottom left point (left _ down), bottom right point (right _ down), top left point (left _ top), and top right point (right _ top), as shown in fig. 3.

In this embodiment, the preset keypoint detection network is a centret keypoint detection network, and the centret keypoint detection network uses a ResNet50 or a Hourglass network as a main feature extraction network to obtain feature information in an image, and then performs thermodynamic diagram prediction by using the feature information obtained by the network, so as to realize detection of keypoints.

The input of the centret keypoint detection network is an RGB color image, and the output is the pixel coordinates of all keypoints in the image.

In this embodiment, the specific steps of detecting the neck key points in step S2 include:

step S21, inputting the obtained color image into a CenterNet key point detection network, and obtaining the pixel coordinates of the neck key point;

step S22, indexing a three-dimensional information matrix by using the pixel coordinates and acquiring the three-dimensional coordinates of each key point;

and step S23, acquiring position indexes of all key points in the point cloud according to the three-dimensional coordinates to obtain the spatial position and normal vector information of each key point.

And step S3, planning a path according to the neck key point information and the set matching rule. The method specifically comprises the following steps:

and step S31, utilizing the obtained spatial position of the key point and the normal vector information as nodes on the motion path of the tail end of the mechanical arm, and interpolating between the nodes to obtain a smooth and complete path so as to realize the planning of the autonomous path.

The space coordinate of the key point A is set as (x) _A ,y _A ,z _A ) The spatial coordinate of the key point B is (x) _B ,y _B ,z _B ) If the number of points inserted between two points is N, the spatial coordinate (x) of the point i is inserted _i ,y _i ,z _i ) Satisfies the following conditions:

step S32, matching each path point from the point cloud, specifically: and (3) calculating Euclidean distances between each point and the interpolation point in the point cloud, setting the minimum Euclidean distance as a matching point of the interpolation point, and acquiring the spatial position and normal vector information of the matching point from the point cloud, wherein the rendering effect of each path point in the human body point cloud is shown in figure 4.

And step S4, guiding the mechanical arm to carry out neck ultrasonic scanning according to the path planning, wherein the neck ultrasonic scanning is specifically divided into normal vector-to-quaternion, hand-eye calibration and mechanical arm motion inverse resolving. The method comprises the following steps:

step S41, constructing a Cartesian three-dimensional coordinate system Oxyz by taking the normal vector as the positive direction of the z axis, and passing the coordinate system through a rotation matrix T _m Conversion to the camera coordinate System Oxyz _camera Rotation matrix T using C + + Eigen library _m Converted into quaternion data. The algorithm pseudo code is as follows (need: unit normal vector Tmpvec of measured point):

tmpvec and camera coordinate system Oxyz _camera The unit direction vector of the positive direction of the z axis is cross-multiplied to obtain a unit direction vector x; tmpvec is cross-multiplied with x to obtain a unit direction vector y; constructing a Cartesian coordinate system Oxyz coordinate system by taking Tmpvec, x and y as the positive direction of a z axis, the positive direction of an x axis and the positive direction of a y axis respectively; constructing a rotation matrix T _m X, y, Tmpvec }; recall Eigen library m ═>Quaterninond obtains quaternion attitude data.

The pose information of the measured object obtained by the algorithm is a cameraThe original data in the coordinate system and the pose information for controlling the motion of the mechanical arm are data in the basic coordinate system of the mechanical arm, as shown in fig. 5, so that the calibration of hands and eyes is required. There are four coordinate systems for this system: (1) base coordinate system (T) _{object-in-base} ) (ii) a (2) End joint coordinate system (T) _tool-in-base ) (ii) a (3) Tool tip coordinate system (T) _cam-in-tool ) (ii) a (4) Camera coordinate system (T) _{object-in-can} ) (ii) a Has the following relations:

step S42, converting the quaternion data in the camera coordinate system into data in the base coordinate system according to the following formula:

T _object-in- ＝T _tool-in-base *T _cam-in-too *T _{object-in-can}

step S43, calculating the angle of each joint by knowing the spatial position and attitude of the end of the robot arm.

Specifically, as shown in fig. 6, which is a schematic diagram of the joint angles of the robot arm, the inverse solution of the kinematics of the robot arm is to calculate the possible angles of each joint by knowing the spatial position and attitude of the end of the robot arm. And deducing a transformation matrix from the tool end to the base coordinate system by a D-H parameter calibration method. And (3) performing inverse transformation on the transformation matrix of each joint, performing equation matching and variable separation to obtain the angle of each joint, and selecting a group of solutions which are most consistent under the practical conditions of considering the motion attribute of the mechanical arm, safety problems and the like. The robot arm Aubo-C5 is called circularly to move the command robottservicejoinmove (waypoingvector. at (i). joointpos, true), each waypoint is added into a waypoint container (waypoingvector) created in advance, and then the controller drives the end effector to move to a desired position according to a target position in the waypoint container and other robot arm settings, such as a maximum movement speed, a maximum acceleration and a current position of the end effector.

In this embodiment, in the ultrasonic examination process, the probe of the diagnostic apparatus is required to scan closely to the skin of the human body, so that the proper force contact of the diagnostic apparatus to the skin needs to be ensured. And C + + multithreading is adopted to judge and process real-time terminal force-touch information so as to prevent safety accidents and ensure user experience.

In the actual thyroid gland diagnosis process, a tail end contact force signal is fed back by the pressure sensor, the control mode of the mechanical arm of the operating platform is position control, and if the coordinated control of force and position is to be realized, a flexible control algorithm can be adopted to convert a feedback force into a position signal or convert a feedback position into a force signal. Therefore, the embodiment can design a control system for compliant operation of the tail end based on force-touch feedback, the system adopts PD position control as an inner control ring, a compliance algorithm as an outer control ring, and signals collected by the pressure sensor are processed and then used as the input of the inner control ring when the mechanical arm moves. In order to realize compliance control, firstly, a direct force touch feedback control algorithm is utilized to directly feed back force fed back by a tail end force sensor to an initial position by utilizing a proportional coefficient, the force is used as a position correction quantity to correct the initial position, and the corresponding direct force touch feedback control rate is as follows:

τ＝K _pd ·q _d

wherein the position correction amount q is used _d To correct the initial position q _d ，K _pd The direct force feedback parameters are expressed, and the corresponding control quantity input to the position controller is as follows:

q _r ＝q _d -τ/K _pd

a corresponding direct force-touch feedback control algorithm is shown in fig. 7, according to which compliance control can be realized under the condition that the actual position deviates from the preset position, wherein the force F generated under the action force with the environment can be converted into the action moment based on the joint layer through conversion of the jacobian matrix.

Referring to fig. 8, the present invention further provides a system of the robotic arm automatic neck ultrasonic scanning method according to the three-dimensional visual guidance, comprising: the system comprises a three-dimensional reconstruction module 1 for performing three-dimensional reconstruction on the neck of a human body, and a key point detection module 2 for performing neck key point detection on an image after the three-dimensional reconstruction according to a preset key point detection network to obtain neck key point information; the path planning module 3 is used for planning paths according to the neck key point information and the set matching rule; and the guiding module 4 is used for guiding the mechanical arm to carry out neck ultrasonic scanning according to the path planning.

According to the invention, the characteristic points are extracted according to the three-dimensional visual information of the patient shot by the camera, the path planning of the mechanical arm is automatically planned, and the mechanical arm is guided to carry out neck ultrasonic scanning through the path planning, so that the thyroid diagnosis of the human body is facilitated.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, various changes or modifications may be made by the patentees within the scope of the appended claims, and within the scope of the invention, as long as they do not exceed the scope of the invention described in the claims.

Claims

1. A three-dimensional vision-guided automatic neck ultrasonic scanning method of a mechanical arm is characterized by comprising the following steps:

s1, carrying out three-dimensional reconstruction on the neck of the human body;

s3, planning a path according to the neck key point information and a set matching rule;

2. The three-dimensional visually guided robotic automatic neck ultrasound scanning method of claim 1, wherein said step S1 specifically comprises:

3. The three-dimensional visual guidance mechanical arm automatic neck ultrasonic scanning method according to claim 1, wherein the preset key point detection network is a centret key point detection network, the centret key point detection network adopts ResNet50 or Hourglass network as a main feature extraction network to obtain feature information in an image, and then thermodynamic prediction is performed by using the feature information obtained by the network to realize the detection of key points.

4. The three-dimensional vision-guided robotic automatic neck ultrasonic scanning method of claim 3, wherein the specific step of neck key point detection in step S2 comprises:

s21, inputting the obtained color image into a CenterNet key point detection network, and obtaining pixel coordinates of neck key points;

5. The method of claim 3 or 4, wherein the neck keypoints comprise a bottom center point, a bottom left point, a bottom right point, a top left point, and a top right point.

6. The three-dimensional visually guided robotic automatic neck ultrasound scanning method of claim 1, wherein said step S3 specifically comprises:

7. The three-dimensional visually guided robotic automatic neck ultrasound scanning method of claim 6, wherein said step S4 specifically comprises:

T _object-in-b ＝T _tool-in-base *T _cam-in-tool *T _object-in-ca

wherein, T _tool-in-bas As the end joint coordinate system, T _cam-in-tool Is a tool end coordinate system, T _object-in-ca As camera coordinate system, T _object-in-b Is a base coordinate system;

8. The three-dimensional visually guided robotic neck ultrasound scanning method of claim 7, wherein said step S43 is embodied by: and deducing a transformation matrix from the tool terminal coordinate system to the base coordinate system by a D-H parameter calibration method, inversely transforming the transformation matrix of each joint, and performing equation matching and variable separation to obtain the angle of each joint.

9. The method according to claim 7, wherein in step S4, during the neck ultrasonic scanning of the robot arm, a direct force-touch feedback control algorithm is used to directly feed back the force fed back from the end force sensor to the initial position by using a scaling factor, so as to correct the initial position as a position correction.

10. A system for the three-dimensional visually guided robotic automated neck ultrasound scanning method according to any of claims 1-9, comprising:

the key point detection module is used for detecting the key points of the neck of the image after the three-dimensional reconstruction according to a preset key point detection network to obtain the key point information of the neck;

and the guiding module is used for guiding the mechanical arm to carry out neck ultrasonic scanning according to the path plan.